User interface for electronic device, input processing method and electronic device

Abstract

A user interface (200) enabling input with a pointing medium, comprising: an advertisement (151); a proximity sensor (153) for generating proximity coordinates of the pointing medium in a proximity state to a surface of the display (151) having an original image (303) displayed thereon; a touch sensor (155) for generating contact coordinates of the pointing medium in a contact state with the surface of the display (151); a partial image control section (207) for generating data for displaying a partial image (313) on the display, which is obtained by enlarging a predetermined area of ​​the original image (303) in a manner to be overlapped on the original image (303), when a flying swipe operation performed with the pointing medium is recognized in the approach state from the approach coordinates, wherein the predetermined area of ​​the original image (303) includes start coordinates of the flying swipe operation; and a contact coordinate processing section (213) for outputting the contact coordinates of the partial image (313) as contact coordinates of the original image (303) to a user function.

Description

Field of the InventionThe present invention relates to a technique for facilitating input on a touch screen, and more particularly to a technique for facilitating input on a touch screen on which small operation objects are displayed.BACKGROUND OF THE INVENTIONAn electronic device such as a smartphone or a tablet terminal allows a user to perform a touch operation with a finger or a stylus pen on an operation object displayed on a touch screen composed of a combination of a display and a touch sensor to perform a screen operation or input of characters. When the touch screen is small, the user may touch an operation object adjacent to the target operation object because the operation object also tends to be small.Further, in an OS such as the Windows (registered trademark) 8 / 10, which allows the user to use both a mouse operation mode for operations with a mouse pointer and a touch panel operation mode for touch operations on a touch screen, there is a case where small operation objects are included on the premise of using a mouse pointer. In this case, touch operations become more difficult to perform. Various attempts have been made to solve this problem. Microsoft (registered trademark) Co., Ltd. offers a user interface (UI), called Modem UI, with new display type and operating rules. In the modem UI, the icons called tiles are displayed in an appropriate size for a touch operation.Lenovo Co., Ltd. offers a UI called Aura UI in which a new shell for touch operation is superimposed on an existing operating system. In the case of a multi-touch OS, the screen may be temporarily enlarged with spreading as a touch operation for spreading two fingers apart to prevent misoperation. Patent Document 1 discloses a display device that performs input with a hover operation in which a finger is held at a spatial distance position from the plane of the touch panel, a hover slide operation by sliding the finger which is substantially parallel to the plane of the touch panel in the hover operation, configuring a touch panel, and a touch operation.The display device acquires, on the touch panel, the coordinates of the finger in the hover operation and the hover slide operation and the coordinates of the finger in the touch operation. When the proximity of the finger to the touch panel is detected in the hover operation, the display device displays an enlarged screen having the center located directly below the finger when overlapping the enlarged screen on an original image, and when the hover slide operation is performed, the display device displays an enlarged screen located directly below the target point. Patent Document 2 discloses a touch panel device for providing an enlarged display when a finger approaches a capacitive touch panel. The document teaches that a keypad on the screen is magnified and shown with a screen key in the center that an index finger first approaches the keypad on the screen, and that the magnified image remains displayed until the finger moves away therefrom. Patent Document 3 discloses a touch panel device for providing an enlarged display when a finger approaches a capacitive touch panel. The document teaches that a controller transforms the coordinates of an enlarged display area into the coordinates of an original image before enlarging.Patent Document 4 discloses a control method for a touch panel display in which a screen is enlarged as a finger approaches a touch pad. The document teaches that the enlarged screen is displayed as long as the finger is located within a certain distance from the surface of the touch panel, and that the enlarged screen is erased with a touch operation or when the finger is moved far away from the touch panel. Patent Document 5 discloses a user interface for displaying an enlarged screen at a magnification that varies according to the distance of a finger from a touch panel.Non-patent document 1 discloses an interface using a gesture of fingers near a tablet surface. The document teaches that multi-layered IR planes are synchronized with a shutter signal from a high speed camera to detect the gesture of the fingers. The document also teaches that a card is zoomed or curled by pointing to the card with a finger and varying the height of the finger.Non-Patent Document 2 discloses a 3D touch panel interface for detecting an approach state before a user's finger comes into contact with a touch panel to change focus on a button or to magnify the button depending on the approach position. The document describes a 3D touch panel for detecting a finger proximity state using only a normal touch panel in addition to a contact state of a finger. The document also teaches that the speed or acceleration of the finger as it approaches the touch panel is calculated from a time transition of the proximity distance to switch interfaces.[Patent Document 1] Japanese Patent Application Laid-Open JP 2013-225 261 A.[Patent Document 2] PCT International Publication WO 2012 / / 132802[Patent Document 3] PCT International Publication WO 2011 / 027665[Patent Document 4] Japanese Patent Application Laid-Open JP H11-65 769 A[Patent Document 5] U.S. Patent Application Laid-Open US 2013 / 0293490 A[No Patent Document 1] Yoshiki Takeoka, et al. Z-touch: Infrastructure for 3D gesture interaction in the vicinity of tabletop surfaces, [online], ITS 2010: Devices & Algorithms, [searched on August 1, 2015], the Internet <URL: http: / / lab.rekimoto.org / projects / z-touch / >[No Patent Document 2] Hiroyuki Washino, et al. 3D Touch Panel User Interface, [online], Information Processing Society of Japan Interaction 2009, [searched on August 1, 2015], the Internet <URL: http: / / www.interaction-ipsj.org / archives / page2009 / interactive / 0037 / 0037.pdf>EP 2 624 116 A1 discloses a user device comprising: a touch screen display having a touch screen input device; a proximity detector array comprising a plurality of sensors for detecting a location of an input device; and a control unit comprising a processor. The controller is configured to receive data from the proximity detector to determine the location of the input device, including a value related to the distance of the input device from the display. It also controls the display to perform a zoom function and to enlarge or reduce the content of a part of the display based on the value related to the distance of the input device from the display. The proximity detector arrangement and the control unit are configured to detect the position of the input device within a detection volume and to allow gradual change in the increase or decrease in content size depending on how far the input device is from the display.From US 2011 / 0 261 078 A1 it is known that a magnification window is overlaid on a screen of a user interface and dynamically repositioned in real time depending on where a pointing device is located on the screen.SUMMARY OF THE INVENTIONIt is an object of the present invention to enable an improved input on a display of a user interface.This object is achieved by the subject matters of the main claim 1 and the subordinate claims 14, 15 and 16 which determine the present invention.Preferred embodiments of the present invention are the subject of the dependent claims.The touch screen is excellent in the ability to perform intuitive operation with an operation object on a screen, but it is required that erroneous operation should be prevented. Although the modem UI or the aura UI touch operation achieve few erroneous operations in a specific environment, a method that can achieve touch operations in such an environment that do not affect an existing application or an existing operating system or OS is desirable. Further, in an electronic device such as a smartphone or tablet terminal having a touch screen used as a fixed input device, it is common to set operation objects larger than a finger or a stylus pen in advance. On the other hand, this limits the number of operation items included on the screen having a predetermined size, and that is, the screen density or the amount of information becomes coarse mesh.In an OS (operating system) that allows the user to use both the mouse operation mode and the touch panel operation mode when the operation objects are set to an appropriate size for the mouse operation, an erroneous operation may occur in a touch operation, while when the operation object is set to an appropriate size for the touch operation, the screen may become more coarse than a degree to which the mouse operation is possible. A method of enlarging the screen by means of spreading may be accompanied by a spreading operation to restore the screen, and this represents an extra burden on the user. There is also a method of enlarging a predetermined area over coordinates of a center, in which the touch operation is performed to perform a touch operation on the enlarged image. However, since the enlarged image is always displayed even if the touch operation is possible without enlarging the image, the user can feel strange.In the documents of the related art, a gesture made near the surface is detected to magnify the screen before the touch operation. These related art documents have some of the following disadvantages: First, the user may stop a finger on the touch panel before the touch operation is performed to select or think of an operation target. At this time, when an enlarged screen is displayed immediately after the proximity of the finger is detected, a user who continuously desires to see a view of a wide area of the original image without being enlarged is forced to perform an operation for deleting the enlarged screen, and thus the user can feel uncomfortable. Further, even when an operation object is large enough to enable accurate touch operation without being enlarged, displaying an enlarged screen causes the user to restore the feeling of touch operation on an original image.It is difficult for the user to perform a touch while controlling the distance from the touch panel as a finger approaches the touch panel and the speed or acceleration of the approach. Further, continuously maintaining the finger at a certain distance range from the touch panel to continuously display the enlarged screen may place the user in an emergency posture. Further, when the enlarged screen is displayed just under the finger, the operation object may be hidden under the finger.When an enlarged screen is displayed, the operations added to a touch operation on the operation object must be as simple as possible so as not to cancel the feeling of the touch operation on the touch screen. It is also desirable that the conventional touch operation performed without displaying the enlarged screen be maintained. Further, it is desirable that the enlarged screen is displayed depending on the intention of the user.Therefore, it is an object of the present invention to provide a user interface for preventing erroneous operation while preventing the feel of an operation touch from being reversed on the touch screen. Another object of the present invention is to provide a user interface that is superior in matching to an existing OS or application. It is still another object of the present invention to provide a user interface for displaying a sub-image at a predetermined magnification relative to an original image only when intended by the user. Another object of the present invention is to provide an input processing method, an electronic device, and a computer program adapted to such a user interface.The present invention relates to a user interface that enables input to an electronic device having a pointing medium. The user interface may be included in the electronic device or connected to the electronic device wirelessly or via a cable. The user interface includes a display; a proximity sensor for generating proximity coordinates of the pointing medium in a proximity state to a surface of the display with an original image displayed thereon; a touch sensor for generating contact coordinates of the pointing medium in a contact state with the surface of the display; a sub-image control section for generating data for displaying a sub-image on the display obtained by enlarging a predetermined range of the original image in a manner overlapping the original image when a flying swipe operation performed with the pointing medium in the proximity state is recognized by the proximity coordinates; and a contact coordinate processing section for outputting contact coordinates on the sub-image as contact coordinates of the original image to a user function.The pointing medium may be a fingertip or stylus pen. According to the above configuration, since a predetermined area of the original image can be enlarged with a simple flying swipe operation, spreading to enlarge the entire screen is unnecessary when looking at small characters. The partial image is not displayed even when the pointing medium is in an approach state unless a flying patrol operation is performed. Thus, since the partial image can be displayed only when intended by the user, the user arranging a fingertip in the proximity state while watching and superimposing the original image before the touch operation is not disturbed.Further, it may be rather convenient for the user not to display a partial image of a sufficiently large operation object. However, in the present invention, touch operation can be performed on the original image without displaying the sub-image in the conventional manner as long as the flying patrol operation is not performed. Therefore, erroneous operation can be prevented while preventing the feeling of touch operation on the touch screen from being reversed. Further, there is no need to change an existing operating system or application to output, to the user function, the contact coordinates as the contact coordinates of the original image.The proximity sensor and the touch sensor may be configured by a touch panel, and a touch panel may be configured by the display and the touch panel. The contact coordinate processing section may output, to the user function, contact coordinates corresponding to a first patrol operation performed on the original image with the pointing medium in the contact state. Since the user interface processes the flying patting operation with the proximity coordinates, patting operations known heretofore such as for moving an operation object in a patting direction processed at the contact coordinates, scrolling the screen, and the like can also be turned on. The user interface may also have a pointing device for creating an operation to control a mouse pointer displayed by the original image. In this case, even if the original image includes a small operation object suitable for operation with the mouse pointer, erroneous operation by a touch operation on the partial image can be prevented.A predetermined area of the original image may include start coordinates for a flying patrol operation. Since the user can easily recognize the start coordinates of the flying patrol operation, the user can obtain, in the partial image, an operation object overlapping with the start coordinates simply by including the start coordinates in the partial image. The sub-image control section may display an image point just below the pointing medium in the proximity state. This allows the user to use the pixel to perform a touch operation accurately on an operation object on which a touch operation is likely to be performed without displaying the partial image. The partial image control section may change the display of the one operation object when the pointing medium is placed just above an operation object of the original image in the proximity state. Thus, the user can recognize the presence of an operation object that is difficult to recognize in the original image by the display change. Further, the user can perform a touch operation accurately on an intended operation object without displaying the partial image.The sub-picture control section can delete the sub-picture when a predetermined time has elapsed since the pointing medium is moved in a separated state after the sub-picture is displayed. Once the sub-image is displayed, the user can merely restore the screen to the original image state by subtracting the pointing medium when the input is abandoned. The sub-image control portion may generate data for displaying the sub-image such that an operation object included in the sub-image corresponds to an operation object of the original image arranged around the operation object included in the original image.The sub-image control portion may determine a position of the sub-image that is relative to the original image corresponding to a moving direction from the start coordinates to the flying patting operation. This allows the user to control the display position of the sub-image in the direction of the flying patting operation. The sub-image control portion may determine a magnification of the sub-image relative to the original image according to the moving direction from the start coordinates of the flying patting operation. This allows the user to select the direction of the flying patting operation to display an enlarged partial image or a reduced partial image. Further, the user can display a screen without a pinch effect on which only a required area is reduced.The sub-image control portion may determine a magnification of the sub-image relative to the original image according to the speed and acceleration of the pointing medium moving when the flying patting operation is performed. When a more magnified partial image is to be displayed, the user can perform a faster flying patrol operation. The sub-image control section may display the sub-image at a magnification varying with respect to the original image on a step by step basis each time in a flying patrol operation repeatedly performed at a predetermined time interval or shorter. This allows the user to repeat the flying patting operations to vary the magnification of the sub-image in small steps step by step.The sub-image control section may recognize the arrangement of a plurality of operation objects included in the original image to set both or only one of the shape and the size of the sub-image according to the arrangement of the operation objects. This allows the user to obtain more operation objects on which touch operations are most likely to be performed in a certain-size partial image when the plurality of operation objects are arranged in the original image.According to the present invention, a user interface for preventing an erroneous operation can be provided while preventing the feeling of a touch operation from being reversed on the touch screen. A user interface can also be provided which is better when adapting to an existing OS (operating system) or an application. Further, a user interface for displaying a sub-image at a predetermined magnification to the original image may be provided only when the user intends. Furthermore, an input processing method, an electronic device and a computer program can be provided, which are adapted to such a user interface.Brief Description of the DrawingsFIG. 1 is a diagram for describing a state when a touch operation and a flying swipe operation are performed on a display 100. FIG. 2 is a functional block diagram for describing the configuration of a user interface 200. FIG. 3 is a main flow diagram for describing an example of a process when the user interface 200 displays a patrol window. FIG. 4 is a flowchart for describing an example of a process for displaying a sub-image. FIG. 5 is a diagram for describing a state of a tap operation on an original image 303. FIG. 6 is a diagram for describing a state when a pixel 307 is displayed on coordinates immediately below a finger syringe and a state when the flying stripe is started. FIG. 7 is a diagram for describing a state when a partial image 313 is displayed in a manner to overlap an original image 303. FIG. 8 is a diagram for describing an example of a rigid window 325 configured by using the motion information about the flying rigid operation. FIG. 9 is a diagram for describing another example of the rigid window 325 configured by using the motion information about the flying rigid operation. FIG. 10 is a diagram for describing an example of determining the shape of the rigid window 325 based on the arrangement of the operation objects included in the original image 303.DETAILED DESCRIPTION OF THE INVENTION[Touch operation and Flying Pat No.]FIG. 1 is a diagram for describing a state when a touch operation and a flying swipe operation are performed on a display 100. The XY axes are defined on a surface 100 aof the display 100, and a Z axis is defined in a direction perpendicular to the XY plane. A fingertip is, relative to the display 100, in a state separated from the surface 100 aof the touch screen by a distance Z 1 in the Z-axis direction, in a state separated by a distance less than the distance Z 1 and greater than the distance Z 2, in a state separated by a distance less than the distance Z 2 as shown in FIG. 1B, or in a state in which the fingertip is in contact with the touch screen when the distance Z is zero as shown in FIG. 1C.A user interface according to the embodiment accepts an input with a touch operation in a contact state in which the finger is in contact with the surface 100 aas illustrated in FIG. 1C. At this time, the flying patting operation performed in the proximity state in which the finger approaches the surface 100 aby a certain distance as illustrated in FIG. 1B may be used in addition, if necessary. The state in which the fingertip is separated from the surface 100 aby the distance Z 1 or more is referred to as a separation state. When the finger approaches the surface 100 afrom the separation state to a distance Z 2 or less, the state of the approach state occurs. After the state has entered the proximity state, the proximity state is maintained until the fingertip comes into a contact state or a state separated by the distance Z 1 or more.Thus, when the position of the fingertip in the Z-axis direction is within a range up to the distance Z 2, in the extreme case up to the distance 0, the state corresponds to the proximity state. When the fingertip is located between the distance Z 1 and the distance Z 2, the state is either the proximity state or the separation state depending on how the fingertip reaches the position. Note that the touch operation and the flying patting operation may be used using a conductive pointing medium such as a stylus pen instead of a fingertip. Here, the flying patting operation will be described. A patting operation in a contact state is well known. The flying swipe operation is a gesture for performing the same movement of the fingertip in the proximity state as each of the swipe operation in the contact state (swipe operation). However, since the flying scraping operation is performed in the air, there is no guide surface on which the fingertip can slide as in the contact scraping operation that causes easy movement of the fingertip in the Z-axis direction.Although a variety of gestures are defined for touch operations performed in the contact state, the movement of the finger in the XY direction to be extracted from the flying patting operation may be detected by the same algorithm as the contact patting operation. Touch operations mean all input operations performed by bringing the finger or a stylus pen (both of which will be referred to simply as a fingertip in this specification hereinafter) into contact with the surface 100 aof the display 100. The touch operations include both an input operation on an operation object such as an icon associated with a specific application or file to be displayed on the display 100 or a character or image object associated with a predetermined address and an input operation on any display area other than the operation object.The touch operations include tap operations without any change in the position of a touch operation in a series of operations and gesture operations with a change in the position. A system that detects a tap operation may obtain information such as the coordinates of the touch operation, the amount of time that the touch operation is continued on the coordinates, and the number of touch operations. The gesture operations include single touch operations such as a swipe operation, a swipe operation, a drag operation, and a turn operation, and multiple touch operations such as a pinch-in and a pinch-out operation. The system that detects a gesture operation may include information such as a coordinate trajectory of the touch operations and the speed and acceleration of the coordinate change. Then, the system may identify the type of gesture by the coordinate trajectory patterns, the direction of the change, and the like.The touch-strip operation is an operation for moving a finger with which a touch operation is performed at a short distance in an approximately constant direction. The swiping operation is an operation, also referred to as a sliding operation, for moving the finger with which the touch operation is performed at a distance larger than that of the swiping operation in an approximately constant direction. The touch operation is an operation in which the moving speed of the finger is faster than that of the swiping operation. The drag operation is an operation for moving, to a specific position, the finger with which a touch operation is performed on a specific object.In the touch operation features, there is no need for the user to be notified of the coordinates of a destination of the fingertip, and for the moving distance to be shorter and the moving speed to be faster than in the swiping operation and the dragging operation. Therefore, the touch-up operation is suitable for use in scrolling a page, scrolling a screen at high speed, and moving the fingertip in a specified direction to ensure input similar to the swiping inputs on a keyboard, all of which little affects the destination of the coordinates.Although the fingertip is in the state of approaching the surface 100 ain a flying patting operation, it is easily applicable to an existing system because the system can detect the flying patting operation by the same algorithm as a contact patting operation with respect to the moving speed and the moving distance of the fingertip in the XY direction. In order to detect the flying patting operation, a proximity sensor is required to detect the coordinates of the fingertip in the proximity state.[Configuration of User Interface]FIG. 2 is a functional block diagram for describing the configuration of a user interface 200. The user interface 200 is configured to include an input device composed of a display 151, a proximity sensor 153, and a touch sensor 155, and hardware sources such as a CPU, a RAM, and a chipset in cooperation with software sources such as applications, an OS, and device drivers.The user interface 200 may also include a pointing device 157 such as a mouse, a pointing pen, or a touch pad to control the mouse cursor. In other words, the user interface 200 may be supported by both the mouse operation mode and the touch panel operation mode. Before being performed, the software sources are stored in a nonvolatile memory mounted in the user interface 200 or an electronic device having the user interface 200 such as a computer, a tablet terminal, or a smartphone.The proximity sensor 153 outputs a physical value capable of generating the XYZ coordinates of a fingertip in the proximity state with respect to the display 151. There is no need to particularly limit the type and basic principles of the proximity sensor 153, and an electromagnetic induction sensor, an optical line sensor using infrared light, a stereo camera, an ultrasonic sensor, or the like provided as an assembly different from the display 151 may be used.The touch sensor 155 outputs a physical value capable of outputting the XY coordinates of a fingertip in the state of contact with the surface of the display 151. Thus, there is no need to particularly well limit the type and basic principles of touch sensor 155. A capacitive touch panel functioning like a touch sensor may detect, as the size, the capacitance while the distance Z of the fingertip is in the proximity state from the touch panel surface. In the embodiment, description is made by exemplarily showing the capacitive touch panel composed of transparent electrodes functioning as the proximity sensor 153 and the touch sensor 155 and a touch screen 150 set on the display 151.In an example, an approximate coordinate generation section 211, a contact coordinate processing section 215, and an operation generation section 217 may be configured by device drivers and a device controller. The proximity coordinate generation section 211 generates the XYZ coordinates 150 of a fingertip in the proximity state from a physical value output through the touch screen 150. In the proximity state, the distance between the surface of the touch screen 150 and the fingertip takes different values. When the proximity state is detected from the Z-axis distance of the fingertip and guide historical information to the distance, the proximity coordinate generating section 211 outputs, as proximity coordinates, coordinates corresponding to a physical value identifiable as the proximity state to the proximity coordinate processing section 209.The contact coordinate generation section 215 generates the XY coordinates of a fingertip in the contact state (contact coordinates) from a physical value output through the touch screen 150. When the contact state is detected by a Z-axis distance of the fingertip or from a physical value output through the touch screen 150, the contact coordinate generating section 215 outputs the contact coordinates. When the proximity coordinate system and the contact coordinate system are different, both must be associated with each other to execute a touch operation partial image 313 (FIG. 7 ), as will be described later as a touch operation on the original image 303 (FIG. 7 ). However, in the embodiment, both are the same to configure the two coordinate systems on the touch screen 150. The pointing device 157 outputs a physical value corresponding to a user operation for moving a mouse cursor. The operation generation section 217 outputs an operation corresponding to an operation for moving the mouse cursor or a mouse click.In the proximity state, the fingertip performs a change in capacitance in a region of the cells wider than that in a contact state. The approximate coordinate processing section 209 outputs, as approximate coordinates, a range of cells wider than that when the contact coordinate processing section 215 generates contact coordinates. The YX coordinates on the touch screen 150 formed by a right-angle line from the finger tip pad to the touch screen 105 are referred to as just-below coordinates. The approximate coordinate processing section 209 generates the coordinates just below from a set of approximate coordinates and notifies a sub-image control section 207 of the generated coordinates. For example, the just-below coordinates may be set as a gravity center of the set of approximation coordinates.A contact coordinate processing section 213 generates coordinates of a fingertip from a set of contact coordinates and sends the generated coordinates to a user function 201. The contact coordinates processing section 213 acquires contact coordinates when a touch operation is performed to employ the contact coordinates as start coordinates, and transmits the generated coordinates to a user function 201. The contact coordinate processing section 213 acquires contact coordinates when a touch operation is performed to employ the contact coordinates as start coordinates, and sends operation data including information on moving direction and moving speed or acceleration to the user function 201. The touch operation includes a touch strip operation. In the embodiment, the touch swipe operation is performed as a direct input to the user function 201, while a flying swipe operation is used only as an input associated with a swipe window 325 (FIG. 7 ). Thus, since the flying patrol operation is provided only for providing display by overlapping a sub-image 313 on an original image 303, it does not affect the OS or an existing application.The user function 201 includes applications, an OS, and the like for providing browser browsing, editing of photos, document generation, and the like to a user through the touch screen 150. The user function 201 sends image information to an original image generation section 203 and to the original image data generation control section 207. The original image generation section 203 includes a buffer for storing image data after being forced to generate image data from the image information received from the user function 201 to provide display on the touch screen 150.The sub-picture control section 207 generates data on the sub-picture 313 from the image information received from the user function 201. The sub-image control section 207 monitors a temporal change in the coordinates just below to detect a flying patting operation. The partial image control section 207 extracts coordinates (start coordinates) just below from the flying patrol operation as a position where the flying patrol operation is started, and movement information such as the movement direction, the movement speed or the acceleration, and the like.The sub-image control section 207 generates data on the sub-image 313 displayed in a patrol 325 from the image information received from the user function 201, the start coordinates extracted from the flying patrol operation, and outputs the data to an image data synthesis section 205. The partial image control section 207 may generate and transmit data on a spot image to the image data synthesis section 205 to be displayed on the just-below coordinates in the proximity state and in the flying patrol operation. The spot image is used to allow the user to recognize an operation object overlapped with the coordinates just below in the proximity state or to perform a tap operation without displaying the slide window 325.When the coordinates just below overlap an operation object such as a button, a character, or a graphic included in an original image displayed on the touch screen 150, the partial image control section 207 may send a cause for highlighting the operation object to the user function 201. Since the user can clearly recognize the highlighted operation object, it is advantageous to prevent erroneous operation when a touch operation is performed directly without a flying patting operation.The sub-image control section 207 may determine the size, shape, and coordinates of the rigid window 325 and the like based on the start coordinates included in the flying patrol operation and the movement information. The sub-image control section 207 may determine magnification for enlarging or reducing the sub-image relative to the original image based on the start coordinates included in the flying patrol operation and the movement information. The sub-image control section 207 sends the coordinates of the sub-image 313 to the contact coordinate processing section 213. The contact coordinate processing section 213 transforms the coordinates of the touch operation to the partial image 313 into the coordinates of the touch operation to the original image 303, and sends the coordinates to the user function 201.[Operation of User Interface]FIG. 3 is a main flowchart for describing an example of a process when the user interface 200 displays a slide window, and FIG. 4 is a flowchart for describing an example of a process for displaying the slide window. FIGS. 5 to 10 are diagrams 325 for description states when the patrol 325 is displayed with a flying patrol operation.Prior to starting operation at block 501, an original image 303 includes objects 303a-303d in a window that displays a title bar 302 on touch screen 150, as shown in FIG. 5A. On the touch screen 150, a mouse cursor 305 controlled by the pointing device 157 is also displayed. Although the user interface 200 can operate both the mouse operation mode and the touch panel operation mode for the original image 303, the present invention can also be applied to a user interface that supports only the touch panel operation mode.Since an original image operated in the mouse operation mode tends to have operation objects smaller than those of an original image operated in the touch panel operation mode, the application of the present invention is effective. Further, the present invention can be applied to an original image in both multi-window and single-window systems. The title bar 302 includes operation keys 301 ato 301 cas examples of the operation objects. A user flips the operation button 301 ato close the window, flips the operation button 301 bto reduce or store the window size, and flips the operation button 301 cto minimize the window.The mouse cursor 305 is moved to each of these operation objects to activate the object with a click operation or by tapping with a fingertip to perform a function defined by the operation object. In block 503, the user attempts to tap the button 301a as shown in Figure 5B to close the window of the original image 303. Since the size of the button 301 ais not large enough compared to the size of the fingertip, the user may randomly tap the adjacent button 301 bin typing the button 301 a.The user moves his or her fingertip directly around the key 301a. When the proximity state is detected, the proximity-coordinate generating section 211 starts outputting a set of proximity coordinates. The approximate coordinate processing section 209 outputs coordinates just below. In block 505, the sub-picture control section 207 displays a spot picture 307 on the just-below coordinates as needed, as shown in Fig. 6A. Although the spot image 307 appears to be overlapped with the original image 303, the spot image 307 is displayed on the just-below coordinates without affecting the operation of the user function 201. By the spot image 303, the user can easily recognize the start coordinates of the flying patting operation, which also makes it easy to perform touch operation on a large operation object overlapped with the spot image 307 even with parallax without displaying the patting window 325.In another example, when an operation object is on the coordinates just below, the sub-image control section 207 instructs the user function 201 to highlight the operation object. The highlighted operation object is part of the original image 303, and this allows the user to perform a touch operation on the accurately highlighted relatively large operation object even with parallax without displaying the slide window 325.Before the tap operation in a state where the fingertip enters the proximity state, the user can see whether the window should actually be closed. In some cases, the user may be uncertain whether to reduce or minimize the window. However, even if the fingertip enters the proximity state, the user does not feel uncomfortable through the swipe window 325 because the partial image control portion 207 does not display the swipe window 325 when there is no need for the user unless a flying swipe operation is detected.In block 507, the user performs a flying patting operation using the operation object 301 a, as represented by the starting coordinates 306 in FIG. 6B. The sub-image control section 207 displays the patrol 325 and the sub-image even when the start coordinates of the flying patrol operation do not overlap any of the operation objects. Therefore, in order to display the slide window, the user does not need to set, as start coordinates, an operation object intended to be operated. The user can perform the flying patting operation by moving the previous part of the second link of the index finger to quickly move the fingertip. The field control section 207 generates and sends the field 313 to the image data synthesizing section 205.In block 509, the image data synthesis section 205 sets the window of patrol 325 to overlap with the original image 303 as shown in FIG. 7, and displays the partial image 313. The sub-image 313 includes operation objects 315 a, 315 b, and 315 ccorresponding to the operation objects 301 a, 301 b, and 301 c. The process in block 509 will be described later with reference to FIG. 4. At this point, there is no touch operation on the original image 303. In the block 511, the proximity coordinate generation section 211 continuously monitors the proximity state and stops outputting the proximity coordinates when the proximity state is shifted to the disconnection state. If the proximity state is maintained, the process continues to block 513, while if the proximity state is shifted to the disconnection state, the process continues to block 551. In block 513, the user performs a tap operation on the operation object 315 aincluded in the sub-image 313.When the partial image 313 has been made larger than the original image 303, the tap operation on the operation object 315 acan be more accurately performed with the fingertip. At the moment when the flying patting operation is completed, because the index finger is not on the operation object 315 aas illustrated in FIG. 6B, the user can clearly see the operation object 315 ain the partial image 313 displayed around the start coordinates 306 as illustrated in FIG. 7A, and this makes the operation easy. Further, when the partial image 313 is located at a position away from the start coordinates 306 as illustrated in FIG. 7B, the user can perform the tap operation while looking at both the original image 303 and the partial image 313. In either method, the partial image 313 may be displayed in a position convenient for the user.In block 513, when the user performs the tap operation on the operation object 315 a, the contact coordinate processing section 213 transforms the coordinates and sends the coordinates of the corresponding operation object 301 aincluded in the original image 303 to the user function 201. In block 515, the user function 201 performs processing defined for the touch operation on the operation object 301 aof the original image 303. The user function 201 does not need to recognize the coordinates of the sub-image 313 to process the tap operation of the sub-image 313. Therefore, the user function 201 does not need a change operation involved in displaying the sub-image 313 in the slide window 325.The user may perform a touch-swipe operation separately from the flying swipe operation. When coordinate data corresponding to a contact scroll operation is received from the contact coordinate processing section 213, the user function 201 performs processing corresponding to scrolling the screen while turning a page, with scroll input to a software key pad, or the like. When the display of the partial image 313 is realized with a touch operation, this function cannot be used. In the embodiment, however, these functions are not affected because the display of the sub-image 313 is realized on a layer different from the touch operation.In block 551, the partial image control section 207 determines whether a predetermined time has elapsed since the fingertip moves to the separation state from the time of stopping the input of the coordinates just below. When the predetermined time has elapsed, the field control section 207 recognizes the end of the proximity state from the time of stopping the input at block 531 to cancel the insertion of the strip window 325 and stop the output data over the field 313. As a result, the image data synthesis section 205 erases the display of the sub-image 313.Thus, when the user can give the tap operation to the partial image 313, even when the partial image 313 is displayed, the user can move the fingertip to the separation state to delete the partial image 313. In block 553, if the fingertip moves back to the proximity state within the given time period, the process proceeds to block 513 to enable a tap operation. Thus, since the flying patting operation can be completed, even when the fingertip is temporarily moved to the separation state during a period from the start coordinate data to the end of the flying patting operation, an accurate gesture is not required by the user.Referring next to FIG. 4, an example of the operation in block 509 to display the sub-image will be described. The sub-image control section 207 that recognizes a flying patting operation in the block 601 determines whether enlargement / reduction has been set in the block 603. If an increase / decrease is set in the field control section 207, the operation proceeds to block 651, while if it is not set, the operation proceeds to block 605. In block 605, the sub-image control section 207 determines whether a repeated flying patrol operation is set.The repeated flying patting operation means that multiple flying patting operations are continuously performed at predetermined time intervals or shorter. Since the second and subsequent flying patrol operations can be regarded as a coherent operation integrated with the first flying patrol operation with respect to time, the start coordinates of the second and subsequent flying patrol operations need not be matched to the start coordinates of the first flying patrol operation. Further, the start coordinates of the repeated flying patting operations are set to the start coordinates of the first flying patting operation. If the repeated flying patting operation is set in the partial image control section 207, the operation proceeds to block 631, while if it is not set, the operation proceeds to block 607.In block 607, the sub-image control section 207 recognizes the direction of the flying patting operation. The direction of the flying patting operation may be set in the direction of the XY coordinates with the start coordinates such as the start point. The partial image control section 207 may be predetermined, for the direction of the flying patting operation, in about four directions or eight directions with the start coordinates 306 as the start point to associate with corresponding directions the display method of the patting window 325 and the partial image 313 and the like.In block 609, the sub-image control section 207 determines the size and shape of the patrol 325 from motion information of the flying patrol operation. The size of the patting window 253 may be determined by default values or based on the speed or acceleration of the flying patting operation. For example, the partial image control section 207 may increase the size as the moving speed of the fingertip becomes faster.In this case, the user can adjust the size of the patting window 325 only by controlling the moving speed of the fingertip in the XY direction with which the flying patting operation is performed. Although a Z-axis component may be included in the movement of the fingertip, the field control section 207 may extract only the component in the XY direction. The user can more easily control the moving speed of the fingertip in the XY direction as compared with the control in the Z-axis direction.The shape of the window bar 325 may be a rectangle of vertical length, a rectangle of horizontal length, or a round shape. At this time, the partial image control section 207 may determine the shape of the slide window 325 from the arrangement of the operation objects around the start coordinates 306. For example, when the operation objects 301 a, 301 b, and 301 care arranged side by side in the X-axis direction of the original image 303, as illustrated in FIG. 5A, the image data generation section 207 may set a strip window 325 of horizontal length in the X-axis direction, as illustrated in FIGS. 7A and 7B.Further, as illustrated in FIG. 10, when the operation objects 381 of the original image 303 are arranged in the Y-axis direction, a patrol 325 may set a vertical length in the Y-axis direction to display the operation objects 383 corresponding to the operation objects 381 as the partial image 313 f. Further, when there are no operation objects around the start coordinates 306, a direct strip window such as a round shape or a square shape may be set.In block 611, the sub-image control section 207 determines the coordinates of the inert window 325. The coordinates of the swipe window 325 may determine that the center of the center of gravity of the swipe window 325 coincides with the starting coordinates 306. In such a case, if the part of the slide window 325 protrudes from the screen, the coordinates of the slide window 325 may be determined at a position where the operation objects 315 aof the slide window 325 and the operation objects 301 aof the original image 303 are as close to each other as possible, as illustrated in FIG. 7A, and that the entire area of the slide window 325 is included within the screen.In another example, the coordinates of the patrol 325 may be determined at a position where the patrol 325 is separated by a predetermined distance from the start coordinates 306 toward the flying patrol operation, as illustrated in FIG. 7B. For example, the field control section 207 can set the distance narrower as well as the speed or acceleration faster. In this case, the user can determine the position of the patting window 325 only by controlling the moving speed of the fingertip in the XY direction with which the flying patting operation is performed.In block 613, the field control section 207 determines the scale of magnification of the field 313 displayed in the inert window 325. The scale of the magnification may be determined by a specification or based on movement information from the flying patrol operation. For example, the partial image control section 207 may increase the scale of enlargement as the moving speed of the fingertip becomes faster. The user can adjust the magnification scale of the partial image 313 only by controlling the moving speed of the fingertip with which the flying patting operation is performed.In block 614, the sub-image control section 207 notifies the contact coordinate processing section 213 of the coordinates of the sub-image 313 of the striped window 325. Thereafter, the contact coordinate processing section 213 transforms the contact coordinates acquired from a touch operation on the partial image 313 into the coordinates of the original image 303, and transfers the coordinates to the user function 201. In block 615, the field control section 207 generates data on the field 313 to be displayed in the slide window 325 in terms of size, shape and coordinates, and transmits the data to the image data synthesizing section 205. In block 509 of FIG. 3, the sub-image data synthesis section 205 shows the sub-image 313 in a position of the inert window 325.In block 651, the sub-image control section 207 recognizes the direction of the flying patting operation. Processing for the direction of the flying patrol operation may follow block 607. For example, as illustrated in FIG. 8A, it is assumed that the field control section 207 associates the direction 321 and the direction 323 with the Y-axis direction with the direction 321, reduces generation of a field 313 aas illustrated in FIG. 8B, and associates with the direction 323 generation of a field 313 bwhich is enlarged as illustrated in FIG. 9A. Although the start coordinates 306 may be arranged in a position corresponding to an operation object or in a position not corresponding thereto, the start coordinates 320 in FIG. 8A are arranged in a position where no operation object of the original image 303 exists for the purpose of illustration.The partial image 313 aand the partial image 313 bare reduced / magnified images in a predetermined range of the original image 303 with the start coordinates 320 as the center. In block 653, the sub-image control section 207 determines the size and shape of the patrol window 325. The process at this time may follow the process in block 609. In block 655, the sub-image control section 207 defines the coordinates of the swipe window 325. The process at this time may follow the process in block 611.In block 657, when the direction of the flying patting operation with respect to the start coordinates 306 is in the direction 321, the sub-image control section 207 generates data on the reduced sub-image 313 aillustrated in FIG. 8B, while when it is the direction 323, the sub-image control section 207 generates data on the enlarged sub-image 313 billustrated in FIG. 9A. The process for determining the scales of the reduction and the enlargement of the partial images 313 aand 313 bmay follow the process in block 613.In block 631, the sub-image control section 207 determines whether a repeated flying patting operation is being performed. It is assumed that the sub-image control section 207 associates with the direction 321 the generation of the reduced sub-image data as shown in FIG. 8A, and associates with the direction 323 the generation of the enlarged sub-image data as shown in FIG. 9A.The sub-image control section 207 recognizes the direction of the first flying patting operation in block 633 and determines a reduction or an enlargement. Processing for the direction of the flying patting operation may follow block 607. The sub-image control portion 207 determines the size and shape of the patrol 325 from the first flying patrol operation at block 635 and determines the coordinates of the patrol 325 at block 637. The process at this time may follow the process of blocks 609 and 611.In block 639, the sub-image control section 207 detects subsequent flying patting operations. When the direction 323 is detected, the sub-image control section 207 generates image data for displaying, in a patrol 325 b, 325 c, a sub-image 313 d, 313 ethat is larger than a sub-image 313 c, 313 dof the preceding patrol 325 a, 325 bas illustrated in FIG. 9B each time the flying patrol operation performed within a predetermined period of time is detected, in block 641.When the direction 321 is recognized, the sub-image control section 207 generates image data for displaying, in the patrol window 325, a sub-image made smaller than the previous sub-image each time the flying patrol operation performed within a predetermined period of time is recognized. Note that the sizes of the striped windows 325a to 325c may be fixed to change only the scale of the field. If the repeated flying patting operation is set, the partial image control section 207 transfers data on a reduced or enlarged partial image at a magnification varied on a step by step basis to the image data synthesizing section 205 each time the user performs a flying patting operation, in block 615.While the present invention has been described with reference to the specific embodiment illustrated in the drawings, the present invention is not limited to the embodiment illustrated in the drawings. Needless to say, any configuration known heretofore may be employed as long as it exhibits the effects of the present invention.[Description of Symbols]100 Display 100 adisplay surface 200 user interface 301 ato 301 c, 381 original image operation object 303 original image 305 mouse cursor 306, 320 start coordinates 307 spot image 313, 313 ato 131 fsubimage 315 ato 315 coperable object of subimage 313 325, 325 ato 325 cmanipulation windows 351 ato 351 b, 353 ato 353 c, 355 ato 355 coperable object 381 original image operation object 303 383operable object of subimage 313

Claims

A user interface (200) that enables input with a pointing medium, comprising: a display (151); a proximity sensor (153) for generating proximity coordinates of the pointing medium in a proximity state to a surface of the display (151) with an original image (303) displayed thereon; a touch sensor (155) for generating contact coordinates of the pointing medium in a contact state with the surface of the display (151); a sub-image control section (207) for generating data for displaying a sub-image (313) on the display obtained by enlarging a predetermined area of the original image (303) in a manner to be overlapped on the original image (303) when a flying patrol operation performed with the pointing medium is recognized in the proximity state from the proximity coordinates, wherein the predetermined area of the original image (303) includes start coordinates of the flying patrol operation; and a contact coordinate processing section (213) for outputting the contact coordinates of the sub-image (313) as contact coordinates of the original image (303) to a user function.The user interface (200) of claim 1, wherein the proximity sensor (153) and the touch sensor (155) are comprised by a touch panel, and wherein a touch screen (150) comprises the display (151) and the touch panel.The user interface (200) according to claim 1, wherein the contact coordinate processing section (213) outputs, to the user function, contact coordinates corresponding to a patrol operation performed on the original image (303) with the pointing medium at the contact stage.The user interface (200) of claim 1, further comprising a pointing device (157) for generating an operation to control a mouse cursor displayed on the original image (303).The user interface (200) according to claim 1, wherein the sub-image control section (207) generates data of a spot image displayed on coordinates directly below the pointing medium in the proximity state.The user interface (200) according to claim 1, wherein the partial image control section (207) sends a process of changing a display of an operation object of the original image (303) when the pointing medium is disposed just above the operation object in the proximity state.The user interface (200) according to claim 1, wherein the sub-picture control section (207) stops generating data on the sub-picture (313) when a predetermined time has elapsed since the pointing medium is moved to a separation state after the sub-picture (313) is displayed.The user interface (200) according to claim 1, wherein the sub-image control section (207) generates data for displaying the sub-image (313) such that an operation object included in the sub-image (313) and corresponding to an operation object of the original image (303) is arranged around the operation object included in the original image (303).The user interface (200) according to claim 1, wherein the sub-image control section (207) determines a position of the sub-image (313) relative to the original image (303) corresponding to a moving direction from the start coordinates of the flying patting operation.The user interface (200) according to claim 1, wherein the sub-image control section (207) determines a magnification of the sub-image (313) relative to the original image (303) corresponding to a moving direction from the start coordinates of the flying patting operation.The user interface (200) according to claim 1, wherein the sub-image control section (207) determines a magnification of the sub-image (313) relative to the original image (303) according to a moving speed or a moving acceleration of the pointing medium with which the flying patting operation is performed.The user interface (200) according to claim 1, wherein the sub-image control section (207) of the sub-image (313) generates data on the sub-image (313) at a magnification that varies with respect to the original image (303) on a step-by-step basis each time a flying patrol operation is repeatedly performed at a predetermined time interval or shorter.The user interface (200) according to claim 1, wherein the sub-image control section (207) sets either both or only one of the shape and size of the sub-image (313) according to an arrangement of a plurality of operation objects included in the original image (303).An electronic device comprising a user interface (200) according to any one of claims 1 to 13.A method for a user interface comprising a display, comprising: - acquiring (503) coordinates of a pointing medium moved in an approach state to the display for displaying an original image; - acquiring (507) a flying patting operation of coordinates of the pointing medium in the approach state; - displaying (509) a sub-image obtained by partially enlarging a predetermined area of the original image to be overlapped on the original image in response to the acquiring of the flying patting operation, wherein the predetermined area of the original image includes start coordinates of the flying patting operation; and - processing (515) a touch operation on the sub-image as a touch operation on the original image.The method of claim 15, further comprising: - detecting a touch-strip operation performed on the display with the pointing medium moved in a contact state; and - performing (515) processing on the original image in response to detecting the touch-strip operation.The method of claim 15, further comprising: - performing (513) a touch operation on an operation object included in the original image without performing the flying swipe operation; and - performing (515) processing on the operation object without displaying the partial image.The method of claim 15, further comprising: - extracting motion information from the flying patrol operation; and - determining, based on the motion information, any one of a size, a shape, the coordinates, and a magnification of the sub-image relative to the original image.A computer program product that causes a user interface comprising a display to perform a method according to any of claims 15 to 18.

Images