System, method, and program therefor for realizing a user interface based on finger recognition

Abstract

Provided are a system, a method, and a program for realizing an intuitive and efficient user interface based on identification of each finger of a user. The present invention involves presenting actions by using one or both of an icon and a label which enable instant recognition, and displaying one or both the icon and the label which enable instant recognition at finger tips of a virtual hand representing fingers identified by a camera or the like, whereby a user can recognize a function to be executed. The present invention enables, by mode switching, multiple kinds of direct operations equal to or more than the number of fingers for an object to be operated.

Description

Technical Field

[0001] The present invention relates to a system, a method, and a program therefor for realizing a user interface based on finger identification.

Background Art

[0002] Various methods have been proposed for a user interface for a user to interact with a computer. One of such methods is realized by identifying each finger of a user's hand and assigning a function to each.

[0003] A basic example of such a technique is to assign different functions to each finger. However, if the functions are fixedly assigned, only 10 functions out of 10 fingers can be assigned. Since 10 functions are not sufficient in many applications, it is necessary to appropriately switch the set of functions to be assigned. In order to smoothly use the assigned functions, it is important to make the correspondence between the functions and the fingers easy to understand.

[0004] Patent Document 1 describes a technique of associating a TV function with a finger and associating a cancel function or a setting function of the TV function with another finger in a state where the finger is in contact with the touch screen 2 (FIG. 21 etc.).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In a user interface that identifies a user's finger and assigns functions to a plurality of identified fingers, further improvement is possible.

[0007] The present invention has been made in view of such points, and its problem is to provide a system, method, or program for providing a user interface to a user, and to realize a more intuitive and efficient user interface based on the identification of each finger of the user.

Means for Solving the Problem

[0008] A first aspect of the present invention is a method for providing a user interface to a user, the method including steps of identifying a plurality of fingers of the user, assigning a mode switching function to a first finger among the identified plurality of fingers, and switching functions assigned to a second finger and a third finger among the plurality of fingers to different functions respectively in response to a touch operation being performed by the first finger, wherein at least one of the second finger and the third finger is different from the first finger.

[0009] Also, a second aspect of the present invention is the method of the first aspect, wherein the touch operation is a touch operation on an object.

[0010] Also, a third aspect of the present invention is the method of the second aspect, wherein the different function is any one of a mode switching function, a parameter control function, and an object selection function.

[0011] Also, a fourth aspect of the present invention is the method of the second aspect, wherein the function assigned to the second finger after switching is a parameter control function, and the function assigned to the third finger after switching is a modification function of the parameter control function.

[0012] Also, a fifth aspect of the present invention is the method of the second aspect, further including a step of assigning a modification function of the parameter control function to a finger different from the second finger among the plurality of fingers in response to a touch operation being performed by the second finger, wherein the function assigned to the second finger after switching is a parameter control function.

[0013] Moreover, a sixth aspect of the present invention is the method according to the second aspect, wherein the function assigned to the second finger after switching is a command processing function.

[0014] Moreover, a seventh aspect of the present invention is the method according to the sixth aspect, wherein the command processing function is any one of an editing function, a conversion function, a search function, a saving function, a copying function, an arithmetic function, a transmitting function, and any combination thereof.

[0015] Moreover, an eighth aspect of the present invention is the method according to any one of the first to fourth aspects, wherein the second finger is the same finger as the first finger.

[0016] Moreover, a ninth aspect of the present invention is the method according to any one of the first to ninth aspects, further including a step of returning the functions of the second finger or the third finger to the functions before switching when the touch operation of the first finger is released.

[0017] Moreover, a tenth aspect of the present invention is the method according to any one of the first to eighth aspects, further including a step of returning the functions of the second finger or the third finger to the functions before switching when the touch operation of the first finger is released within a predetermined time.

[0018] Moreover, an eleventh aspect of the present invention is the method according to any one of the second to eighth aspects, wherein the selection state of the object is maintained even after the touch operation is released.

[0019] Moreover, a twelfth aspect of the present invention is the method according to the eleventh aspect, further including a step of switching the functions assigned to the second finger and the third finger to functions different from those before and during the touch operation, respectively, in response to the release of the touch operation.

[0020] Further, a 13th aspect of the present invention is the method according to the 11th or 12th aspect, further comprising: a step of ending the selection state in response to a touch operation by any one of the plurality of fingers; and a step of switching the functions assigned to the second finger and the third finger back to the functions before the switch in response to the selection state ending.

[0021] Further, a 14th aspect of the present invention is the method according to any one of the 1st to 13th aspects, further comprising a step of displaying, on a display used by the user, an icon or a label representing the function assigned to each finger together with a virtual hand representing the plurality of identified fingers.

[0022] Further, a 15th aspect of the present invention is the method according to the 14th aspect, wherein the label is displayed only when the finger associated with the label is located at a predetermined height or more above the desk used by the user.

[0023] Further, a 16th aspect of the present invention is the method according to the 14th aspect, wherein when the user's finger is outside the area of the display screen, the icon or the label is displayed staying at the edge of the screen.

[0024] Further, a 17th aspect of the present invention is the method according to any one of the 1st to 16th aspects, wherein the plurality of fingers of the user include any finger of the user's right hand and any finger of the user's left hand.

[0025] Further, an 18th aspect of the present invention is the method according to any one of the 1st to 17th aspects, wherein at least one of the first finger, the second finger, and the third finger is a set of fingers including a plurality of fingers.

[0026] Moreover, a 19th aspect of the present invention is a program for causing a computer to execute a method for providing a user interface to a user, the method including: identifying a plurality of fingers of the user; assigning a mode switching function to a first finger among the identified plurality of fingers; and switching the functions assigned to a second finger and a third finger among the plurality of fingers to different functions respectively in response to a touch operation by the first finger, wherein at least one of the second finger and the third finger is different from the first finger.

[0027] Moreover, a 20th aspect of the present invention is an apparatus for providing a user interface to a user for an application, the apparatus identifying a plurality of fingers of the user, assigning a mode switching function to a first finger among the identified plurality of fingers, switching the functions assigned to a second finger and a third finger among the plurality of fingers to different functions respectively in response to a touch operation by the first finger, wherein at least one of the second finger and the third finger is different from the first finger.

[0028] According to one aspect of the present invention, an intuitive and efficient user interface based on the identification of each finger of the user can be realized.

Brief Description of the Drawings

[0029]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0030] Embodiments of the present invention will be described below with reference to the drawings. All the drawings are illustrative. In this specification, a system, method, and program therefor related to an interaction with a computer (hereinafter also referred to as "finger-identification multi-touch interaction") realized by identifying a plurality of fingers of a user and performing a plurality of touch operations using the identified fingers are disclosed.

[0031] Figure 1 shows the overall configuration of a system for realizing finger identification multi-touch interaction according to an embodiment of the present invention. The images of one or both hands of the user (101) are captured by a camera (102) installed above the head or the like, and the positions of each finger are recognized by the processing of a control program (described later) operating on a computer (103), thereby realizing a user interface for an application. It is desirable to recognize the position of the finger not only in the horizontal direction but also in the vertical direction. In order for the user (101) to easily understand the recognition of the vertical position of the finger (especially the tap operation described later), it is desirable that the finger operation be performed on a horizontal plane such as a desk. Hereinafter, the user interface with the application will be described as the main example, but the present invention can also be applied to the user interface with the OS or middleware. Specifically, the "screen" mentioned below is not limited to the screen displayed by the application, but may also be the screen displayed by the OS or middleware.

[0032] As a guide for the user (101), it is desirable that a diagrammatic user's hand (hereinafter also referred to as "virtual hand") be displayed on the display (104). It is desirable that the virtual hand and the application screen be superimposed and displayed on the display (104) so that the user can directly perform operations such as graphic editing on the application screen. In order to improve the accuracy of the hand image recognition processing by the camera (102), a single-color sheet may be laid under both hands of the user (101). The color is preferably black. Also, a tactile feedback (haptics) device (not shown) may be placed on the desk to provide feedback for operations such as touch operations by the user. A device for feeding back the user's input by means of sound, light or other means may be provided. The display (104) and the operation surface may have an integrated structure.

[0033] Here, in order to improve the accuracy of recognizing the position of the finger, the camera (102) preferably is a camera equipped with a depth sensor. Also, it is desirable to capture a color image in order to improve the accuracy of finger recognition. Further, it is desirable that the input by the user (101) can be performed not only with one hand but also with both hands. The recognition of the finger operation of the user (101) may be realized by any recognition method such as a conventional touch screen, touch pad, smart glove incorporating a sensor, or a combination thereof, rather than by shooting with the camera (102). The function of the computer (103) may be realized by an external server such as a cloud. The computer (103) may be provided with other input devices (such as a mouse and a keyboard) that are used supplementarily, but are not shown in the figure.

[0034] FIG. 2 shows an example of a control program for realizing finger identification multi-touch interaction according to an embodiment of the present invention. The finger recognition unit (201) plays a role of recognizing the position of each finger of the hand of the user (101) by performing contour extraction processing on the image captured by the camera (102) and converting it into coordinate data. It is preferable to recognize not only the horizontal position but also the vertical position of the position. By recognizing the vertical position, it is also possible to perform determination processing of a tap operation or a hover operation described later. The virtual hand display unit (202) plays a role of displaying a virtual hand representing the state of each finger of the hand of the user (101) and related icons or labels on the display (104). The finger function assignment unit (203) plays a role of assigning functions such as appropriate command processing, temporary mode switching, target selection, and parameter control according to the situation at that time to each finger of the user (101), and also plays a role of changing the assignment of functions to the finger during a specific operation. The finger function processing unit (204) plays a role of executing the function assigned to the finger at that time when the user performs a touch operation. The command transmission unit (205) plays a role of transmitting a command for processing the function assigned to the finger that has performed the touch operation to the application. Here, "command processing" means processing specified by a command, and different commands are executed according to the type of processing.

[0035] Here, the "touch operation" is typically an operation of lowering a finger held above to the surface of the machine. However, it may also be contact with an object such as a touch screen other than the machine, a touch pad, or other parts of the body such as a leg or the other hand, and may be realized by any method (including these operations is also called the "touch operation"). Multiple fingers may perform a touch operation simultaneously as a set of fingers. It is also possible to detect that not only contact has occurred but also the finger has been pushed in with a certain amount of pressure or more and determine it as a touch operation.

[0036] Note that each part of the control program (200) does not necessarily correspond to one program module. Also, some or all of the functions may be realized by the cooperation of the control program (200) and an application or other programs. Examples of such programs include a web browser, JavaScript (registered trademark) operable on the web browser, and a native program capable of cooperating with the JavaScript. Also, some or all of the functions may be executed on a peripheral device such as a camera or an external computer such as the cloud. Further, it can be recorded on a computer-readable storage medium to form a non-transitory program product.

[0037] The finger identification multi-touch interaction according to an embodiment of the present invention may display an icon or label representing the function assigned to the finger at the fingertip of the virtual hand representing the finger identified by a camera or the like. Thereby, together with the temporary mode switching described later, it is possible to execute more than the number of fingers of various direct operations on the operation target. Also, it may be possible to switch the assignment of function groups by temporary mode switching and hierarchically select functions. In addition, the mode can also be switched by selecting the operation target, enabling function selection according to the context.

[0038] FIG. 3 shows an example of the correspondence between each finger on the virtual hand and a function in a slide creation application according to an embodiment of the present invention. The right thumb is assigned the function of "add object," and the right index finger is assigned the function of "select." The functions may include command processing, temporary mode switching, selection of an operation target, and parameter control. In the example of FIG. 3, the right index finger and the left index finger are assigned the function of "selection of an operation target," and the right thumb and the left thumb are assigned the function of "temporary mode switching." Specifically, the right thumb is assigned the function of "add object," and in response to a touch operation performed by the right thumb, the mode is temporarily switched and a different set of functions is assigned to each finger. The assignment of functions to each finger may be performed by a finger function assignment unit (203) of the control program (200). As an example, when the user (101) performs a touch operation with a certain finger, the function that was assigned to the corresponding finger on the virtual hand at that time is processed by the finger function processing unit (204). If the function is command processing in an application, the command information is sent to the application via the command sending unit (205) of the control program (200), and a predetermined process is performed.

[0039] Here, even if the fingertip is outside the screen area, if the fingertip is within the recognition range of fingertip position recognition means such as a camera (102), it is desirable to display an icon at the edge of the screen, allowing the user to see and touch it. If the fingertip is at the left or right screen edge, it is desirable for the ordinate position to move in conjunction with the corresponding fingertip, and if the fingertip is at the top or bottom screen edge, it is desirable for the abscissa position to move in conjunction with the corresponding fingertip. This allows the icon to be presented seamlessly when the finger goes outside or enters the screen, preventing the screen edge from being fixedly hidden by the icon. If the fingertip is within the fingertip position recognition means such as a camera (102), it is desirable to show the user that the finger is outside the recognition range by hiding the icon or label, or by changing the color or transparency.

[0040] Visual exploration may be facilitated by displaying the icons to be displayed together with the virtual hand in a larger size. In the case of a user interface such as a conventional toolbar, if the icons are large, they will fixedly occupy the screen space. However, in the finger identification multi-touch interaction according to the present embodiment, even if the icons are displayed relatively large, the icons can be moved only by moving the hand, so they do not fixedly occupy the screen. It is desirable that the icons can be set to be displayed larger for users with poor eyesight or beginners, and smaller for skilled users.

[0041] In order to indicate that an object can be selected, or to indicate the selection hot spot or area, it is desirable to display a cursor on one or more fingertips. This is because, in addition to the finger for object selection, there are fingers for command processing or mode switching, and it is necessary to clarify which finger can be used for object selection. When it is not the finger for object selection, it is desirable to display a specific cursor indicating that, or to display only the icon.

[0042] When the user is a beginner, it is desirable to display both the icons and the labels in order to search for functions by visual exploration or direct exploration. However, in the finger identification multi-touch interaction according to the present embodiment, it is possible to select without the need for visual recognition, and it is not always necessary to present the labels. Therefore, for skilled users, it is desirable to make the labels non-displayed and display them only when the finger is positioned at a certain height or more above the table, so that even a skilled user can quickly confirm if they forget.

[0043] (Temporary mode switching) Figures 4A and 4B show examples of object addition in a graphic editing application according to an embodiment of the present invention. When a touch operation is performed with a specific finger (corresponding to the "first finger"), a new set of functions can be assigned to a plurality of fingers (corresponding to the "second finger" and the "third finger"). A new function may be assigned to the specific finger that performs the touch operation. The specific finger may be one finger such as the index finger of the right hand, or a specific set of multiple fingers. The mode can be switched only while the touch operation by the specific finger is continued, and the mode can be restored to the original state when the touch operation is released. Figure 4A shows a state where no touch operation is performed with the right thumb, and Figure 4B shows a state where a touch operation is performed with the right thumb. In this example, by performing a touch operation with the right thumb to which a temporary mode switching function to the "Add object" mode is assigned, the set of functions assigned to other fingers is switched, and the object to be added can be determined. For example, while maintaining the touch operation with the right thumb, when a touch operation is performed with the index finger of the left hand, a command processing function of adding text is executed. When the touch operation of the right thumb is released, the mode returns to the original state, and the state shown in the upper part of Figure 4A is obtained. When performing an object addition operation, control may be performed to add an object at the position of the index finger of the dominant hand. The index finger of the dominant hand is often assigned a command processing function for operating an object such as intuitive movement for the user. Since the movement after object addition is a frequently performed pattern, the operation efficiency can be improved.

[0044] In the basic process of temporary mode switching, when the touch operation by the finger that activated the temporary mode is released, the mode returns to its original state, and the assignment of function groups to the fingers returns to the state before the switch. Alternatively, when a touch operation is performed for a short time, for example, for a period equal to or less than a predetermined time (tap operation), the mode may be permanently switched, and when the touch operation is performed for a period exceeding the predetermined time (long press operation), control may be performed to return to the state before the switch in response to the release of the touch operation. Although the functions assigned to multiple fingers have been mainly described as being switchable on a mode-by-mode basis, it is also conceivable to perform control to return, or not return, only for a part of the multiple fingers to the state before the switch when a predetermined condition is satisfied.

[0045] (Temporary mode switching involving object selection) FIG. 5 shows the correspondence between fingers and functions during object editing according to an embodiment of the present invention. In this example, a touch operation is performed by the left thumb to which the function of switching to the direct edit mode is assigned, and while this operation continues, command processing for image editing is assigned to other fingers. For example, Output ( Exposur e) Command processing and Contrast command processing is assigned to the right middle finger to touch the image, and exposure and contrast can be edited. This operation has the advantage that the selection function of the image to be edited and the execution of the command processing function can be performed in one operation. Here, when the touch operation by the left thumb is released, the mode may return to its original state, and control may be performed to return the assignment of function groups to the fingers to the state of FIG. 4A before the switch, but as in the previous paragraph, control to return the mode may be performed only under a predetermined condition.

[0046] Figures 6A and 6B show an example of graphic editing according to an embodiment of the present invention. In this example, a set of functions assigned to a finger is switched triggered by the selection of an object to be operated. Similar to the above-described temporary mode switching, the mode is temporarily switched only while the touch operation is continued, and control may be performed so that the mode returns to the original state when the touch operation is released, or the mode may be returned only under a predetermined condition. Figure 6A shows a state where no touch operation is performed on the object, and Figure 6B shows a state where a touch operation is performed on the object with the index finger of the right hand and the object is selected. The assignment of functions to each finger can be switched only while the touch operation is maintained, and operations on the graphic can be performed.

[0047] Also, even if the touch operation is released, the selection state of the object to be operated may be continued. In this case, control may be performed so that the mode is not returned even if the touch operation is released. For example, the selection state of the object can be terminated when a touch operation is performed again on the object or when the touch operation is released, and any other operation may be used as an operation to terminate the selection state. Then, the mode may be returned according to the termination of the selection state. Alternatively, when the touch operation is released while the object is selected, control may be performed to assign a different function group to the finger than before the selection of the object. The fact that the object is selected means that there is a high possibility that operations on the object will continue. Further, when the touch operation is released while the object is selected, control may be performed to assign a different function group to the finger than before and during the touch operation on the object.

[0048] It is possible to make a plurality of objects selectable and switch the function groups assigned to each finger according to the types of the selected plurality of objects. In this case, the switching of the function groups can be performed, for example, when a plurality of objects are selected, when an additional object is selected while a plurality of objects are already selected, when the selection state of any one of the objects ends while a plurality of objects are selected, when a touch operation is performed on any one of the objects while a plurality of objects are selected, and the like.

[0049] In the object selection state, while continuously performing a touch operation of a specific finger (for example, the right thumb) as needed, control may be performed such that when a touch operation is performed on another object with the finger (for example, the right index finger) to which the selection function is assigned, it becomes an additional selection. Alternatively, in the object selection state, while continuously performing a touch operation of a specific finger as needed, when the drag operation of the finger used to select the object is performed to an area where another object exists, the other object may be additionally selected. A multiple selection function may be assigned to a finger, and a touch operation may be performed multiple times with the finger, and a plurality of objects existing within a rectangular, polygonal, circular, or elliptical area connecting the points where those touch operations are performed may be selected. Also, a multiple selection function may be assigned to a finger, and a drag operation may be performed with the finger, and a plurality of objects existing within the area surrounded by the trajectory may be selected. By continuing the touch operation of a specific finger, the function of the finger to which the selection function is assigned may be switched to a multiple selection function.

[0050] Here, a hover operation may be used as a trigger for object selection instead of a touch operation. That is, in the example of FIG. 6B, for example, when the coordinates of the right index finger are positioned on the object to be operated, in other words, just by hovering, a temporary mode switch can be made. Then, when a touch operation is performed with a finger other than the right index finger, the function assigned to that finger may be executed. In determining whether a hover operation has been performed, it may be determined that a hover operation is in progress when a predetermined finger is at or below a predetermined height. Typically, this is when the distance between the desk surface and the finger is at or below a predetermined value. Alternatively, a hover operation may be determined based on the difference in height from other fingers, and it may be regarded as a hover when it is at a position lower than a predetermined distance from other fingers. In this case, as the height of the other fingers, the height of the lowest finger among the fingers other than the finger performing the hover operation, the average value of the heights of a plurality of fingers other than the finger performing the hover layer, the height of the finger adjacent to the finger performing the hover operation, etc. may be used. By taking into account the height of the finger in the determination of the hover operation in this way, it is possible to avoid the problem that the screen is frequently rewritten due to mode switching.

[0051] Figure 7 shows the process of color change by parameter control according to an embodiment of the present invention. Although the illustrated example is monochrome, in reality, the circles arranged in an annular shape are each displayed in a different color. Figure 7 shows a process 700 of controlling parameters by sliding a finger while maintaining a touch operation. First, perform a touch operation on the graphic with the right middle finger to which the color change function is assigned. Then, a color change menu with a plurality of color options arranged in an annular shape is displayed. And while maintaining the touch operation with the right middle finger, the color of the graphic can be changed by selecting one of the color options. When the touch operation of the right middle finger is released, the function assignment to the finger may be restored. Generally, with a touch operation of one finger, 2-degree-of-freedom control in the up, down, left, and right directions is possible. While maintaining the touch operation of the finger that was first touched, parameter control may also be assigned to other fingers so that other parameters can be controlled. Depending on the function, it may be possible to control parameters with two or more fingers. Thus, theoretically, a total of 20 degrees of freedom of control is possible with 10 fingers on both hands. The selection of options from the displayed menu may also be possible with a finger different from the finger that performed the touch operation on the graphic. In addition to color change, the parameter control shown in Figure 7 can also be applied to brushes, fonts, etc., and an operation of selecting one from a large number of options can be efficiently performed.

[0052] FIG. 8 shows a diagram for explaining the rotation of a figure with parameter control according to an embodiment of the present invention. In the example of FIG. 8, in the editing function of rotating a figure, it is an example of restricting the parameter of the rotation angle to units of 45 degrees. The figure can be selected by the right index finger to which the target selection function is assigned, and the rotation operation of the figure can be performed by the right thumb to which the parameter control function with the rotation function and the rotation angle as parameters is assigned. Here, a parameter restriction function for restricting the rotation angle to units of 45 degrees is assigned to the right ring finger, and by performing a touch operation with the ring finger, the rotation angle determined by the drag operation of the right thumb can be set to units of 45 degrees. The rotation function assigned to the right thumb is executed at predetermined time intervals while the touch operation of the right thumb continues, and the rotation angle can be determined according to the drag operation.

[0053] More generally, when the parameter control function is assigned to a specific finger, a function for changing the parameter control function is assigned to another finger, and while the touch operation by the other finger continues, at least any one of the type of parameter determined by the operation of the specific finger, the number thereof, and the increment / decrement unit of the value of each parameter can be changed. An example of changing the number of parameters is changing from a two-dimensional rotation angle to a three-dimensional rotation angle. The assignment of the change function to the other finger may be performed at the time of assigning the parameter control function to the specific finger, or alternatively, it may be performed in response to the touch operation by the specific finger.

[0054] Figure 9 shows the alignment of a figure by parameter control according to an embodiment of the present invention. Although many functions can be assigned to fingers by mode switching, it is considered that the deeper the hierarchy, the more complex it becomes and the longer the function selection time becomes. Therefore, by making a plurality of groupable functions selectable from a context menu, complication can be avoided. Figure 9 is an example of the Align command process, and a change 900 from the state on the left side where no touch operation is performed to the state on the right side where a menu is displayed by a touch operation is shown. In this example, if the finger is released as it is, it will be aligned to the horizontal center, and in addition, the upper, lower, left, right, and vertical centers can be selected.

[0055] In the above description, examples of object editing have been mainly given, but the command process may be a process for an object other than editing. Examples of such processes include conversion such as translation of selected text, web search based on the selected text, saving such as writing to a file of the selected object, copying of the selected object, calculation based on the selected text, transmission of the selected object, etc., and any combination thereof is also possible. For example, when an object is selected and moved with a specific finger (for example, the right index finger), that is, when an object is selected with the finger to which the selection function is assigned and the movement operation function is assigned to that finger, instead of the editing process of simply moving the object by moving after a touch operation with another finger (for example, the right ring finger), combined control such as copying and then moving may be performed.

Claims

1. A method for providing a user interface to a user, comprising: identifying a plurality of fingers of the user; assigning a mode switching function or a mode switching function and an object selection function to a first finger among the identified plurality of fingers; switching the functions assigned to a second finger and a third finger among the plurality of fingers to different functions respectively in response to a touch operation or a hover operation by the first finger; wherein at least one of the second finger and the third finger is different from the first finger; when the mode switching function is assigned to the first finger and the object selection function is not assigned, further comprising switching the function of the second finger or the third finger back to the function before switching in response to the release of the touch operation or the hover operation of the first finger.

2. The method according to claim 1, wherein when the mode switching function and the object selection function are assigned to the first finger, the selected state of the object is maintained even after the touch operation or the hover operation on the object is released.

3. The method according to claim 1 or 2, wherein one or more functions assigned to the second finger include at least any one of a command processing function, a parameter control function, and the object selection function.

4. The method according to claim 3, wherein one or more functions assigned to the second finger after switching include a parameter control function; the function assigned to the third finger after switching includes a function for changing the parameter control function.

5. The method according to claim 3, wherein one or more functions assigned to the second finger after switching include a parameter control function; further comprising assigning a function for changing the parameter control function to a finger different from the first finger and the second finger among the plurality of fingers in response to a touch operation or a hover operation by the second finger.

6. The method according to claim 3, wherein the function assigned to the second finger after switching includes any one of an editing function, a search function, a save function, a copy function, a calculation function, a transmission function, and any combination thereof.

7. The method according to claim 6, wherein The editing function includes any one of movement, alignment, rotation, image editing, and character addition.

8. The method according to claim 3, wherein the second finger is the same finger as the first finger.

9. The method according to claim 1, wherein when a mode switching function is assigned to the first finger and an object selection function is not assigned, in response to the touch operation or hover operation of the first finger being released within or less than a predetermined time, the method further includes a step of returning the functions of the second finger or the third finger to the functions before the switching.

10. The method according to claim 1, further including a step of displaying, on a display used by the user, an icon or label representing the function assigned to each finger together with a virtual hand representing the plurality of identified fingers.

11. The method according to claim 10, wherein the label is displayed only when the finger associated with the label is located at a height equal to or higher than a predetermined height from the desk used by the user.

12. The method according to claim 10, wherein when the user's finger is outside the area of the display screen, the icon or the label is displayed staying at the edge of the screen.

13. A program for causing a computer to execute a method for providing a user interface to a user, the method including: identifying a plurality of fingers of the user; assigning a mode switching function or a mode switching function and an object selection function to a first finger among the plurality of identified fingers; switching the functions assigned to a second finger and a third finger among the plurality of fingers to different functions respectively in response to a touch operation or a hover operation being performed by the first finger; and including wherein at least one of the second finger and the third finger is different from the first finger, and when a mode switching function is assigned to the first finger and an object selection function is not assigned, the method further includes a step of returning the functions of the second finger or the third finger to the functions before the switching when the touch operation or hover operation of the first finger is released.

14. An apparatus for providing a user interface to a user for an application, identifying a plurality of fingers of the user, Assign a mode switching function or a mode switching function and an object selection function to the first finger among the identified plurality of fingers, and according to a touch operation or a hover operation performed by the first finger, the functions assigned to the second finger and the third finger among the plurality of fingers are respectively switched to different functions, at least one of the second finger and the third finger is different from the first finger, when the mode switching function is assigned to the first finger and the object selection function is not assigned, when the touch operation or the hover operation of the first finger is released, the function of the second finger or the third finger is returned to the function before switching.

Images