Input device, input method, and input program

Abstract

To provide an input device capable of preventing an input position from changing to a direction not intended by a user due to change of a gesture operation of the user in an input device which performs an input process to the input position on a display screen by detecting a gesture operation of the user, and an input method and input program thereof.SOLUTION: An input device includes: an operation position determination unit for determining an operation position of a gesture operation of a user; a motion vector calculation unit for calculating a motion vector of the input position on the basis of an amount of motion of the operation position when the operation position has moved; an input processing unit which performs a first input process at the input position when a first gesture operation has detected, and performs a second input process at the input position when a second gesture operation has detected; and a motion vector correction unit which corrects the motion vector when a change from the first gesture operation to the second gesture operation is determined.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an input device, an input method, and an input program for receiving an input by a user's gesture operation on a display screen. [Background technology]

[0002] Conventionally, there is known an input device that allows input (screen operation) by gesture operation on the display screen of a display panel. For example, Patent Document 1 discloses an interface device in which, when a user holds his / her hand in a one-fingered position toward the device, an icon of the number "1" is displayed on the display and the display of the first menu, which is the television, is highlighted, and, when the user holds his / her hand in a two-fingered position, an icon of the number "2" is displayed on the display and the display of the second menu, which is the network, is highlighted, and, by maintaining the same hand position for a certain period of time, the first menu or the second menu is selected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-185456 Summary of the Invention [Problem to be solved by the invention]

[0004] Here, the input device specifies a position (input position) on the display screen designated by, for example, a gesture operation with the user's hand based on the position of the hand (operation position). For example, the input device specifies the positions of the user's finger joints or the center (center of gravity) of the entire area onto which the hand is projected during the gesture operation as the input position. However, this method has the problem that, for example, when the user changes from an operation of opening the hand (first gesture operation) to an operation of closing (grinding) the hand (second gesture operation), the input position changes in a direction unintended by the user.

[0005] An object of the present invention is to provide an input device, an input method, and an input program that are capable of preventing the input position from changing in an unintended direction due to a change in the user's gesture operation in an input device that detects a user's gesture operation and performs input processing at an input position on a display screen. [Means for solving the problem]

[0006] An input device according to one aspect of the present invention is an input device that detects a gesture operation by a user and performs input processing at an input position on a display screen, and includes a gesture operation detection unit that detects the gesture operation by the user, an operation position determination unit that determines an operation position of the gesture operation detected by the gesture operation detection unit, a movement vector calculation unit that calculates a movement vector consisting of a movement direction and a movement amount for moving the input position based on the movement amount of the operation position when the operation position moves, an input processing unit that executes a first input process at the input position when the gesture operation detection unit detects a first gesture operation of the user, and executes a second input process at the input position when the gesture operation detection unit detects a second gesture operation of the user, a gesture operation change determination unit that determines a change from the first gesture operation to the second gesture operation, and a movement vector correction unit that corrects the movement vector when the gesture operation change determination unit determines a change from the first gesture operation to the second gesture operation.

[0007] An input method according to another aspect of the present invention is an input method for detecting a gesture operation of a user and performing input processing at an input position on a display screen, in which one or more processors execute the following steps: a gesture operation detection step for detecting the gesture operation of the user; an operation position determination step for determining an operation position of the gesture operation detected in the gesture operation detection step; a movement vector calculation step for calculating a movement vector consisting of a movement direction and a movement amount for moving the input position based on the movement amount of the operation position when the operation position moves; an input step for executing a first input process at the input position when a first gesture operation of the user is detected in the gesture operation detection step, and executing a second input process at the input position when a second gesture operation of the user is detected in the gesture operation detection step; a gesture operation change determination step for determining a change from the first gesture operation to the second gesture operation; and a movement vector correction step for correcting the movement vector when a change from the first gesture operation to the second gesture operation is determined in the gesture operation change determination step.

[0008] an input program according to another aspect of the present invention is an input program that detects a gesture operation of a user and performs input processing at an input position on a display screen, the program causing one or more processors to execute the following steps: a gesture operation detection step of detecting the gesture operation of the user; an operation position determination step of determining an operation position of the gesture operation detected in the gesture operation detection step; a movement vector calculation step of calculating a movement vector consisting of a movement direction and a movement amount for moving the input position based on the movement amount of the operation position when the operation position moves; an input step of executing a first input process at the input position when a first gesture operation of the user is detected in the gesture operation detection step, and executing a second input process at the input position when a second gesture operation of the user is detected in the gesture operation detection step; a gesture operation change determination step of determining a change from the first gesture operation to the second gesture operation; and a movement vector correction step of correcting the movement vector when a change from the first gesture operation to the second gesture operation is determined in the gesture operation change determination step. Effect of the Invention

[0009] According to the present invention, it is possible to provide an input device, an input method, and an input program that are capable of preventing the input position from changing in an unintended direction due to a change in the user's gesture operation in an input device that detects a user's gesture operation and performs input processing at an input position on a display screen. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of a display device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a virtual operation screen in the display device according to the embodiment of the present invention. [Diagram 3]FIG. 3 is a diagram showing an example of an input operation in a conventional display device. [Figure 4A] FIG. 4A is a diagram showing an example of an operation position for an input operation in a conventional display device. [Figure 4B] FIG. 4B is a diagram showing an example of an operation position for an input operation in a conventional display device. [Diagram 5] FIG. 5 is a diagram showing an example of an input operation in a conventional display device. [Figure 6] FIG. 6 is a diagram showing changes in the display form of icons for gesture operations and input operations on the display device according to the embodiment of the present invention. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure of a display control process executed by the display device according to the embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 15] FIG. 15 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. [Figure 16]FIG. 16 is a diagram showing an example of a method for setting a virtual operation surface in the display device according to the embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the following embodiment is an example of the present invention, and does not limit the technical scope of the present invention.

[0012] As shown in Fig. 1, a display device 1 according to an embodiment of the present invention includes a control unit 11, a storage unit 12, a display panel 13, an operation unit 14, and a motion sensor 15. Fig. 2 shows a schematic diagram of the display device 1. The motion sensor 15 is installed above the display panel 13, and detects an input operation by a user.

[0013] The display device 1 accepts a non-contact input operation (e.g., a gesture operation) by the user on the display screen 13A. For example, when the display device 1 detects the user's input operation on the virtual operation surface R2, it executes an input process according to the user's input operation on the display screen 13A. For example, when the user touches a predetermined position on the virtual operation surface R2, the display device 1 detects a position (input position) on the display screen 13A corresponding to the touch position (operation position) on the virtual operation surface R2 and accepts the touch input. In addition, the display device 1 detects the user's gesture operation and performs input processing on the input position on the display screen 13A. The display device 1 is an example of an input device of the present invention. A specific configuration of the display device 1 will be described below.

[0014] The motion sensor 15 includes, for example, two cameras and three infrared LEDs, and detects the user's input operation within a predetermined detection range. The motion sensor 15 outputs detection information to the control unit 11. The detection information includes position coordinates (X coordinate, Y coordinate, Z coordinate) of a detection target (for example, a user's hand, fingertip, stylus pen, support rod, etc.) based on the motion sensor 15. The motion sensor 15 can detect, for example, the back (palm) of the user's hand (right hand RH, left hand LH), finger joints, finger bone structure, angle, position, etc. The motion sensor 15 can apply well-known technology. The motion sensor 15 and the camera are an example of an imaging unit of the present invention.

[0015] The display panel 13 is a display that displays images, such as a liquid crystal display. The operation unit 14 is an operation device such as a mouse, a keyboard, etc. The operation unit 14 may be configured as a touch panel.

[0016] The storage unit 12 is a non-volatile storage unit such as a hard disk drive (HDD) or a solid state drive (SSD) that stores various information. Specifically, the storage unit 12 stores data such as operation area information D1 and virtual operation surface information D2.

[0017] The operation area information D1 is information indicating the operation area R1 on the display screen 13A of the display panel 13. The operation area R1 is an area of ​​the display screen 13A where the user can perform an input operation via the virtual operation surface R2, that is, an area where the user's input operation can be received. The operation area R1 may be set in the entire area of ​​the display screen 13A, or may be set in a part of the area of ​​the display screen 13A. For example, when the entire area of ​​the display screen 13A is set as the operation area R1, the operation area information D1 includes information on the coordinates C11 to C14 (see FIG. 2) of the four corners of the display screen 13A as coordinate information defining the operation area R1. The operation area information D1 is registered in the storage unit 12 every time the operation area R1 is set or updated.

[0018] The virtual operation surface information D2 is information indicating an area of ​​the virtual operation surface R2 that accepts the user's input operation on the display screen 13A. Specifically, the virtual operation surface R2 corresponds to the operation area R1, and coordinates C21 to C24 (see FIG. 2) of the four corners that define the virtual operation surface R2 correspond to coordinates C11 to C14 that define the operation area R1. The virtual operation surface information D2 includes information on the coordinates C21 to C24 of the four corners that define the virtual operation surface R2. The virtual operation surface information D2 is registered in the storage unit 12 every time the virtual operation surface R2 is set or updated. The size and position of the virtual operation surface R2 may be set in advance or may be set by the user's operation.

[0019] The storage unit 12 also stores control programs such as a display control program for causing the control unit 11 to execute a display control process (see FIG. 7) described below. For example, the display control program is non-temporarily recorded on a computer-readable recording medium such as a CD or a DVD, read by a reading device (not shown) such as a CD drive or a DVD drive provided in the display device 1, and stored in the storage unit 12. The display control program may be distributed from a cloud server and stored in the storage unit 12.

[0020] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 11 controls the display device 1 by executing, by the CPU, various control programs that are stored in advance in the ROM or the storage unit 12.

[0021] Meanwhile, in conventional technology, for example, a position (input position) on the display screen 13A designated by a gesture operation with the user's hand is identified based on the position of the hand (operation position). For example, conventional display devices identify the positions of the user's finger joints or the center (center of gravity) of the entire area onto which the hand is projected during the gesture operation as the input position. However, this method has a problem in that, for example, when the user changes from an operation of opening the hand (first gesture operation) to an operation of closing the hand (second gesture operation), the input position changes in a direction unintended by the user. A specific example of this problem will be described below.

[0022] 3 to 5 are diagrams for explaining the problems caused by the conventional technology. For example, when the product order screen shown in FIG. 3 displays the selection button images of "Order", "Back", and "Other" as images to be selected, if the user holds the palm of his / her right hand RH over the display screen 13A on the virtual operation surface R2 (first gesture operation) and moves it in the up, down, left, right directions (X direction and Y direction), the input operation icon M1 on the display screen 13A moves in response to the movement operation. When the user wants to order a desired product, the user closes his / her right hand RH with the input operation icon M1 overlapping the selection button image of "Order" (second gesture operation) in order to press the selection button image of "Order" displayed on the display screen 13A. This allows the user to perform an operation to select (order) the desired product.

[0023] Here, the conventional display device specifies the center (center of gravity) position P0 (operation position) of the entire area F1 onto which the user's hand (open hand) is projected as the input position, as shown in Fig. 4A. Therefore, when the user closes his / her hand, the operation position moves from the center position P0 of the entire area F1 to the center position P2 of the entire area F2, as shown in Fig. 4B. Therefore, the input position on the display screen 13A also changes according to the movement of the operation position. Note that a movement vector consisting of the movement amount and movement direction of the input position is calculated based on the movement amount of the operation position (the movement amount from center position P0 to center position P1).

[0024] For this reason, for example, as shown in FIG. 5, when a user moves the palm of his / her right hand RH in the up, down, left and right directions (X and Y directions) on the virtual operation surface R2 while keeping the palm open, and closes the right hand RH while the input operation icon M1 overlaps with the "order" selection button image, the input position may move downward and the operation of pressing the "back" selection button image may be accepted. Thus, in the conventional technology, a problem occurs in which the input position changes in a direction unintended by the user as the user's gesture operation changes. In contrast, in the display device 1 according to the present embodiment, it is possible to prevent the input position from changing in a direction unintended by the user as the user's gesture operation changes, as shown below.

[0025] Specifically, as shown in FIG. 1, the control unit 11 includes various processing units such as an operation surface setting unit 111, an input operation detection unit 112, an operation position determination unit 113, a display processing unit 114, a movement vector calculation unit 115, an input processing unit 116, a gesture operation change determination unit 117, and a movement vector correction unit 118. The control unit 11 functions as the operation surface setting unit 111, the input operation detection unit 112, the operation position determination unit 113, the display processing unit 114, the movement vector calculation unit 115, the input processing unit 116, the gesture operation change determination unit 117, and the movement vector correction unit 118 by executing various processes according to the display control program and by the CPU. In addition, some or all of the processing units included in the control unit 11 may be configured with electronic circuits. In addition, the display control program may be a program for causing a plurality of processors to function as the various processing units.

[0026] The operation surface setting unit 111 sets a virtual operation surface R2 that accepts input operations from the user. For example, as shown in FIG. 2, the operation surface setting unit 111 sets the virtual operation surface R2 at a position a predetermined distance away from the display screen 13A in the Z direction. The operation surface setting unit 111 may set the virtual operation surface R2 of a preset size at a preset position, or may set the virtual operation surface R2 of a size according to the user's operation at a position specified by the user. Details will be described later (see [Virtual Operation Surface Setting Method]), but for example, the user can set the virtual operation surface R2 of a desired size at a desired position by performing a predetermined gesture operation.

[0027] The input operation detection unit 112 detects input operations including a predetermined gesture operation by the user. Specifically, the input operation detection unit 112 detects the input operation by the user on the virtual operation surface R2 set by the operation surface setting unit 111. For example, the input operation detection unit 112 detects detection coordinates (operation position) on the virtual operation surface R2 based on detection information acquired from the motion sensor 15, and calculates input coordinates (input position) in the operation area R1 from the detection coordinates. The input operations include a gesture operation for executing a predetermined input process, a drawing operation for drawing a handwritten image on the display screen 13A, and the like. The input operation detection unit 112 is an example of a gesture operation detection unit of the present invention.

[0028] Here, when the ratio of the virtual operation surface R2 to the operation area R1 is "W2:W1=H2:H1=a:b" (see FIG. 2), the input operation detection unit 112 can calculate the input coordinates [dx, dy] based on the detected coordinates [sx, sy] on the virtual operation surface R2 by the formulas dx=sx×b / a, dy=sy×b / a. Note that the display resolution [rx, ry] is Min[dx, dy]=[0,0], Max[dx, dy]=[dx, dy].

[0029] Here, a specific example in which the input operation detection unit 112 detects the gesture operation will be described.

[0030] For example, the user performs a first gesture operation when he / she wishes to move an input operation icon M1 (e.g., a mouse cursor image) displayed on the display screen 13A. The first gesture operation is, for example, an operation in which the user opens his / her right hand RH in front of the display screen 13A (e.g., an operation in which the right hand RH is made into a "paper" shape). Also, for example, the user performs a second gesture operation when he / she wishes to select (click) a selection target with the input operation icon M1 displayed on the display screen 13A. The second gesture operation is, for example, an operation in which the user closes his / her right hand RH in front of the display screen 13A (e.g., an operation in which the right hand RH is made into a "rock" shape). The input operation detection unit 112 detects the first gesture operation and the second gesture operation.

[0031] The combination of the first gesture operation and the second gesture operation is not limited to the above-mentioned example. For example, the first gesture operation may be a pose of raising the index finger of the right hand RH, and the second gesture operation may be a pose of closing the right hand RH. The first gesture operation may be a pose of opening the right hand RH and holding it vertically, and the second gesture operation may be a pose of opening the right hand RH and tilting it 90 degrees. The first gesture operation may be a pose of opening the right hand RH, and the second gesture operation may be a pose of raising the thumb of the right hand RH or a pose of an OK sign with the right hand RH.

[0032] Furthermore, the input operation detection unit 112 detects a third gesture operation that is similar to (is similar to) the first gesture operation and the second gesture operation. The third gesture operation is, for example, a gesture operation in the middle of changing from the first gesture operation to the second gesture operation, for example, a gesture of the right hand RH that is in a shape between a "paper" shape and a "rock" shape.

[0033] The input operation detection unit 112 detects the first gesture operation, the second gesture operation, and the third gesture operation based on at least one of hand and finger information of the skeleton, angle, and position of the user's fingers. The input operation detection unit 112 also detects the first gesture operation, the second gesture operation, and the third gesture operation based on a hand grip degree Gn (described later) according to the hand and finger information. For example, the input operation detection unit 112 sets in advance a first threshold value of the grip degree Gn corresponding to the first gesture operation, and detects the first gesture operation when the detected grip degree Gn is less than the first threshold value. For example, the input operation detection unit 112 sets in advance a second threshold value (where the first threshold value<the second threshold value) of the grip degree Gn corresponding to the second gesture operation, and detects the second gesture operation when the detected grip degree Gn is equal to or greater than the second threshold value. The input operation detection unit 112 also detects the third gesture operation when the detected grip degree Gn is equal to or greater than the first threshold value and less than the second threshold value.

[0034] The storage unit 12 stores in advance setting information that associates predetermined gesture operations with operation contents corresponding to the gesture operations. For example, the first gesture operation and the third gesture operation are associated with a moving operation of an input operation icon M1, and the second gesture operation is associated with a selection operation (click operation) using the input operation icon M1. The predetermined gesture operations may include a gesture operation for setting a virtual operation surface R2 (see "Virtual Operation Surface Setting Method" described later). The predetermined gesture operations may be set individually for each user who uses the display device 1.

[0035] The operation position determination unit 113 determines the operation position on the virtual operation surface R2 of the gesture operation detected by the input operation detection unit 112. Specifically, the operation position determination unit 113 determines a reference point of the gesture operation based on a captured image of the gesture operation captured by the motion sensor 15, and determines the operation position based on the position of the reference point. In addition, the operation position determination unit 113 determines the center of gravity position of the captured image of the gesture operation captured by the motion sensor 15 as the reference point. For example, the operation position determination unit 113 creates first finger information of at least one of the skeleton, angle, and position of the user's fingers based on the captured image of the gesture operation captured by the motion sensor 15, and determines the position of the reference point based on the first finger information. The center position P0 shown in FIG. 4A and the center position P1 shown in FIG. 4B are examples of the reference point.

[0036] The display processing unit 114 displays various information on the display screen 13A of the display panel 13. For example, the display processing unit 114 displays an image of an input operation icon M1 (mouse cursor image), images of selection items that the user can select by operating the input operation icon M1 (for example, the selection button images of "Order", "Back", and "Other" in FIG. 3), and the like on the display screen 13A. In addition, the display processing unit 114 displays the input operation icon M1 at the input position corresponding to the operation position determined by the operation position determination unit 113.

[0037] The display processing unit 114 may display the input operation icon M1 in different display modes depending on the type of gesture operation. For example, when the input operation detection unit 112 detects the first gesture operation of the user, the display processing unit 114 displays the input operation icon M1 on the display screen 13A in a first display mode. Also, when the input operation detection unit 112 detects the second gesture operation of the user, the display processing unit 114 displays the input operation icon M1 on the display screen 13A in a second display mode. Specifically, as shown in FIG. 6, when the input operation detection unit 112 detects the first gesture operation of the user (a "paper" pose), the display processing unit 114 displays the input operation icon M1 in a first size (size) and a first color (white). On the other hand, when the input operation detection unit 112 detects the second gesture operation of the user (a "rock" pose), the display processing unit 114 displays the input operation icon M1 in a second size smaller than the first size and in a second color (black).

[0038] When the operation position of the user's gesture operation moves, the movement vector calculation unit 115 calculates a movement vector consisting of a movement direction and a movement amount for moving the input position based on the movement amount of the operation position. For example, as shown in Figures 4A and 4B, when the user changes the gesture operation from an open hand to a closed hand, the movement vector calculation unit 115 calculates the movement direction and movement amount (movement vector) of the input position based on the movement direction and movement amount of the reference point (center position P0, P1) corresponding to the operation position. The movement vector calculation unit 115 converts the movement vector of the operation position into the movement vector of the input position based on, for example, the ratio between the virtual operation surface R2 and the operation area R1 (see Figure 2).

[0039] The input processing unit 116 executes an input process at the input position according to the user's input operation. When the input operation detection unit 112 detects a first gesture operation by the user, the input processing unit 116 executes a first input process at the input position, and when the input operation detection unit 112 detects a second gesture operation by the user, the input processing unit 116 executes a second input process at the input position. For example, when the input operation detection unit 112 detects a moving operation that moves the input operation icon M1 according to the first gesture operation, the input processing unit 116 moves the input operation icon M1 on the display screen 13A. Also, for example, when the input operation detection unit 112 detects a selection operation that selects a selection target by the input operation icon M1, the input processing unit 116 executes a process according to the selection target on the display screen 13A. For example, when the input operation detection unit 112 detects an operation to select the selection button image of "order" on the display screen 13A, the input processing unit 116 executes an order process assigned to the selection button image of "order". Furthermore, when the input operation detection unit 112 detects the drawing operation, the input processing unit 116 executes drawing processing for drawing a handwritten image on the display screen 13 A. The input processing unit 116 is an example of the input processing unit of the present invention.

[0040] The gesture operation change determination unit 117 determines a change from the first gesture operation to the second gesture operation. For example, the gesture operation change determination unit 117 determines a change from an open hand to a closed hand of the user.

[0041] Specifically, the gesture operation change determination unit 117 calculates the degree of approximation between the user's gesture operation (e.g., the third gesture operation) detected by the input operation detection unit 112 and each of the first gesture operation (with the hand open) and the second gesture operation (with the hand closed), and determines a change from the first gesture operation to the second gesture operation based on the degree of approximation. Also, the gesture operation change determination unit 117 creates second hand and finger information of at least any of the skeleton, angle, and position of the user's fingers based on an image of the user's gesture operation (the third gesture operation) captured by the motion sensor 15, and determines the degree of approximation based on the second hand and finger information. For example, the gesture operation change determination unit 117 determines that the first gesture operation has changed to the second gesture operation when the amount of change over time in the degree of approximation is greater than a predetermined first reference value.

[0042] Here, an example of a method for determining the degree of approximation will be described with reference to Fig. 6. For example, when the input operation detection unit 112 detects the third gesture operation of the user, the gesture operation change determination unit 117 calculates the degree of approximation between the third gesture operation and each of the first gesture operation and the second gesture operation. The gesture operation change determination unit 117 calculates the degree of approximation based on the second hand and finger information. The degree of approximation is an index indicating the gripping degree Gn of the user's hand. Fig. 6 shows the gripping degree Gn of the user's hand that changes with time.

[0043] For example, as shown in FIG. 6, the grip degree Gn of the first gesture operation (hand opening operation) is set to "0" (or 0 to 0.1), and the grip degree Gn of the second gesture operation (hand clenching operation) is set to "1" (or 0.9 to 1). Information on the grip degree Gn corresponding to each of the first gesture operation and the second gesture operation is stored in advance in the storage unit 12. Here, when the input operation detection unit 112 detects the third gesture operation of the user, the gesture operation change determination unit 117 calculates the grip degree Gn corresponding to the third gesture operation based on the second hand and finger information of at least any one of the skeleton, angle, and position of the user's fingers. The closer the grip degree Gn is to "0", the closer the third gesture operation is to the first gesture operation, and the closer the grip degree Gn is to "1", the closer the third gesture operation is to the second gesture operation. The gesture operation change determination unit 117 calculates a similarity A1 between the first gesture operation and the third gesture operation and a similarity A2 between the second gesture operation and the third gesture operation based on the grip degree Gn corresponding to the third gesture operation. Then, the gesture operation change determination unit 117 determines a change from the first gesture operation to the second gesture operation based on the similarity degrees A1 and A2. Furthermore, the gesture operation change determination unit 117 determines that the first gesture operation has changed to the second gesture operation when an amount of change over time in the similarity degrees A1 and A2 is greater than a predetermined first reference value.

[0044] Movement vector correction unit 118 corrects the movement vector when gesture operation change determination unit 117 determines a change from the first gesture operation to the second gesture operation.

[0045] For example, the movement vector correction unit 118 corrects the movement vector by subtracting a predetermined correction movement amount from the movement amount calculated by the movement vector calculation unit 115. Also, for example, the movement vector correction unit 118 corrects the movement vector by subtracting a correction movement vector consisting of a correction movement direction and a correction movement amount set in advance from the movement vector calculated by the movement vector calculation unit 115. For example, a plurality of combinations of the movement amount of the operation position of the gesture operation of the user and the correction movement amount corresponding to the movement amount may be registered in advance in the storage unit 12. The movement vector correction unit 118 refers to the storage unit 12, acquires the correction movement amount corresponding to the movement amount calculated by the movement vector calculation unit 115, and corrects the movement vector.

[0046] As another embodiment, movement vector correction unit 118 may correct the movement vector by multiplying the movement amount calculated by movement vector calculation unit 115 by a predetermined correction coefficient that is a preset value smaller than 1. For example, a plurality of combinations of movement amounts of the operation position of the user's gesture operation and correction coefficients corresponding to the movement amounts may be registered in advance in storage unit 12. Movement vector correction unit 118 refers to storage unit 12, acquires the correction coefficient corresponding to the movement amount calculated by movement vector calculation unit 115, and corrects the movement vector.

[0047] In another embodiment, when the movement amount calculated by movement vector calculation unit 115 is greater than a predetermined upper limit value set in advance, movement vector correction unit 118 may correct the movement vector so that the movement amount is equal to or less than the upper limit value. The upper limit value may be registered in advance in storage unit 12. When the movement amount calculated by movement vector calculation unit 115 is greater than the upper limit value, movement vector correction unit 118 executes a correction process to correct the movement vector. With this configuration, the correction process is executed only when necessary, and therefore the processing load can be reduced.

[0048] In this manner, the movement vector correction unit 118 corrects the movement vector so as to offset the movement of the input position caused by the change of the reference point from the position of the reference point of the first gesture operation (center position P0 in FIG. 4A) to the position of the reference point of the second gesture operation (center position P1 in FIG. 4B).

[0049] Here, in the case where gesture operation change determination unit 117 is configured to determine that the first gesture operation has changed to the second gesture operation when the amount of change in the degree of approximation over time is greater than a predetermined first reference value, movement vector correction unit 118 may correct the movement vector during a period from the time when it is determined that the amount of change in the degree of approximation over time is greater than the first reference value until a predetermined duration has elapsed. This allows the correction process to be executed only during the duration, thereby reducing the processing load. Note that the duration may be set in advance in association with the magnitude of the degree of approximation.

[0050] Furthermore, the motion vector correction unit 118 may correct the motion vector from the time when the time change in the degree of approximation is determined to be greater than a first reference value until the time when the time change in the degree of approximation is determined to be equal to or smaller than the first reference value, thereby making it possible to reliably reduce the motion amount of the motion vector to a predetermined amount.

[0051] Furthermore, when the amount of change in the degree of approximation over time is greater than a predetermined second reference value, the movement vector correction unit 118 may increase the effect of correcting the movement vector compared to when the amount of change in the degree of approximation over time is smaller than the second reference value. This allows the movement vector to be corrected according to the degree of change in the gesture operation, making it possible to efficiently execute the correction process.

[0052] As described above, the correction process by the movement vector correction unit 118 can fix the position of the reference point regardless of changes in the user's gesture operation, thereby preventing the input position from moving due to changes in the gesture operation.

[0053] [Display control processing] Hereinafter, the display control process executed by the control unit 11 of the display device 1 will be described with reference to FIG.

[0054] The present invention can be understood as an invention of a display control method (one example of the input method of the present invention) that executes one or more steps included in the display control process, and one or more steps included in the display control process described here may be omitted as appropriate. The steps in the display control process may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the control unit 11 executes each step in the display control process, another embodiment can also be a display control method in which multiple processors execute each step in the display control process in a distributed manner.

[0055] First, in step S11, the control unit 11 determines whether or not an input operation by the user has been detected. Specifically, the control unit 11 detects the input operation by the user on the virtual operation surface R2. For example, the control unit 11 detects detection coordinates on the virtual operation surface R2 based on detection information acquired from the motion sensor 15, and calculates input coordinates in the operation area R1 of the display screen 13A from the detection coordinates. If the control unit 11 detects the input operation (S11: Yes), the process proceeds to step S12. If the control unit 11 does not detect the input operation (S11: No), the process proceeds to step S18.

[0056] In step S12, the control unit 11 judges whether or not a predetermined gesture operation of the user is detected. For example, the control unit 11 detects an operation of opening the user's right hand RH (an operation of making the right hand RH into a "paper" shape) or an operation of closing the user's right hand RH (an operation of making the right hand RH into a "rock" shape) based on at least any of finger information of the skeleton, angle, and position of the user's fingers. If the control unit 11 detects the user's gesture operation (S12: Yes), the process proceeds to step S13. On the other hand, if the control unit 11 does not detect the user's gesture operation (S12: No), the process proceeds to step S17. Step S12 is an example of a gesture operation detection step of the present invention.

[0057] Next, in step S13, the control unit 11 determines the operation position of the gesture operation. Specifically, the control unit 11 determines a reference point of the gesture operation on the virtual operation surface R2 based on a captured image of the gesture operation captured by the motion sensor 15, and determines the operation position based on the position of the reference point. For example, the control unit 11 creates first finger information of at least one of the skeleton, angle, and position of the user's fingers based on the captured image of the gesture operation captured by the motion sensor 15, determines the position of the reference point (for example, the center position P0 shown in FIG. 4A) based on the first finger information, and determines the operation position based on the position of the reference point. Step S13 is an example of an operation position determination step of the present invention.

[0058] Furthermore, in step S13, the control unit 11 displays the input operation icon M1 at an input position on the display screen 13A that corresponds to the determined operation position.

[0059] Next, in step S14, the control unit 11 executes an input process corresponding to the gesture operation of the user. For example, when the user moves the right hand RH in the up, down, left and right directions (X direction and Y direction) in an open state (the first gesture operation) on the virtual operation surface R2, the control unit 11 accepts the movement operation and moves the input operation icon M1 displayed on the display screen 13A so as to follow the movement of the user's right hand RH. Step S14 is an example of an input step of the present invention.

[0060] In step S14, the control unit 11 calculates a movement vector consisting of a movement direction and a movement amount for moving the input position based on the movement amount of the operation position when the operation position of the user's gesture operation is moved. The control unit 11 calculates the movement vector of the input position based on the movement vector of the operation position, and moves the input operation icon M1 on the display screen 13A. Step S14 is an example of a movement vector calculation step of the present invention.

[0061] Next, in step S15, the control unit 11 determines whether the gesture operation has changed. Specifically, the control unit 11 calculates the degree of approximation between the detected gesture operation of the user and the first gesture operation (hands open) and the second gesture operation (hands closed), and determines the change from the first gesture operation to the second gesture operation based on the degree of approximation. For example, the control unit 11 determines that the detected gesture operation has changed from the first gesture operation to the second gesture operation when the amount of change in the degree of approximation over time is greater than a predetermined first reference value. If the gesture operation has changed (S15: Yes), the process proceeds to step S16. If the gesture operation has not changed (S15: No), the process proceeds to step S18. Step S15 is an example of a gesture operation change determination step of the present invention.

[0062] Next, in step S16, the control unit 11 corrects the movement vector of the input position. For example, the control unit 11 corrects the movement vector by subtracting a predetermined correction movement amount that is set in advance from the movement amount of the input position corresponding to the movement amount of the operation position of the user's gesture operation.

[0063] Furthermore, control unit 11 may correct the movement vector by multiplying the calculated movement amount by a predetermined correction coefficient that is a preset value smaller than 1. Furthermore, when the calculated movement amount is greater than a preset upper limit value, control unit 11 may correct the movement vector so that the movement amount is equal to or less than the upper limit value.

[0064] In this way, control unit 11 corrects the movement vector so as to offset the movement of the input position caused by the change of the reference point from the position of the reference point of the first gesture operation (center position P0 in FIG. 4A) to the position of the reference point of the second gesture operation (center position P1 in FIG. 4B). Step S16 is an example of a movement vector correction step of the present invention.

[0065] Next, in step S17, the control unit 11 executes an input process corresponding to the gesture operation of the user. For example, the control unit 11 executes an input process by the input operation icon M1 based on the corrected movement vector. For example, when the control unit 11 determines a change from the first gesture operation (open hand) to the second gesture operation (closed hand) at a position where the input operation icon M1 overlaps with the "order" selection button image on the display screen 13A, the control unit 11 offsets the change in the input position and executes the order process assigned to the "order" selection button image.

[0066] Next, in step S18, the control unit 11 determines whether or not the input operation has ended. If the input operation has ended (S18: Yes), the control unit 11 ends the display control process. If the input operation has not ended (S18: No), the control unit 11 returns to step S11.

[0067] In this manner, the control unit 11 executes the display control process.

[0068] As described above, the display device 1 according to the present embodiment detects the gesture operation of the user and determines the operation position of the detected gesture operation. When the operation position moves, the display device 1 calculates a movement vector consisting of a movement direction and a movement amount for moving the input position based on the movement amount of the operation position. When the display device 1 detects a first gesture operation of the user, the display device 1 executes a first input process at the input position, and when the display device 1 detects a second gesture operation of the user, the display device 1 executes a second input process at the input position. When the display device 1 determines a change from the first gesture operation to the second gesture operation, the display device 1 corrects the movement vector. Specifically, the display device 1 corrects the movement vector so as to offset the movement of the input position accompanying the movement of the reference point of the gesture operation.

[0069] This makes it possible to prevent the input position from changing in a direction unintended by the user, for example, when the user changes from an operation of opening his / her hand (first gesture operation) to an operation of closing (grippling) his / her hand (second gesture operation).

[0070] [How to set up the virtual control surface] The operation surface setting unit 111 may set a virtual operation surface R2 of a desired size at a desired position based on a predetermined setting gesture operation by the user.

[0071] Specifically, the operation surface setting unit 111 detects a setting gesture operation by the user. Specifically, the operation surface setting unit 111 detects the setting gesture operation based on detection information acquired from the motion sensor 15. For example, the operation surface setting unit 111 identifies the shape of the user's hand based on coordinate information included in the detection information, and identifies a corresponding gesture operation from among a plurality of gesture operations registered in advance in the storage unit 12.

[0072] When detecting a predetermined first gesture operation for setting by the user, the operation surface setting unit 111 sets an area corresponding to the first gesture operation for setting as a virtual operation surface R2 that accepts an input operation by the user on the display screen 13A. The operation surface setting unit 111 may set the virtual operation surface R2 when the first gesture operation for setting is continuously performed for a predetermined time. The first gesture operation for setting is, for example, an operation of holding the palms of the left hand LH and the right hand RH toward the display screen 13A. In other words, the first gesture operation for setting is a setting operation for the user to set the virtual operation surface R2.

[0073] 8, when the user holds the palm of the left hand LH over the display screen 13A at an arbitrary position on the upper left and holds the palm of the right hand RH over the display screen 13A at an arbitrary position on the lower right, the operation surface setting unit 111 detects the coordinate Ph1 of the left hand LH, the coordinate Ph2 of the right hand RH, and a first setting gesture operation of holding the left hand LH and the right hand RH, based on the detection information obtained from the motion sensor 15. When the operation surface setting unit 111 detects the first setting gesture operation, it sets a virtual operation surface R2 based on the coordinate Ph1 of the left hand LH and the coordinate Ph2 of the right hand RH.

[0074] For example, as shown in Fig. 8, the operation surface setting unit 111 sets a rectangular virtual operation surface R2 whose diagonal is a line connecting the position of the left hand LH (coordinate Ph1) and the position of the right hand RH (coordinate Ph2). Specifically, the operation surface setting unit 111 calculates coordinates C21 to C24 (see Fig. 2) of the corners of the rectangle based on the coordinate Ph1 of the left hand LH and the coordinate Ph2 of the right hand RH to set the virtual operation surface R2.

[0075] Also, for example, the operation surface setting unit 111 sets a virtual operation surface R2 at a position a predetermined distance L1 away from the display screen 13A. Note that the predetermined distance L1 is a distance corresponding to the coordinate Ph1 (Z coordinate) of the left hand LH and the coordinate Ph2 (Z coordinate) of the right hand RH.

[0076] Also, for example, the operation surface setting unit 111 may set a virtual operation surface R2 whose aspect ratio is the same as that of the display screen 13A. Specifically, as shown in Fig. 9, the operation surface setting unit 111 sets a virtual operation surface R2 whose aspect ratio (H1:W1) of the display screen 13A is the same as that of the virtual operation surface R2 (H2:W2) (H1:W1 = H2:W2).

[0077] In this way, the size of the display screen 13A (operation area R1) and the size of the virtual operation surface R2 may be the same or different. Here, when the virtual operation surface R2 is smaller than the operation area R1, it is suitable for use in operating a large display panel 13 close to the user. On the other hand, when the virtual operation surface R2 is larger than the operation area R1, it is suitable for use in operating a small display panel 13 from a distant position.

[0078] Also, as shown in FIG. 10, the operation surface setting unit 111 may set a virtual operation surface R2 having a predetermined angle d1 that is not parallel to the display screen 13A. That is, the virtual operation surface R2 may be set in an oblique direction with respect to the display screen 13A. For example, the operation surface setting unit 111 sets the predetermined angle d1 based on the coordinate Ph1 (Z coordinate) of the left hand LH and the coordinate Ph2 (Z coordinate) of the right hand RH. This allows the user to perform an input operation from an oblique direction with respect to the display screen 13A. Note that the operation surface setting unit 111 may display information on the predetermined angle d1 on the display screen 13A. This allows the user to grasp the angle (degree of inclination) of the virtual operation surface R2 with respect to the display screen 13A.

[0079] Also, the operation surface setting unit 111 may set a virtual operation surface R2 corresponding to a partial area of ​​the display screen 13A. For example, as shown in FIG. 11, the operation surface setting unit 111 sets a virtual operation surface R2 corresponding to an operation area R1 of a part (left area) of the display screen 13A. The position and size of the operation area R1 can be set by a setting operation by the user. Note that, when setting the virtual operation surface R2, the operation surface setting unit 111 may display an object image T1 indicating the operation area R1 on the display screen 13A as shown in FIG. 12 so that the user who sets the virtual operation surface R2 can easily grasp the operation area R1.

[0080] In addition, the operation surface setting unit 111 can use well-known coordinate transformation (projective transformation, affine transformation, etc.) to set a virtual operation surface R2 associated with the operation area R1 of the display screen 13A based on the coordinates corresponding to the first setting gesture operation.

[0081] Furthermore, the operation surface setting unit 111 may execute a process of adjusting the set virtual operation surface R2. Specifically, when the operation surface setting unit 111 detects a predetermined second setting gesture operation by the user after the virtual operation surface R2 is set, the operation surface setting unit 111 changes at least one of the size and the position of the virtual operation surface R2 based on the second setting gesture operation. The second setting gesture operation is, for example, a pointing operation with the right hand RH (see FIG. 13).

[0082] For example, as shown in FIG. 13, when the user points at the display screen 13A with the right hand RH after the virtual operation surface R2 is set, the operation surface setting unit 111 detects the coordinates Ph3 of the right hand RH and the second setting gesture operation of the pointing operation by the right hand RH based on the detection information acquired from the motion sensor 15. When the operation surface setting unit 111 detects the second setting gesture operation, it sets the virtual operation surface R2 to be movable based on the coordinates Ph3 of the right hand RH and accepts the user's operation to move the virtual operation surface R2. For example, when the user moves the right hand RH to the left while keeping the pointing state, the operation surface setting unit 111 moves the virtual operation surface R2 to the left by an amount corresponding to the amount of movement of the right hand RH. That is, the operation surface setting unit 111 sets the virtual operation surface R2 to the coordinates Ph3 of the right hand RH after the movement.

[0083] 14, for example, when the user performs an operation of clenching the right hand RH while holding the left hand LH after the virtual operation surface R2 is set, the operation surface setting unit 111 detects the coordinates Ph1 of the left hand LH, the coordinates Ph2 of the right hand RH, and a second setting gesture operation of holding the left hand LH and clenching the right hand RH based on the detection information acquired from the motion sensor 15. When the operation surface setting unit 111 detects the second setting gesture operation, it sets the size of the virtual operation surface R2 to be changeable based on the coordinates Ph2 of the right hand RH, and accepts the user's operation of changing the size of the virtual operation surface R2. For example, when the user moves the right hand RH in a lower right direction while holding the right hand RH in a fist, the operation surface setting unit 111 expands the size (area) of the virtual operation surface R2 by an amount corresponding to the amount of movement of the right hand RH. That is, the operation surface setting unit 111 sets a virtual operation surface R2 defined by the coordinate Ph1 of the left hand LH and the coordinate Ph2 of the right hand RH after the movement.

[0084] In the example shown in FIG. 15, after the virtual operation surface R2 is set, the user performs an operation of clenching the left hand LH while holding the right hand RH over the screen. In this case, the operation surface setting unit 111 detects the coordinates Ph1 of the left hand LH, the coordinates Ph2 of the right hand RH, and a second setting gesture operation of holding the right hand RH over the screen and clenching the left hand LH, based on the detection information acquired from the motion sensor 15. When the operation surface setting unit 111 detects the second setting gesture operation, it sets the size of the virtual operation surface R2 to be changeable based on the coordinates Ph1 of the left hand LH, and accepts the user's operation of changing the size of the virtual operation surface R2. For example, when the user moves the left hand LH in a lower right direction while holding the fist over the screen, the operation surface setting unit 111 reduces the size (area) of the virtual operation surface R2 by an amount corresponding to the amount of movement of the left hand LH. That is, the operation surface setting unit 111 sets a virtual operation surface R2 defined by the coordinate Ph2 of the right hand RH and the coordinate Ph1 of the moved left hand LH.

[0085] In addition, when the operation surface setting unit 111 detects the second setting gesture operation after the virtual operation surface R2 is set, the operation surface setting unit 111 may display an object image T2 representing the virtual operation surface R2 on the display screen 13A in response to the second setting gesture operation. Fig. 16 shows an example of the object image T2 representing the virtual operation surface R2 after the size has been changed. With this configuration, the user can visually grasp the size, position, etc. of the virtual operation surface R2 after the change.

[0086] Furthermore, the display device of the present invention can be constructed by freely combining the embodiments described above, or by appropriately modifying or omitting parts of each embodiment, within the scope of the invention described in each claim. [Explanation of symbols]

[0087] 1:Display device 13: Display panel 13A:Display screen 15: Motion sensor 111: Operation surface setting section 112: Input operation detection unit 113: Operation position determination section 114: Display processing unit 115: Movement vector calculation unit 116: Input processing section 117: Gesture operation change determination unit 118: Motion vector correction unit

Claims

1. An input device that detects a gesture operation by a user and performs input processing at an input position on a display screen, a gesture operation detection unit that detects the gesture operation of the user; an operation position determination unit that determines an operation position of the gesture operation detected by the gesture operation detection unit; a movement vector calculation unit that calculates a movement vector including a movement direction and a movement amount for moving the input position based on a movement amount of the operation position when the operation position is moved; an input processing unit that executes a first input process at the input position when the gesture operation detection unit detects a first gesture operation of the user, and executes a second input process at the input position when the gesture operation detection unit detects a second gesture operation of the user; a gesture operation change determination unit that determines a change from the first gesture operation to the second gesture operation; a movement vector correction unit that corrects the movement vector when the gesture operation change determination unit determines a change from the first gesture operation to the second gesture operation; Equipped with the gesture operation change determination unit calculates a degree of similarity between the gesture operation of the user detected by the gesture operation detection unit and the first gesture operation and the second gesture operation, and determines that the gesture operation has changed from the first gesture operation to the second gesture operation when a time change amount of the degree of similarity is greater than a predetermined first reference value; The movement vector correction unit corrects the movement vector for a predetermined duration from a point in time when it is determined that the amount of change over time of the degree of approximation is greater than the first reference value until the elapse of the predetermined duration.

2. the motion vector correction unit corrects the motion vector by subtracting a predetermined correction motion amount set in advance from the motion amount calculated by the motion vector calculation unit. The input device according to claim 1 .

3. the motion vector correction unit corrects the motion vector calculated by the motion vector calculation unit by subtracting a correction motion vector having a preset correction motion direction and correction motion amount from the motion vector calculated by the motion vector calculation unit. The input device according to claim 1 .

4. the motion vector correction unit corrects the motion vector by multiplying the motion amount calculated by the motion vector calculation unit by a predetermined correction coefficient that is a preset value smaller than 1; The input device according to claim 1 .

5. the motion vector correction unit corrects the motion vector when the motion amount calculated by the motion vector calculation unit is greater than a predetermined upper limit value set in advance, so that the motion amount is equal to or less than the upper limit value. The input device according to any one of claims 1 to 4.

6. an imaging unit that images the gesture operation; the operation position determination unit determines a reference point of the gesture operation based on a captured image of the gesture operation captured by the imaging unit, and determines the operation position based on a position of the reference point; The input device according to any one of claims 1 to 5.

7. the operation position determination unit determines, as the reference point, a center of gravity position of an image of the gesture operation captured by the imaging unit; 7. The input device according to claim 6.

8. the operation position determination unit creates first hand and finger information of at least any one of a skeleton, an angle, and a position of the user's fingers based on a captured image of the gesture operation captured by the imaging unit, and determines the position of the reference point based on the first hand and finger information; 7. The input device according to claim 6.

9. the movement vector correction unit corrects the movement vector so as to cancel a movement of the input position caused by a change of a reference point from a position of the reference point of the first gesture operation to a position of the reference point of the second gesture operation. The input device according to any one of claims 6 to 8.

10. an imaging unit that images the gesture operation; the gesture operation change determination unit creates second hand and finger information of at least any one of a skeleton, an angle, and a position of the user's fingers based on a captured image of the gesture operation of the user captured by the imaging unit, and determines the degree of approximation based on the second hand and finger information. The input device according to claim 1 .

11. the motion vector correction unit corrects the motion vector during a period from a point in time when it is determined that the amount of change over time of the degree of approximation is greater than the first reference value to a point in time when it is determined that the amount of change over time of the degree of approximation is equal to or smaller than the first reference value. The input device according to any one of claims 1 to 10.

12. the motion vector correction unit increases a correction effect of correcting the motion vector when the amount of change over time of the degree of approximation is greater than a predetermined second reference value, compared to when the amount of change over time of the degree of approximation is smaller than the second reference value. The input device according to any one of claims 1 to 11.

13. An input method for detecting a gesture operation of a user and performing input processing on an input position on a display screen, comprising: One or more processors a gesture operation detection step of detecting the gesture operation of the user; an operation position determination step of determining an operation position of the gesture operation detected in the gesture operation detection step; a movement vector calculation step of calculating, when the operation position has moved, a movement vector including a movement direction and a movement amount for moving the input position based on the movement amount of the operation position; an input step of executing a first input process at the input position when a first gesture operation of the user is detected in the gesture operation detection step, and executing a second input process at the input position when a second gesture operation of the user is detected in the gesture operation detection step; a gesture operation change determination step of determining a change from the first gesture operation to the second gesture operation; a movement vector correction step of correcting the movement vector when a change from the first gesture operation to the second gesture operation is determined in the gesture operation change determination step; Run in the gesture operation change determination step, a degree of similarity between the gesture operation of the user detected in the gesture operation detection step and the first gesture operation and the second gesture operation is calculated, and when an amount of change over time of the degree of similarity is greater than a predetermined first reference value, it is determined that the gesture operation has changed from the first gesture operation to the second gesture operation; an input method, wherein in the movement vector correcting step, the movement vector is corrected for a predetermined duration from a point in time when it is determined that the amount of change over time of the degree of approximation is greater than the first reference value until the elapse of a predetermined duration.

14. An input program that detects a gesture operation by a user and performs input processing at an input position on a display screen, a gesture operation detection step of detecting the gesture operation of the user; an operation position determination step of determining an operation position of the gesture operation detected in the gesture operation detection step; a movement vector calculation step of calculating, when the operation position has moved, a movement vector including a movement direction and a movement amount for moving the input position based on the movement amount of the operation position; an input step of executing a first input process at the input position when a first gesture operation of the user is detected in the gesture operation detection step, and executing a second input process at the input position when a second gesture operation of the user is detected in the gesture operation detection step; a gesture operation change determination step of determining a change from the first gesture operation to the second gesture operation; a movement vector correction step of correcting the movement vector when a change from the first gesture operation to the second gesture operation is determined in the gesture operation change determination step; on one or more processors, in the gesture operation change determination step, a degree of similarity between the gesture operation of the user detected in the gesture operation detection step and the first gesture operation and the second gesture operation is calculated, and when an amount of change over time of the degree of similarity is greater than a predetermined first reference value, it is determined that the gesture operation has changed from the first gesture operation to the second gesture operation; an input program, comprising: a step of correcting the movement vector during a period from a time when it is determined that the amount of change over time in the degree of approximation is greater than the first reference value until a predetermined duration has elapsed;

Images