Input device, input method, and input program

Abstract

To provide an input device configured to improve operability of an input device that detects a gesture operation of a user and executes input processing to an input position on a display screen, an input method, and an input program.SOLUTION: An input device includes: an acquisition processing unit which acquires a captured image obtained by imaging a user; a detection processing unit which detects a hand of the user from the captured image acquired by the acquisition processing unit; a first setting processing unit which sets an operation region of a gesture operation corresponding to an input possible region in a display screen where gesture operation input is accepted, the region including the hand of the user detected by the detection processing unit; and a display processing unit which displays the captured image acquired by the acquisition processing unit, on the display screen, and superimposes an operation region image corresponding to the operation region set by the first setting processing unit on the captured image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an input device, an input method, and an input program that detect a gesture operation by a user's hand and perform input processing at an input position on a display screen. [Background technology]

[0002] Conventionally, there is known an input device that allows input (screen operation) by gesture operation on the display screen of a display panel. For example, Patent Document 1 discloses a device that grasps at least one of the shape, movement, and position of a user's hand from an image captured of the hand in a virtual space and determines input information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-75991 A Summary of the Invention [Problem to be solved by the invention]

[0004] Here, the input device specifies a position (input position) on the display screen designated by, for example, a gesture operation with the user's hand, based on the position of the hand. The user also performs an input operation by moving his / her hand or changing the shape of his / her hand while looking at a mouse cursor image displayed on the display screen. However, with conventional technology, a problem occurs in that it is difficult for the user to grasp the correspondence between the range in which the user moves his / her hand and the movable range within the display screen of the mouse cursor image that moves on the display screen in response to the movement of the user's hand. For example, it is difficult for the user to grasp how far the mouse cursor image on the display screen will move when the user moves his / her hand by a predetermined distance. Thus, conventional input devices have a problem of low operability for users.

[0005] An object of the present invention is to provide an input device, an input method, and an input program that can improve the operability of an input device that detects a user's gesture operation and executes input processing at an input position on a display screen. [Means for solving the problem]

[0006] An input device according to one aspect of the present invention is an apparatus comprising: an acquisition processing unit that acquires an image of a user; a detection processing unit that detects the user's hand from the image acquired by the acquisition processing unit; a first setting processing unit that sets an operation area for the gesture operation which is an area including the user's hand detected by the detection processing unit and corresponds to an inputtable area on the display screen which can be input by the gesture operation; and a display processing unit that displays the image acquired by the acquisition processing unit on the display screen and displays an operation area image corresponding to the operation area set by the first setting processing unit superimposed on the image.

[0007] An input method according to another aspect of the present invention is an input method that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, in which one or more processors execute the following steps: an acquisition step of acquiring an image of the user; a detection step of detecting the user's hand from the image acquired in the acquisition step; a first setting step of setting an operation area for the gesture operation which is an area including the user's hand detected in the detection step and corresponds to an inputtable area on the display screen which can be input by the gesture operation; and a display step of displaying the image acquired in the acquisition step on the display screen, and superimposing an operation area image corresponding to the operation area set in the first setting step on the image.

[0008] An input program according to another aspect of the present invention is an input program that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, the program causing one or more processors to execute the following steps: an acquisition step of acquiring an image of the user; a detection step of detecting the user's hand from the image acquired in the acquisition step; a first setting step of setting an operation area for the gesture operation which is an area including the user's hand detected in the detection step and corresponds to an inputtable area on the display screen which can be input by the gesture operation; and a display step of displaying the image acquired in the acquisition step on the display screen and displaying an operation area image corresponding to the operation area set in the first setting step superimposed on the image. Effect of the Invention

[0009] According to the present invention, it is possible to provide an input device, an input method, and an input program that can improve the operability of an input device that detects a user's gesture operation and executes input processing at an input position on a display screen. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of a display device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a display screen 13A in the display device according to the embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram showing an example of a captured image displayed on the display device according to the embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing a method of setting a reference point by the first setting method according to the embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing an example of a captured image displayed on the display device according to the embodiment of the present invention. [Figure 6]FIG. 6 is a diagram showing a method of setting a reference point by the first setting method according to the embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing a method of setting a reference point by the second setting method according to the embodiment of the present invention. [Figure 8] FIG. 8 is a diagram showing a method for setting a reference point by the third setting method according to the embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing an example of a rectangular area in a captured image displayed on the display device according to the embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of an operation region in a captured image displayed on the display device according to the embodiment of the present invention. [Figure 11] FIG. 11 is a diagram showing an example of an operation region in a captured image displayed on the display device according to the embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing a method of calculating an input position by the first calculation method according to the embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing a method of calculating an input position by the second calculation method according to the embodiment of the present invention. [Figure 14] FIG. 14 is a diagram showing a calculation method of an input position by the third calculation method according to the embodiment of the present invention. [Figure 15] FIG. 15 is a flowchart illustrating an example of a procedure of a display control process executed by the display device according to the embodiment of the present invention. [Figure 16] FIG. 16 is a flowchart illustrating an example of a procedure of a reference point setting process by the first setting method according to the embodiment of the present invention. [Figure 17] FIG. 17 is a flowchart illustrating an example of a procedure of a reference point setting process by the second setting method according to the embodiment of the present invention. [Figure 18] FIG. 18 is a flowchart illustrating an example of a procedure of a reference point setting process by the third setting method according to the embodiment of the present invention. [Figure 19]FIG. 19 is a flowchart illustrating an example of a procedure of an input position calculation process by the first calculation method according to an embodiment of the present invention. [Figure 20] FIG. 20 is a flowchart illustrating an example of a procedure of an input position calculation process using the second calculation method according to an embodiment of the present invention. [Figure 21] FIG. 21 is a flowchart illustrating an example of a procedure of an input position calculation process using the third calculation method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the following embodiment is an example of the present invention, and does not limit the technical scope of the present invention.

[0012] As shown in Fig. 1, a display device 1 according to an embodiment of the present invention includes a control unit 11, a storage unit 12, a display panel 13, an operation unit 14, and a camera 15. Fig. 2 shows a schematic diagram of the display device 1. The camera 15 is installed on the upper part of the display panel 13, and captures an image of a predetermined area in front of the display panel 13. The display device 1 is an example of an input device of the present invention. The input device of the present invention is not limited to a display device, and may be an information processing device such as a server capable of data communication with the display panel 13 and the camera 15.

[0013] The display device 1 accepts a non-contact input operation by the user on the display screen 13A. For example, when the display device 1 detects an input operation by a gesture by the user (gesture operation) in front of the camera 15, it executes an input process according to the input operation by the user on the display screen 13A. For example, when the user closes his / her hand (makes a fist) in front of the camera 15, the display device 1 accepts a touch input at the position of the mouse cursor M1 on the display screen 13A. The specific configuration of the display device 1 will be described below.

[0014] Camera 15 is a digital camera that captures an image of a subject and outputs it as digital image data. Camera 15 captures images at a predetermined frame rate and sequentially outputs the digital image data to control unit 11. There may be one camera 15 or multiple cameras. Specifically, camera 15 is installed above display panel 13 and captures an image of a predetermined area in front of display panel 13.

[0015] The display panel 13 is a display that displays an image on a display screen 13A, and is, for example, a liquid crystal display. The operation unit 14 is an operation device such as a mouse, a keyboard, etc. The operation unit 14 may be configured as a touch panel.

[0016] The display panel 13 displays various images according to instructions from the control unit 11. For example, as shown in FIG. 2, the display panel 13 displays a mouse cursor M1 (one example of an input operation icon of the present invention) that moves following the user's hand, operation buttons (here, "Yes" button and "No" button) that accept touch input in response to a predetermined gesture operation by the user's hand, a captured image Wp (wipe screen) captured by the camera 15, and the like. FIG. 3 shows an enlarged view of the captured image Wp. Although details will be described later, the captured image Wp displays a frame image that represents the outer frame of a rectangular area A1 (one example of a specific area of ​​the present invention) that surrounds the user's hand, and a frame image that represents the outer frame of an operation area A11 that corresponds to an inputtable area F1 (see FIG. 2) on the display screen 13A where input is possible. The user moves the position of the mouse cursor M1 and selects (clicks) an operation button with the mouse cursor M1 while viewing the captured image Wp that is displayed superimposed on the display screen 13A. For example, the user moves the display position of the mouse cursor M1 by moving the mouse cursor M1 with an open hand (open hand) (see FIG. 3), and when the mouse cursor M1 is located over the "Yes" button, the user selects (clicks) the "Yes" button by closing the hand (rock) (see FIG. 5). The operation of opening the hand is an example of the first gesture operation of the present invention, and the operation of closing the hand is an example of the second gesture operation of the present invention.

[0017] The storage unit 12 is a non-volatile storage unit such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information. The storage unit 12 stores control programs such as a display control program for causing the control unit 11 to execute a display control process (see FIG. 15, etc.) described below. For example, the display control program is non-temporarily recorded on a computer-readable recording medium such as a CD or a DVD, read by a reading device (not shown) such as a CD drive or a DVD drive provided in the display device 1, and stored in the storage unit 12. The display control program may be distributed from a cloud server and stored in the storage unit 12.

[0018] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for various processes executed by the CPU. The control unit 11 controls the display device 1 by executing, by the CPU, various control programs that are stored in advance in the ROM or the storage unit 12.

[0019] However, in the conventional technology, for example, when a user changes from an operation of opening his / her hand (see FIG. 3) to an operation of closing his / her hand (see FIG. 5), a problem occurs in which the input position on the display screen 13A changes in a direction that is not intended by the user. A specific example of this problem will be described below.

[0020] FIG. 4 shows an enlarged view of an open hand, and FIG. 6 shows an enlarged view of a closed hand. In the conventional technology, the center position P0 of the entire area (rectangular area A1) onto which the hand is projected during gesture operation is specified as the input position, so that when the hand is changed from an open state to a closed state, the center position P0 moves to the center position P0' of the rectangular area A2 as shown in FIG. 6. Therefore, when the user changes the hand from an open state to a closed state, the position of the mouse cursor M1 corresponding to the center position P0 also moves, and the user cannot correctly perform the intended operation, for example, the selection operation of the "Yes" button. In contrast, in the display device 1 according to the present embodiment, it is possible to prevent the input position from changing in an unintended direction due to a change in the user's gesture operation, as shown below.

[0021] Specifically, as shown in Fig. 1, the control unit 11 includes various processing units such as a display processing unit 111, an acquisition processing unit 112, a detection processing unit 113, a determination processing unit 114, a reference point setting processing unit 115, an operation area setting processing unit 116, a virtual inputtable area setting processing unit 117, and an input processing unit 118. The control unit 11 functions as each processing unit by executing various processes according to the display control program and by the CPU. Some or all of the processing units included in the control unit 11 may be configured with electronic circuits. The display control program may be a program for causing a plurality of processors to function as the various processing units.

[0022] The display processing unit 111 displays various information on the display panel 13. The display processing unit 111 displays various images according to the execution of various applications. For example, the display processing unit 111 displays the display screen 13A shown in FIG. 2 on the display panel 13. The display processing unit 111 also displays the captured image Wp on the end (corner) of the display screen 13A. The captured image Wp is a partial image of a specific range including a detected face image of a user among the captured images of a predetermined area captured by the camera 15. The display processing unit 111 displays the captured image Wp on a small screen in the corner of the display screen 13A. The display processing unit 111 also updates the captured image Wp in real time according to the captured image acquired from the camera 15. The display processing unit 111 also displays the captured image Wp on the display screen 13A, and displays a frame image (an example of an operation area image of the present invention) corresponding to the operation area A11 superimposed on the captured image Wp. The display position of the captured image Wp is not limited. Furthermore, the display processing unit 111 may display the captured image Wp on the condition that the user's face or hand is detected.

[0023] Furthermore, the display processing unit 111 displays a mouse cursor M1 (icon for input operation) at an input position corresponding to the reference point P1 within an inputtable area F1 set a predetermined width inward from the bottom edge of the display screen 13A. The predetermined width is set to the vertical width of the captured image Wp, for example. The user can move the mouse cursor M1 within the inputtable area F1. The display processing unit 111 is an example of a display processing unit of the present invention.

[0024] The acquisition processing unit 112 acquires captured images of the user. Specifically, the acquisition processing unit 112 sequentially acquires digital image data of images captured by the camera 15 at a predetermined frame rate from the camera 15. The acquisition processing unit 112 is an example of the acquisition processing unit of the present invention.

[0025] The detection processing unit 113 detects a user (e.g., a face) from the captured image acquired by the acquisition processing unit 112. The detection processing unit 113 also detects the user's hands from the captured image. A well-known method can be applied as a method for detecting the user's face and hands. For example, the detection processing unit 113 performs processing such as cropping, resizing, inversion, and rotation on the digital image data to detect the user's face and hands. When the detection processing unit 113 detects a hand, it tracks the movement of the hand. The detection processing unit 113 is an example of a detection processing unit of the present invention.

[0026] The determination processing unit 114 determines the shape of the user's hand detected by the detection processing unit 113. For example, the determination processing unit 114 determines whether the user's hand is in an open state (paper state) (see FIG. 3) or a closed state (rock state) (see FIG. 5). In addition, the determination processing unit 114 determines whether the user's hand detected by the detection processing unit 113 is a right hand or a left hand. The determination processing unit 114 is an example of the determination processing unit of the present invention.

[0027] The reference point setting processor 115 sets a reference point P1 at a predetermined position on the bottom side of a rectangular area A1 surrounding the user's hand detected by the detection processor 113. Specifically, the reference point setting processor 115 sets as the reference point P1 a position obtained by dividing the distance from one end E1 to the other end E2 on the bottom side of the rectangular area A1 at a predetermined ratio. The reference point setting processor 115 is an example of the reference point setting processor of the present invention.

[0028] The reference point P1 is a position in the virtual space corresponding to an input position on the display screen 13A. The input processing unit 118 executes input processing for the input position corresponding to the reference point P1. Specifically, when the shape of the user's hand detected by the detection processing unit 113 is a predetermined shape, the input processing unit 118 executes input processing for the input position corresponding to the reference point P1 set by the reference point setting processing unit 115. For example, the input processing unit 118 displays a mouse cursor M1 on the display screen 13A at a position where the reference point P1 is mapped to an area corresponding to the display screen 13A. The input processing unit 118 is an example of an input processing unit of the present invention.

[0029] Specific methods for setting the reference point P1 will be described below, with a first setting method, a second setting method, and a third setting method being given as specific examples of the setting method.

[0030] [First setting method] FIG. 4 shows an example of the first setting method. As shown in FIG. 4, the reference point setting processing unit 115 sets the center position of the lower side L1 of the rectangular area A1 to the reference point P1. Here, the rectangular area A1 is a rectangular area that contacts the outermost end of the hand in an open state, and the reference point P1 is the center position of the lower side L1 of the rectangular area A1 in an open state. Therefore, as shown in FIG. 6, even if the hand is closed, the position of the reference point P1 does not change. In this way, the reference point setting processing unit 115 sets the center position of the lower side L1 of the rectangular area A1 in an open state to the reference point P1 regardless of the change in the shape of the hand. Note that the reference point setting processing unit 115 may set the center position of the lower side L1 of the rectangular area A1 in a closed state to the reference point P1.

[0031] In the first setting method, the input processing unit 118 calculates the coordinates of the center position of the bottom side L1 of the rectangular area A1 as the coordinates of the input position. For example, when the width of the rectangular area A1 is W1, the height is H1, and the upper left end E0 is the origin [x, y], the input processing unit 118 calculates [x+(W1) / 2, y+H1] as the coordinates of the reference point P1. The input processing unit 118 calculates the position coordinates of the mouse cursor M1 based on the calculated coordinates of the reference point P1. The display processing unit 111 displays the mouse cursor M1 at the calculated position coordinates on the display screen 13A.

[0032] [Second setting method] FIG. 7 shows an example of the second setting method. As shown in FIG. 7, the reference point setting processing unit 115 sets a position on the bottom side L1 of the rectangular area A1 at a ratio of 1:2 from the end of the little finger side of the user's hand as the reference point P1. Here, when the hand is changed from an open state (see FIG. 4) to a closed state (see FIG. 6), the horizontal position of the center position P0 moves toward the little finger side. This movement amount is easily affected by the area and movement amount of the thumb. Therefore, the reference point setting processing unit 115 assumes that the 1 / 3 area from the thumb side of the rectangular area A1 is the thumb area, extracts the palm area excluding the thumb area (the 2 / 3 area on the little finger side of the rectangular area A1), and sets the center position of the bottom side L1 of the palm area to the reference point P1. That is, the reference point setting processing unit 115 sets a position on the bottom side L1 of the rectangular area A1 at a ratio of 1:2 from the end E1 on the little finger side of the user's hand as the reference point P1.

[0033] In the second setting method, the input processing unit 118 calculates the coordinates of the input position at a ratio of 1:2 from the end E1 of the bottom side L1 of the rectangular area A1 on the little finger side of the user's hand. For example, the input processing unit 118 calculates [x+(W1) / 3, y+H1] as the coordinates of the reference point P1. The input processing unit 118 calculates the position coordinates of the mouse cursor M1 based on the calculated coordinates of the reference point P1. The display processing unit 111 displays the mouse cursor M1 at the calculated position coordinates on the display screen 13A.

[0034] [Third setting method] FIG. 8 shows an example of the third setting method. As shown in FIG. 8, the reference point setting processing unit 115 specifies a specific pixel that displays the color of the user's hand among a plurality of pixels included in the rectangular area A1, calculates the total number of pixels of the specific pixels for each pixel row in the column direction from the upper side to the lower side L1 of the rectangular area A1, and sets the center position Xc of the partial side corresponding to the range of the pixel row in the lower side L1 where the total number of pixels exceeds the threshold Cth as the reference point P1. The graph shown in FIG. 8 shows the total number of pixels of the specific pixels for each pixel row. The reference point setting processing unit 115 calculates the minimum position Xmin and the maximum position Xmax of the pixel row in which the total number of pixels exceeds the threshold Cth, and calculates the center position Xc of the partial side from the minimum position Xmin to the maximum position Xmax. Then, the reference point setting processing unit 115 sets the calculated center position Xc as the reference point P1.

[0035] In the third setting method, the input processing unit 118 calculates the coordinates of the center position Xc of the partial side corresponding to the range of the pixel row in which the total pixel number exceeds the threshold value Cth of the lower side L1 as the coordinates of the input position. For example, when the coordinates of the minimum position are [Xmin, y+H1], the input processing unit 118 calculates [Xmin+(Xmax-Xmin) / 2, y+H1] as the coordinates of the reference point P1. The input processing unit 118 calculates the position coordinates of the mouse cursor M1 based on the calculated coordinates of the reference point P1. The display processing unit 111 displays the mouse cursor M1 at the calculated position coordinates on the display screen 13A.

[0036] The reference point setting processor 115 sets the reference point P1 by any one of the above-mentioned first setting method, second setting method, and third setting method.

[0037] According to each of the setting methods, by setting the reference point P1 on the bottom side L1 of the rectangular area A1, it is possible to prevent the position from shifting in the vertical direction. Also, by using the information on the rectangular area A1 in the shape of an open hand (first gesture operation), it is possible to prevent the position of the mouse cursor M1 from shifting even when the rectangular area A1 changes to the shape of a closed hand (second gesture operation) (see FIG. 6).

[0038] Furthermore, according to the first setting method, the reference point P1 can be fixed to the center of the lower side L1, so the amount of calculation required for the input position can be reduced. This is effective when the CPU has low processing power or when high-speed processing is required.

[0039] According to the second setting method, the thumb area is assumed to be approximately 1 / 3 of the rectangular area A1 of the hand, so that the reference point P1 can be set by focusing on the palm area, which has little change in area. Therefore, the input position in the lateral direction can be calculated with high accuracy and stability.

[0040] Furthermore, according to the third setting method, the reference point P1 can be set in the palm area by excluding the thumb area, so that the lateral input position can be calculated more accurately and stably.

[0041] The input processing unit 118 executes input processing for an input position corresponding to the reference point P1 set by the reference point setting processing unit 115 using any of the setting methods described above. Specifically, when the shape of the user's hand detected by the detection processing unit 113 is an open hand shape (first shape), the input processing unit 118 executes movement processing for moving the mouse cursor M1 displayed on the display screen 13A in accordance with the movement of the reference point P1. Also, when the shape of the user's hand detected by the detection processing unit 113 changes from an open hand shape to a closed hand shape (second shape), the input processing unit 118 executes selection processing for a selection target (for example, the "Yes" button or the "No" button in FIG. 2) at the display position of the mouse cursor M1 corresponding to the reference point P1.

[0042] Here, the display device 1 specifies a position (input position) on the display screen 13A designated by, for example, a gesture operation by the user's hand based on the position of the hand. The user performs an input operation by moving his / her hand or changing the shape of his / her hand while watching the mouse cursor M1 displayed on the display screen 13A. However, in the conventional technology, a problem occurs in that it is difficult for the user to grasp the correspondence relationship between the range in which the user moves his / her hand and the movable range within the display screen 13A of the mouse cursor M1 that moves on the display screen 13A according to the movement of the user's hand. For example, it is difficult for the user to grasp how far the mouse cursor M1 on the display screen 13A will move when the user moves his / her hand by a predetermined distance. Thus, the conventional technology has a problem of low operability for the user. In contrast, in the display device 1 according to this embodiment, it is possible to improve the operability of the display device 1 as shown below.

[0043] Specifically, the operation area setting processing unit 116 sets an operation area A11 for gesture operation that is an area including the user's hand detected by the detection processing unit 113 and corresponds to the input possible area F1 (see FIG. 2) in the display screen 13A where input is possible by gesture operation. Specifically, when the detection processing unit 113 detects the user's hand, the operation area setting processing unit 116 sets a rectangular area A1 surrounding the outline of the hand (see FIG. 9), and sets an operation area A11 larger than the rectangular area A1 using a preset margin (see FIG. 10). For example, the operation area setting processing unit 116 sets the operation area A11 at the position of the preset upper, lower, left, and right margins m1 to m4 (see FIG. 10) in a partial image (W×3 / 5 area) including the detected rectangular area A1 of the hand in the captured image Wp. In addition, the operation area setting processing unit 116 determines whether the hand is a left hand or a right hand, and determines the direction (left or right) in which the operation area A11 is set. The margins m1 to m4 are set to values ​​that enable the mouse cursor M1 to be moved left / right and up / down throughout the entire input area F1 (see FIG. 2) when the user moves his / her hand left / right and up / down throughout the entire operation area A11.

[0044] Also, for example, the operation area setting processor 116 may set an area obtained by adding half the length (W1) / 2 of the width W1 of the rectangular area A1 to the little finger side of the rectangular area A1, adding half the length (H1) / 2 of the height H1 of the rectangular area A1 to the upper side of the rectangular area A1, and adding the length of the height H1 of the rectangular area A1 to the lower side of the rectangular area A1 as the operation area A11. The operation area setting processor 116 is an example of a first setting processor of the present invention.

[0045] Here, since the position of the bottom side of the rectangular region A1 of the hand (reference point P1) corresponds to the position of the mouse cursor M1, if the operation region A11 is set based on the reference point P1, when the user moves his / her hand upward, the mouse cursor M1 cannot be moved to the top end of the display screen 13A unless the hand is moved to a state where the hand protrudes above the operation region A11, which gives the user a sense of discomfort. Therefore, by displaying the operation region A11 above its actual position, the user can move the mouse cursor M1 within the operation region A11 without feeling uncomfortable. Specifically, the display processing unit 111 displays the captured image Wp on the display screen 13A, and also displays a frame image representing the outer frame of the operation region A11 shifted above its actual position with respect to the captured image Wp. 11, for example, the display processing unit 111 displays a frame image representing the outer frame of the operation area A11 on the display screen 13A, shifted upward from the actual position (dotted frame in FIG. 11) by half the width h ((H1) / 2) of the rectangular area A1 in the vertical direction with respect to the captured image Wp. This allows the user to move the mouse cursor M1 according to the movement of his / her hand, with the sensation that the center position of the palm corresponds to the position of the mouse cursor M1.

[0046] The virtual inputtable area setting processor 117 sets a virtual inputtable area F2 (virtual screen) larger than the inputtable area F1. For example, as shown in Fig. 12, when the width of the display screen 13A is W2 and the height of the display screen 13A is H2, the virtual inputtable area setting processor 117 adds half the length ((W2) / 2) of the width W2 of the display screen 13A to each of the right and left sides of the display screen 13A, and adds half the length ((H2) / 2) of the height H2 of the display screen 13A to each of the upper and lower sides of the display screen 13A, and sets the area (area twice the length and width of the inputtable area F1) as the virtual inputtable area F2. The virtual inputtable area setting processor 117 is an example of a second setting processor of the present invention.

[0047] The input processing unit 118 uses the operation area A11, the inputtable area F1, and the virtual inputtable area F2 to calculate an input position on the display screen 13A and executes input processing.

[0048] Specific methods for calculating the input position will be described below, in which a first calculation method, a second calculation method, and a third calculation method are given as specific examples of the calculation method.

[0049] [First calculation method] FIG. 12 shows an example of the first calculation method. As shown in FIG. 12, the display processing unit 111 displays a frame image of the outer frame of the operation area A11 in the captured image Wp. The input processing unit 118 also calculates a position in the operation area A11 corresponding to the reference point P1 in the virtual inputtable area F2 as the input position. Specifically, the input processing unit 118 maps the reference point P1 of the operation area A11 to the virtual inputtable area F2. Then, if the mapped position (the position of the mouse cursor M1) is within the inputtable area F1, the input processing unit 118 determines the position as the input position and displays the mouse cursor M1 at the input position. On the other hand, if the mapped position (the position of the mouse cursor M1) is outside the inputtable area F1, the input processing unit 118 remaps the position within the inputtable area F1, determines the position of the edge of the inputtable area F1 as the input position, and displays the mouse cursor M1 at the input position.

[0050] Thus, in the first calculation method, when the input processing unit 118 maps the operation area A11 to the input area F1, it maps the operation area A11 to a virtual input area F2 that is twice the vertical and horizontal screen size of the actual display screen 13A, and if the mouse cursor M1 is mapped outside the input area F1, it is remapped within the input area F1.

[0051] [Second calculation method] FIG. 13 shows an example of the second calculation method. As shown in FIG. 13, the display processing unit 111 displays a frame image of the outer frame of the virtual operation area B1, which is larger than the operation area A11, in the captured image Wp. For example, the display processing unit 111 displays a frame image of an extended area (virtual operation area B1) in which half the length of the horizontal width of the operation area A11 is added to the right and left sides of the operation area A11, and half the length of the vertical width of the operation area A11 is added to the upper and lower sides of the operation area A11, in the captured image Wp. That is, the display processing unit 111 does not display a frame image of the operation area A11 in the captured image Wp, but displays a frame image of the virtual operation area B1, which is a virtual operation area obtained by extending the operation area A11. This allows the user to move his / her hand while looking at the frame image of the virtual operation area B1 displayed in the captured image Wp.

[0052] The input processing unit 118 calculates a position in the operation area A11 corresponding to the reference point P1 in the inputtable area F1 as the input position. Specifically, the input processing unit 118 maps the reference point P1 of the operation area A11 to the inputtable area F1. If the mapped position (the position of the mouse cursor M1) is within the inputtable area F1, the input processing unit 118 determines the position as an input position and displays the mouse cursor M1 at the input position. On the other hand, if the mapped position (the position of the mouse cursor M1) is outside the inputtable area F1, the input processing unit 118 remaps the position within the inputtable area F1, determines the position of the edge of the inputtable area F1 as the input position, and displays the mouse cursor M1 at the input position.

[0053] In this way, in the second calculation method, a frame image of a virtual operation area B1 that is twice the size of the operation area A11 in both height and width is displayed within the captured image Wp, while the input processing unit 118 maps the operation area A11 to the inputtable area F1, and if the mouse cursor M1 is mapped outside the inputtable area F1, it is remapped within the inputtable area F1.

[0054] [Third calculation method] FIG. 14 shows an example of the third calculation method. As shown in FIG. 14, the display processing unit 111 displays a frame image of the virtual operation area B1 larger than the operation area A11 in the captured image Wp, similarly to the second calculation method. That is, the display processing unit 111 does not display a frame image of the operation area A11 in the captured image Wp, but displays a frame image of the virtual operation area B1. The input processing unit 118 also calculates a position in the virtual operation area B1 corresponding to the reference point P1 in the virtual input possible area F2 as the input position. Specifically, the input processing unit 118 maps the reference point P1 of the virtual operation area B1 to the virtual input possible area F2. Then, if the mapped position (the position of the mouse cursor M1) is within the input possible area F1, the input processing unit 118 determines the position as the input position and displays the mouse cursor M1 at the input position. On the other hand, if the mapped position (the position of the mouse cursor M1) falls outside the input area F1, the input processing unit 118 remaps it within the input area F1, determines the position of the edge of the input area F1 as the input position, and displays the mouse cursor M1 at that input position.

[0055] In this way, in the third calculation method, a frame image of a virtual operation area B1 that is twice the size of the operation area A11 in both height and width is displayed within the captured image Wp, and the input processing unit 118 maps the virtual operation area B1 to the virtual inputtable area F2, and if the mouse cursor M1 is mapped outside the inputtable area F1, it is remapped within the inputtable area F1.

[0056] The input processing unit 118 calculates the input position by any one of the first calculation method, the second calculation method, and the third calculation method described above.

[0057] According to the first calculation method, the operation area A11 is made relatively narrow compared to the input area F1, so that the user can move the mouse cursor M1 with small hand movements, and can move the mouse cursor M1 over the entire display screen 13A by moving the hand within the small area.

[0058] Furthermore, according to the second calculation method, the operation area A11 can be made wider than the inputtable area F1, so that the user can reflect finer movements of the hand in the movement of the mouse cursor M1 than with the first calculation method.

[0059] Furthermore, according to the third calculation method, the operation area A11 can be made an appropriate size relative to the input area F1, so that finer hand movements can be reflected in the movement of the mouse cursor M1 than with the first calculation method, and the mouse cursor M1 can be moved over the entire display screen 13A without moving the hand or arm as much as with the second calculation method.

[0060] Furthermore, according to each of the calculation methods, it is possible to prevent the mouse cursor M1 from going off the display screen 13A or from moving into the captured image Wp (wipe screen) below the display screen 13A, which would cause a malfunction.

[0061] As described above, when the display device 1 detects the user's hand from the image captured by the camera 15, it sets a reference point P1 on the bottom side L1 of the rectangular area A1 of the hand by a predetermined setting method (the first to third setting methods), calculates the input position of the reference point P1 in the input area F1 by a predetermined calculation method (the first to third calculation methods), and performs input processing in accordance with the user's gesture operation at the calculated input position.

[0062] [Display control processing] Hereinafter, the display control process executed by the control unit 11 of the display device 1 will be described with reference to FIG.

[0063] The present invention can be understood as an invention of a display control method (one example of the input method of the present invention) that executes one or more steps included in the display control process, and one or more steps included in the display control process described here may be omitted as appropriate. The steps in the display control process may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the control unit 11 executes each step in the display control process, another embodiment can also be a display control method in which multiple processors execute each step in the display control process in a distributed manner.

[0064] First, in step S1, the control unit 11 determines whether or not a captured image has been acquired from the camera 15. When the control unit 11 acquires a captured image from the camera 15 (S1: Yes), the process proceeds to step S2.

[0065] In step S2, the control unit 11 determines whether or not a user has been detected. If the control unit 11 detects a user (S2: Yes), the process proceeds to step S3.

[0066] In step S3, the control unit 11 determines whether or not the user's face and hands have been detected. If the control unit 11 detects the user's face and hands (S3: Yes), the process proceeds to step S4.

[0067] In step S4, the control unit 11 displays the captured image Wp (wipe screen) on the display screen 13A. Specifically, the control unit 11 displays a partial image (captured image Wp) of a specific range including the detected face and hand of the user among the captured images of a predetermined area captured by the camera 15 at an end (corner) of the display screen 13A (see FIG. 2). The control unit 11 also displays a frame image representing a rectangular area A1 surrounding the user's hand and a frame image (operation area image) of the operation area A11 corresponding to the inputtable area F1 (see FIG. 2) of the display screen 13A, superimposed on the captured image Wp (see FIG. 3). Note that the display processing unit 111 may display the frame image of the operation area A11 shifted upward from the actual position with respect to the captured image Wp (see FIG. 11).

[0068] In step S5, the control unit 11 executes a process of setting a reference point P1 (a reference point setting process). The control unit 11 executes the reference point setting process corresponding to any one of the first setting method, the second setting method, and the third setting method.

[0069] 16 shows a specific example of a reference point setting process corresponding to the first setting method. Specifically, in step S11, the control unit 11 obtains a detection result of a hand detected from a captured image obtained by the camera 15. Next, in step S12, the control unit 11 determines the shape of the hand. For example, when the control unit 11 determines the shape (first shape) of an open hand (paper state), the process proceeds to step S13. On the other hand, when the control unit 11 determines the shape (second shape) of a closed hand (rock state), the process proceeds to step S121.

[0070] In step S13, the control unit 11 sets the center position of the bottom side L1 of the rectangular area A1 (see FIG. 4) as the reference point P1. After step S13, the process proceeds to step S6.

[0071] In step S121, the control unit 11 sets the previously calculated position (see FIG. 6) of the first shape (par state) as the reference point P1. After step S121, the process proceeds to step S6.

[0072] 17 shows a specific example of a reference point setting process corresponding to the second setting method. Specifically, in step S21, control unit 11 obtains a detection result of a hand detected from a captured image obtained by camera 15. Next, in step S22, control unit 11 determines the shape of the hand. For example, when control unit 11 determines the shape (first shape) of an open hand (paper state), the process proceeds to step S23. On the other hand, when control unit 11 determines the shape (second shape) of a closed hand (rock state), the process proceeds to step S221.

[0073] In step S23, the control unit 11 sets a position at a ratio of 1:2 from the end of the bottom side L1 of the rectangular area A1 on the side of the little finger of the user's hand (see FIG. 7) as the reference point P1. After step S23, the process proceeds to step S6.

[0074] In step S221, the control unit 11 sets the previously calculated position of the first shape (par state) as the reference point P1. After step S221, the process proceeds to step S6.

[0075] 18 shows a specific example of a reference point setting process corresponding to the third setting method. Specifically, in step S31, control unit 11 obtains a detection result of a hand detected from a captured image obtained by camera 15. Next, in step S32, control unit 11 determines the shape of the hand. For example, when control unit 11 determines the shape (first shape) of an open hand (paper state), the process proceeds to step S33. On the other hand, when control unit 11 determines the shape (second shape) of a closed hand (rock state), the process proceeds to step S321.

[0076] In step S33, the control unit 11 identifies specific pixels that display the color of the user's hand from among the multiple pixels included in the rectangular area A1, and calculates the total number of pixels of the specific pixels for each pixel row in the column direction from the top side to the bottom side L1 of the rectangular area A1 (see FIG. 8). Next, in step S34, the control unit 11 extracts the pixel row from the bottom side L1 in which the total number of pixels exceeds a threshold value Cth. Next, in step S35, the control unit 11 calculates the coordinates of the minimum position Xmin and the maximum position Xmax of the partial side corresponding to the extracted pixel row. Next, in step S36, the control unit 11 sets the center position Xc of the minimum position Xmin and the maximum position Xmax to the reference point P1. After step S36, the process proceeds to step S6.

[0077] In step S321, the control unit 11 sets the previously calculated position of the first shape (par state) as the reference point P1. After step S321, the process proceeds to step S6.

[0078] In step S6 (see FIG. 15), the control unit 11 executes a process of calculating an input position corresponding to the reference point P1 (input position calculation process). The control unit 11 executes the input position calculation process corresponding to any one of the first calculation method, the second calculation method, and the third calculation method.

[0079] Fig. 19 shows a specific example of the input position calculation process corresponding to the first calculation method. In the first calculation method, the control unit 11 displays a frame image of the operation area A11 on the captured image Wp (see Fig. 12) in step S4 (see Fig. 15).

[0080] In step S41, the control unit 11 sets a virtual inputtable area F2 that is larger than the inputtable area F1 (see FIG. 12). Next, in step S42, the control unit 11 maps the reference point P1 of the operation area A11 onto the virtual inputtable area F2.

[0081] Next, in step S43, the control unit 11 determines whether the position mapped onto the virtual inputtable area F2 is within the inputtable area F1. If the position mapped onto the virtual inputtable area F2 is within the inputtable area F1 (S43: Yes), the process proceeds to step S45. On the other hand, if the position mapped onto the virtual inputtable area F2 is not within the inputtable area F1 (S43: No), the process proceeds to step S44.

[0082] In step S44, the control unit 11 performs remapping within the inputtable area F1. After that, the process proceeds to step S45.

[0083] In step S45, the control unit 11 determines the mapped position as the input position. If the position mapped to the virtual inputtable area F2 is within the inputtable area F1 (S43: Yes), the control unit 11 determines the position as the input position. If the position mapped to the virtual inputtable area F2 is not within the inputtable area F1 (S43: No), the control unit 11 determines the edge position of the inputtable area F1 as the input position. After that, the process proceeds to step S7.

[0084] Fig. 20 shows a specific example of the input position calculation process corresponding to the second calculation method. In the second calculation method, the control unit 11 displays a frame image of a virtual operation area B1 larger than the operation area A11 in the captured image Wp (see Fig. 13) in step S4 (see Fig. 15).

[0085] In step S51, the control unit 11 maps the reference point P1 of the operation area A11 onto the inputtable area F1.

[0086] Next, in step S52, the control unit 11 determines whether the position mapped to the inputtable area F1 is within the inputtable area F1. If the position mapped to the inputtable area F1 is within the inputtable area F1 (S52: Yes), the process proceeds to step S54. On the other hand, if the position mapped to the inputtable area F1 is not within the inputtable area F1 (S52: No), the process proceeds to step S53.

[0087] In step S53, the control unit 11 performs remapping within the inputtable area F1. After that, the process proceeds to step S54.

[0088] In step S54, the control unit 11 determines the mapped position as the input position. If the position mapped to the input area F1 is within the input area F1 (S52: Yes), the control unit 11 determines the position as the input position. If the position mapped to the input area F1 is not within the input area F1 (S52: No), the control unit 11 determines the edge position of the input area F1 as the input position. After that, the process proceeds to step S7.

[0089] Fig. 21 shows a specific example of the input position calculation process corresponding to the third calculation method. In the third calculation method, the control unit 11 displays a frame image of the virtual operation area B1 on the captured image Wp (see Fig. 14) in step S4 (see Fig. 15).

[0090] In step S61, the control unit 11 sets a virtual inputtable area F2 that is larger than the inputtable area F1 (see FIG. 14). Next, in step S62, the control unit 11 maps the reference point P1 of the virtual operation area B1 onto the virtual inputtable area F2.

[0091] Next, in step S63, the control unit 11 determines whether the position mapped onto the virtual inputtable area F2 is within the inputtable area F1. If the position mapped onto the virtual inputtable area F2 is within the inputtable area F1 (S63: Yes), the process proceeds to step S65. On the other hand, if the position mapped onto the virtual inputtable area F2 is not within the inputtable area F1 (S63: No), the process proceeds to step S64.

[0092] In step S64, the control unit 11 performs remapping within the inputtable area F1 After that, the process proceeds to step S65.

[0093] In step S65, the control unit 11 determines the mapped position as the input position. If the position mapped to the virtual inputtable area F2 is within the inputtable area F1 (S63: Yes), the control unit 11 determines the position as the input position. If the position mapped to the virtual inputtable area F2 is not within the inputtable area F1 (S63: No), the control unit 11 determines the edge position of the inputtable area F1 as the input position. After that, the process proceeds to step S7.

[0094] In step S7 (see FIG. 15), the control unit 11 executes input processing based on the determined input position. Specifically, the control unit 11 displays the mouse cursor M1 at the determined input position. Also, for example, if the determined input position is a position on the "Yes" button and the user performs an operation to close their hand while the mouse cursor M1 is positioned on the "Yes" button, the control unit 11 accepts the "Yes" button operation and executes a predetermined process.

[0095] In step S8, the control unit 11 determines whether or not the input operation by the user has ended. If the input operation has ended (S8: Yes), the control unit 11 ends the display control process. If the input operation has not ended (S8: No), the control unit 11 returns to step S1. In this manner, the control unit 11 executes the display control process.

[0096] As described above, the display device 1 according to this embodiment detects a gesture operation by the user's hand and performs input processing at an input position on the display screen. Specifically, the display device 1 acquires a captured image of the user, detects the user's hand from the acquired captured image, and determines the shape of the user's hand. The display device 1 also sets a reference point P1 at a predetermined position on the bottom side L1 of a specific area (rectangular area A1) surrounding the detected user's hand, and when the shape of the detected user's hand is the predetermined shape, executes input processing with the set reference point P1 as the input position.

[0097] According to this configuration, regardless of changes in the hand shape (gesture operation) (for example, from a paper to a rock), the reference point P1 corresponding to the input position can be fixed. This makes it possible to prevent the input position from changing in a direction unintended by the user due to a change in the user's gesture operation.

[0098] The display device 1 according to the present embodiment acquires a captured image of a user, detects the user's hand from the acquired captured image, and sets an operation area A11 for gesture operation that is an area including the detected user's hand and corresponds to an inputtable area F1 in the display screen 13A where input is possible by a gesture operation. The display device 1 also displays the acquired captured image Wp on the display screen 13A, and displays an operation area image (frame image) corresponding to the set operation area A11 superimposed on the captured image Wp. For example, the display device 1 displays a frame image (see FIG. 12) of the operation area A11 or a virtual operation area B1 (see FIG. 13 and FIG. 14) on the captured image Wp.

[0099] According to this configuration, by checking the frame image displayed on the captured image Wp, the user can easily grasp the correspondence between the range in which the user can move his / her hand and the movable range within the display screen 13A of the mouse cursor M1, which moves on the display screen 13A in response to the movement of the user's hand. This makes it possible to improve the operability of the display device 1.

[0100] As shown in the above-described embodiment, the present invention is a display device equipped with a process for executing any one of the first to third setting methods and a process for executing any one of the first to third calculation methods. However, the present invention is not limited to the above-described embodiment, and may be a display device equipped with at least a process for executing any one of the first to third setting methods. Furthermore, the present invention may be a display device equipped with at least a process for executing any one of the first to third calculation methods.

[0101] Furthermore, the display device of the present invention can be constructed by freely combining the embodiments described above, or by appropriately modifying or omitting parts of each embodiment, within the scope of the invention described in each claim. [Explanation of symbols]

[0102] 1:Display device 11: Control section 12: Storage section 13: Display panel 13A:Display screen 14:Operation section 15: Camera 111: Display processing unit 112: Acquisition processing unit 113: Detection processing unit 114: Judgment processing unit 115: Reference point setting processing unit 116: Operation area setting processing unit 117: Virtual input possible area setting processing unit 118: Input processing section

Claims

1. An input device that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, an acquisition processing unit that acquires an image of the user; a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit; a first setting processing unit that sets an operation area of ​​the gesture operation, the operation area including the user's hand detected by the detection processing unit, the operation area corresponding to an inputtable area on the display screen where input can be made by the gesture operation; a second setting processing unit that sets a virtual inputtable area larger than the inputtable area; a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen, and that displays an operation area image representing an outer frame of the operation area set by the first setting processing unit so as to be superimposed on the captured image, and further displays the operation area image shifted upward with respect to the captured image; a reference point setting processing unit that sets a reference point at a predetermined position on a lower side of the specific area including the user's hand detected by the detection processing unit; an input processing unit that calculates, as the input position, a position in the virtual input possible area that corresponds to a position in the operation area that corresponds to the reference point set by the reference point setting processing unit, and executes the input process on the input position; An input device comprising:

2. An input device that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, an acquisition processing unit that acquires an image of the user; a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit; a first setting processing unit that sets an operation area of ​​the gesture operation, the operation area including the user's hand detected by the detection processing unit, the operation area corresponding to an inputtable area on the display screen where input can be made by the gesture operation; a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen, and displays an operation area image representing an outer frame of a virtual operation area larger than the operation area set by the first setting processing unit, superimposed on the captured image, and further shifts the operation area image upward with respect to the captured image; a reference point setting processing unit that sets a reference point at a predetermined position on a lower side of the specific area including the user's hand detected by the detection processing unit; an input processing unit that calculates, as the input position, a position in the input area that corresponds to the reference point set by the reference point setting processing unit and that is associated with the input area, and executes the input process on the input position; An input device comprising:

3. An input device that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, an acquisition processing unit that acquires an image of the user; a detection processing unit that detects the user's hand from the captured image acquired by the acquisition processing unit; a first setting processing unit that sets an operation area of ​​the gesture operation, the operation area including the user's hand detected by the detection processing unit, the operation area corresponding to an inputtable area on the display screen where input can be made by the gesture operation; a second setting processing unit that sets a virtual inputtable area larger than the inputtable area; a display processing unit that displays the captured image acquired by the acquisition processing unit on the display screen, and displays an operation area image representing an outer frame of a virtual operation area larger than the operation area set by the first setting processing unit, superimposed on the captured image, and further shifts the operation area image upward with respect to the captured image; a reference point setting processing unit that sets a reference point at a predetermined position on a lower side of the specific area including the user's hand detected by the detection processing unit; an input processing unit that calculates, as the input position, a position in the virtual operation area corresponding to the reference point set by the reference point setting processing unit, which is associated with the virtual input possible area, and executes the input process on the input position; An input device comprising:

4. the display processing unit displays the operation area image on the display screen, shifted upward by half the width of the specific area in the vertical direction with respect to the captured image. The input device according to any one of claims 1 to 3.

5. The specific area is a rectangular area surrounding the user's hand. The input device according to any one of claims 1 to 3.

6. An input method for detecting a gesture operation by a user's hand and executing an input process at an input position on a display screen, comprising: One or more processors An acquisition step of acquiring a captured image of the user; a detection step of detecting a hand of the user from the captured image acquired in the acquisition step; a first setting step of setting an operation area of ​​the gesture operation, the operation area including the hand of the user detected in the detection step and corresponding to an inputtable area on the display screen where an input can be made by the gesture operation; a second setting step of setting a virtual inputtable area larger than the inputtable area; a display step of displaying, on the display screen, the captured image acquired in the acquisition step, and displaying an operation area image representing an outer frame of the operation area set in the first setting step so as to be superimposed on the captured image, and further displaying the operation area image so as to be shifted upward with respect to the captured image; a reference point setting step of setting a reference point at a predetermined position on a lower side of the specific area including the user's hand detected in the detection step; an input step of calculating, as the input position, a position in the virtual input possible area that corresponds to a position in the operation area that corresponds to the reference point set in the reference point setting step, and executing the input process on the input position; Input method to perform.

7. An input program that detects a gesture operation by a user's hand and executes input processing at an input position on a display screen, An acquisition step of acquiring a captured image of the user; a detection step of detecting a hand of the user from the captured image acquired in the acquisition step; a first setting step of setting an operation area of ​​the gesture operation, the operation area including the hand of the user detected in the detection step and corresponding to an inputtable area on the display screen where an input can be made by the gesture operation; a second setting step of setting a virtual inputtable area larger than the inputtable area; a display step of displaying, on the display screen, the captured image acquired in the acquisition step, and displaying an operation area image representing an outer frame of the operation area set in the first setting step so as to be superimposed on the captured image, and further displaying the operation area image so as to be shifted upward with respect to the captured image; a reference point setting step of setting a reference point at a predetermined position on a lower side of the specific area including the user's hand detected in the detection step; an input step of calculating, as the input position, a position in the virtual input possible area that corresponds to a position in the operation area that corresponds to the reference point set in the reference point setting step, and executing the input process on the input position; An input program for execution by one or more processors.

Images