Display device, display method, and recording medium recording display program

Abstract

A display device, a display method, and a recording medium having a display program recorded thereon, which can prevent erroneous input based on a user's gesture operation in a display device that accepts a user's non-contact input operation on a display screen. The display device comprises: an input operation detection unit that detects an input operation including a gesture operation; a display processing unit that displays an input operation icon on the display screen in a first display mode when a first gesture operation is detected, and displays the input operation icon on the display screen in a second display mode when a second gesture operation is detected; an acceptance processing unit that accepts a move operation when the first gesture operation is detected, and accepts a selection operation when the second gesture operation is detected; and a change processing unit that changes the display mode of the input operation icon between the first display mode and the second display mode in stages or continuously when a third gesture operation is detected.

Description

Technical Field

[0001] The present invention relates to a display device for receiving a non-contact input operation of a user on a display screen, a display method, and a recording medium recording a display program. Background Art

[0002] There is conventionally known a display device that enables input operations (screen operations) such as instruction operations on a display screen of a display panel in a non-contact manner.

[0003] For example, the following interface device is known: when the user makes a one-finger shape with his hand toward the device, an icon with the number "1" is displayed on the display, and the display of the TV as the first menu is emphasized; when the user makes a two-finger shape with his hand, an icon with the number "2" is displayed on the display, and the display of the network as the second menu is emphasized; if the same hand shape is maintained for a certain period of time, the first menu or the second menu is selected.

[0004] Here, in the system, when the user performs input operations on the display screen with his hand in a non-contact manner, when the user performs a first gesture operation with his hand (making the hand into the shape of one finger), an icon with the number "1" is displayed on the display, and when the second gesture operation is performed (making the hand into the shape of two fingers), an icon with the number "2" is displayed on the display, and the user can recognize which gesture operation is being performed.

[0005] However, in ambiguous gesture states such as the intermediate state between the first gesture operation and the second gesture operation, the user cannot recognize the current state of his or her own gesture. For example, suppose that the user slowly changes the shape of his or her hand from the state of the gesture operation in the middle between the first gesture operation and the second gesture operation to the state of the first gesture operation or the second gesture operation. In such a case, in the state of the intermediate gesture operation, the user cannot recognize whether the current gesture operation state is close to the first gesture operation or the second gesture operation and to what extent. In addition, the user cannot grasp the timing of the change from the intermediate gesture operation state to the first gesture operation or the second gesture operation state. Therefore, there is a problem of erroneous input that violates the user's intention, such as performing the first gesture operation or the second gesture operation at a timing earlier or later than the user's intention. Summary of the invention

[0006] An object of the present invention is to provide a display device, a display method, and a recording medium recording a display program, which are capable of preventing erroneous input based on a user's gesture operation in a display device that accepts a user's non-contact input operation on a display screen.

[0007] A display device according to one embodiment of the present invention accepts non-contact input operations of a user on a display screen, and includes: an input operation detection unit, which detects input operations including specified gesture operations of the user; a display processing unit, which, when the input operation detection unit detects a first gesture operation of the user, causes an input operation icon to be displayed on the display screen in a first display mode, and when the input operation detection unit detects a second gesture operation of the user, causes the input operation icon to be displayed on the display screen in a second display mode; an acceptance processing unit, which, when the input operation detection unit detects the first gesture operation, accepts a movement operation of moving the input operation icon according to the first gesture operation, and when the input operation detection unit detects the second gesture operation, accepts a selection operation of selecting a selection object according to the input operation icon; and a change processing unit, which, when the input operation detection unit detects a third gesture operation of the user, causes the display mode of the input operation icon to be changed in stages or continuously between the first display mode and the second display mode.

[0008] The display method involved in other aspects of the present invention accepts a user's non-contact input operation on a display screen, wherein: one or more processors execute the following steps: an input operation detection step, detecting an input operation including a specified gesture operation of the user; a display step, when a first gesture operation of the user is detected, causing an input operation icon to be displayed on the display screen in a first display mode, and when a second gesture operation of the user is detected, causing the input operation icon to be displayed on the display screen in a second display mode; an acceptance step, when the first gesture operation is detected, accepting a movement operation of moving the input operation icon according to the first gesture operation, and when the second gesture operation is detected, accepting a selection operation of selecting a selection object according to the input operation icon; and a change step, when a third gesture operation of the user is detected, causing the display mode of the input operation icon to be changed in stages or continuously between the first display mode and the second display mode.

[0009] Another embodiment of the present invention relates to a recording medium having a display program recorded thereon, wherein the display program accepts a user's non-contact input operation on a display screen, and the display program is used to cause one or more processors to execute the following steps: an input operation detection step, detecting an input operation including a specified gesture operation of the user; a display step, when a first gesture operation of the user is detected, causing an input operation icon to be displayed on the display screen in a first display mode, and when a second gesture operation of the user is detected, causing the input operation icon to be displayed on the display screen in a second display mode; an acceptance step, when the first gesture operation is detected, accepting a movement operation of moving the input operation icon according to the first gesture operation, and when the second gesture operation is detected, accepting a selection operation of selecting a selection object according to the input operation icon; and a change step, when a third gesture operation of the user is detected, causing the display mode of the input operation icon to be changed in stages or continuously between the first display mode and the second display mode.

[0010] According to the present invention, a display device, a display method, and a recording medium recording a display program can be provided that can prevent erroneous input based on a user's gesture operation in a display device that accepts a user's non-contact input operation on a display screen.

[0011] This specification is presented in a simplified form, with appropriate reference to the accompanying drawings, to summarize the concepts described in the following detailed description. This specification is not intended to limit the important features and essential features of the subject matter described in the claims, nor is it intended to limit the scope of the subject matter described in the claims. In addition, the subject matter described in the claims is not limited to implementations that solve some or all of the disadvantages described in any part of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a block diagram showing the configuration of a display device according to an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram showing an example of a virtual operation surface in the display device according to the embodiment of the present invention.

[0014] Figure 3 A diagram showing an example of input operation in a conventional display device.

[0015] Figure 4 A diagram showing an example of input operation in a conventional display device.

[0016] Figure 5 FIG. 1 is a diagram showing how the display form of the input operation icon changes due to gesture operations in the display device according to the embodiment of the present invention.

[0017] Figure 6 FIG. 1 is a diagram showing an example of input operation in the display device according to the embodiment of the present invention.

[0018] Figure 7 FIG. 1 is a diagram showing an example of input operation in the display device according to the embodiment of the present invention.

[0019] Figure 8 This is a flowchart for explaining an example of the sequence of a display control process executed by the display device according to the embodiment of the present invention.

[0020] Fig. 9 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0021] Fig.10 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0022] Fig.11 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0023] Fig.12 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0024] Fig.13 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0025] Fig.14 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0026] Fig.15 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0027] Fig.16 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention.

[0028] Fig.17 This is a diagram showing an example of a method for setting a virtual operation surface in a display device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In addition, the following embodiment is an example of specific implementation of the present invention and does not have the nature of limiting the technical scope of the present invention.

[0030] like Figure 1 As shown, a display device 1 according to an embodiment of the present invention includes a control unit 11 , a storage unit 12 , a display panel 13 , an operation unit 14 , and a motion sensor 15 . Figure 2 Schematic diagram of the display device 1 is shown. The motion sensor 15 is provided on the upper portion of the display panel 13 to detect an input operation of the user.

[0031] The display device 1 receives a non-contact input operation performed by the user on the display screen 13A. For example, if the display device 1 detects the input operation of the user on the virtual operation surface R2, the input processing corresponding to the input operation of the user on the display screen 13A is performed. For example, when the user performs a touch operation on a predetermined position of the virtual operation surface R2, the display device 1 detects the position on the display screen 13A corresponding to the touch position on the virtual operation surface R2 and receives the touch input. The specific structure of the display device 1 is described below.

[0032] The motion sensor 15 has, for example, two cameras and three infrared LEDs, and detects the user's input operation within a predetermined detection range. The motion sensor 15 outputs detection information to the control unit 11. The detection information includes the position coordinates (X coordinate, Y coordinate, Z coordinate) of the detection object (for example, the user's hand, fingertip, stylus, support rod, etc.) based on the motion sensor 15. The motion sensor 15 can detect, for example, the back of the hand (palm), finger joints, finger skeletons, angles, positions, etc. of the user's hand (right hand RH, left hand LH). The motion sensor 15 can apply known technologies.

[0033] The display panel 13 is a display for displaying images, such as a liquid crystal display. The operation unit 14 is an operation device such as a mouse and a keyboard. Alternatively, the operation unit 14 may be composed of a touch panel.

[0034] The storage unit 12 is a non-volatile storage unit such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores various information. Specifically, the storage unit 12 stores data such as the operation area information D1 and the virtual operation surface information D2.

[0035] The operation area information D1 is information indicating the operation area R1 in the display screen 13A of the display panel 13. The operation area R1 is an area in the display screen 13A where the user can perform input operations via the virtual operation surface R2, that is, an area that can accept user input operations. The operation area R1 can be set in the entire area of ​​the display screen 13A, or it can be set in a part of the area of ​​the display screen 13A. For example, when the entire area of ​​the display screen 13A is set in the operation area R1, the coordinates of the four corners of the display screen 13A are included in the operation area information D1 as coordinate information for specifying the operation area R1. (Refer to Figure 2 The operation region information D1 is registered in the storage unit 12 every time the operation region R1 is set or updated.

[0036] The virtual operation surface information D2 is information indicating the area of ​​the virtual operation surface R2 that receives the input operation of the user on the display screen 13A. Specifically, the virtual operation surface R2 corresponds to the operation area R1, and the coordinates of the four corners of the virtual operation surface R2 are specified. (Refer to Figure 2 ) and the coordinates of the specified operation area R1 The virtual operation surface information D2 includes the coordinates of the four corners of the virtual operation surface R2. Every time the virtual operation plane R2 is set or updated, the virtual operation plane information D2 is registered in the storage unit 12. The size and position of the virtual operation plane R2 may be set in advance or may be set by a user's operation.

[0037] In addition, the storage unit 12 stores information for causing the control unit 11 to execute display control processing described later (see Figure 8 ) or other control programs. For example, the display control program is recorded non-temporarily on a computer-readable recording medium such as a CD or a DVD, and is read by a reading device (not shown) such as a CD drive or a DVD drive provided by the display device 1 and stored in the storage unit 12. In addition, the display control program can be distributed from a cloud server and stored in the storage unit 12.

[0038] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that performs various calculations. The ROM is a non-volatile storage unit that pre-stores control programs such as BIOS and OS for the CPU to perform various calculations. The RAM is a volatile or non-volatile storage unit that stores various information and is used as a temporary memory (operating area) for various processes performed by the CPU. In addition, the control unit 11 controls the display device 1 by executing various control programs pre-stored in the ROM or the storage unit 12 through the CPU.

[0039] In addition, in the prior art, when the user performs an input operation with his hand without touching the display screen 13A, the user moves the mouse pointer (input operation icon P1) displayed on the display screen 13A to the desired position by moving his hand. In addition, the prior art, for example, receives a movement operation of the mouse pointer when the user performs a first gesture operation with his hand, and receives a selection operation (click operation) of the mouse pointer when the user performs a second gesture operation with his hand. In this case, for example, although the user intends to perform the first gesture operation, if the state of the first gesture operation becomes blurred, there will be a problem of receiving the second gesture operation against the user's intention. A specific example of this problem is described below.

[0040] Figure 3 and Figure 4 is a diagram for explaining the above-mentioned problems caused by the prior art. Figure 3 In the case where the product ordering screen shown shows the images of the selection buttons "Order", "Return", and "Others" as the images of the selection objects, when the user performs the operation of covering the palm of the right hand RH with the display screen 13A on the virtual operation surface R2 (first gesture operation) and moving it in the up, down, left, and right directions (X direction and Y direction), the input operation icon P1 on the display screen 13A moves according to the movement operation. When the user places an order for the desired product, in order to press the "Order" selection button image displayed on the display screen 13A, the user performs the operation of holding the right hand RH in a state where the input operation icon P1 overlaps with the "Order" selection button image (second gesture operation). In this way, the user can perform the operation of selecting (ordering) the desired product.

[0041] In contrast, Figure 4 As shown, when the user opens the palm of the right hand RH in the virtual operation surface R2 and directly moves it in the up-down-left-right direction (X direction and Y direction), the user sometimes unconsciously makes the posture of the right hand RH close to the posture of a clenched fist due to fatigue of the right hand RH. In this case, if the second gesture operation is detected, an operation that the user does not expect, such as pressing the selection button image of "Return", is accepted. In this way, in the prior art, it is possible that an erroneous input is generated by the user's gesture operation. In response to this, in the display device 1 involved in this embodiment, as shown below, it is possible to prevent erroneous input caused by the user's gesture operation.

[0042] Specifically, if Figure 1As shown in FIG. 1 , the control unit 11 includes various processing units such as an operation surface setting unit 111, an input operation detection unit 112, a display processing unit 113, an acceptance processing unit 114, an input processing unit 115, a calculation processing unit 116, and a change processing unit 117. In addition, by executing various processes according to the display control program by the CPU, the control unit 11 functions as the operation surface setting unit 111, the input operation detection unit 112, the display processing unit 113, the acceptance processing unit 114, the input processing unit 115, the calculation processing unit 116, and the change processing unit 117. In addition, a part or all of the processing units included in the control unit 11 may also be composed of electronic circuits. In addition, the display control program may also be a program for causing a plurality of processors to function as the various processing units.

[0043] The operation surface setting unit 111 sets a virtual operation surface R2 for accepting user input operations. Figure 2 As shown, the operation surface setting unit 111 sets the virtual operation surface R2 at a position separated from the display screen 13A by a predetermined distance in the Z direction. The operation surface setting unit 111 can set the virtual operation surface R2 of a predetermined size at a predetermined position, or can set the virtual operation surface R2 of a size corresponding to the user's operation at a position specified by the user. The details will be described later (refer to [Virtual Operation Surface Setting Method]), but for example, the user can set the virtual operation surface R2 of a desired size at a desired position by performing a predetermined gesture operation.

[0044] The input operation detection unit 112 detects input operations including user's specified gesture operations. Specifically, the input operation detection unit 112 detects the user's input operation in the virtual operation surface R2 set by the operation surface setting unit 111. For example, the input operation detection unit 112 detects the detection coordinates in the virtual operation surface R2 based on the detection information obtained from the motion sensor 15, and calculates the input coordinates in the operation area R1 based on the detection coordinates. The input operation includes a gesture operation for executing a specified input process, a drawing operation for causing the display screen 13A to draw a handwritten image, and the like. The input operation detection unit 112 is an example of an input operation detection unit of the present invention.

[0045] Here, when the ratio of the virtual operation surface R2 to the operation area R1 is set to "W2:W1=H2:H1=a:b" (see Figure 2 ), the input operation detection unit 112 can calculate the input coordinates [dx, dy] based on the detected coordinates [sx, sy] in the virtual operation surface R2 by the formula dx=sx×b / a, dy=sy×b / a. In addition, the display resolution [rx, ry] is Min[dx, dy]=[0, 0], Max[dx, dy]=[dx, dy].

[0046] Here, a specific example in which the input operation detection unit 112 detects the above-mentioned gesture operation will be described.

[0047] For example, the user performs a first gesture operation when he wants to move the input operation icon P1 (e.g., a mouse cursor image) displayed on the display screen 13A. The first gesture operation is, for example, an operation in which the user opens the right hand RH in front of the display screen 13A (e.g., an operation in which the right hand RH is made into the shape of "paper (the paper in rock-paper-scissors)"). In addition, for example, the user performs a second gesture operation when he wants to select (click) a selection object using the input operation icon P1 displayed on the display screen 13A. The second gesture operation is, for example, an operation in which the user holds the right hand RH in front of the display screen 13A (e.g., an operation in which the right hand RH is made into the shape of "rock (the rock in rock-paper-scissors)"). The input operation detection unit 112 detects the first gesture operation and the second gesture operation.

[0048] In addition, the combination of the first gesture operation and the second gesture operation is not limited to the above examples. For example, the first gesture operation may be a posture of straightening the index finger of the right hand RH, and the second gesture operation may be a posture of holding the right hand RH. In addition, the first gesture operation may be a posture of opening the right hand RH and standing it vertically, and the second gesture operation may be a posture of opening the right hand RH and laying it down 90 degrees. In addition, the first gesture operation may be a posture of opening the right hand RH, and the second gesture operation may be a posture of straightening the thumb of the right hand RH or a posture of the OK sign of the right hand RH.

[0049] In addition, the input operation detection unit 112 detects a third gesture operation that is similar to the first gesture operation and the second gesture operation. The third gesture operation is, for example, a gesture operation in the middle of changing from the first gesture operation to the second gesture operation, for example, a gesture of a shape between the shape of "cloth" and the shape of "fist" of the right hand RH. For example, after the user applies force to the right hand RH to form a "cloth" shape (first gesture operation), if the force of the right hand RH becomes weak, the shape of "cloth" collapses and approaches the shape of "stone" (second gesture operation).

[0050] The input operation detection unit 112 detects the first gesture operation, the second gesture operation, and the third gesture operation based on finger information of at least any one of the skeleton, angle, and position of the user's finger. In addition, the input operation detection unit 112 detects the first gesture operation, the second gesture operation, and the third gesture operation based on the degree of grip Gn (described later) corresponding to the finger information. For example, the input operation detection unit 112 pre-sets a first threshold of the degree of grip Gn corresponding to the first gesture operation, and detects the first gesture operation when the detected degree of grip Gn is less than the first threshold. In addition, for example, the input operation detection unit 112 pre-sets a second threshold of the degree of grip Gn corresponding to the second gesture operation (wherein the first threshold < the second threshold), and detects the second gesture operation when the detected degree of grip Gn is greater than the second threshold. In addition, the input operation detection unit 112 detects the third gesture operation when the detected degree of grip Gn is greater than the first threshold and less than the second threshold.

[0051] In addition, setting information for associating a predetermined gesture operation with an operation content corresponding to the gesture operation is pre-stored in the storage unit 12. For example, a moving operation of the input operation icon P1 is associated with the first gesture operation and the third gesture operation, and a selection operation (click operation) of the input operation icon P1 is associated with the second gesture operation. The predetermined gesture operation may also include a gesture operation for setting a virtual operation surface R2 (refer to the [virtual operation surface setting method] described later). In addition, the above-mentioned predetermined gesture operation may also be set individually for each user using the display device 1.

[0052] The display processing unit 113 causes the display screen 13A of the display panel 13 to display various information. For example, the display processing unit 113 causes the display screen 13A to display an image of the input operation icon P1 (mouse cursor image), an image of a selection object that can be selected by the user operating the input operation icon P1 (for example, Figure 3 The images of the selection buttons for "Place Order", "Return", "Others", etc.

[0053] In addition, when the input operation detection unit 112 detects the first gesture operation of the user, the display processing unit 113 displays the input operation icon P1 on the display screen 13A in the first display mode. In addition, when the input operation detection unit 112 detects the second gesture operation of the user, the display processing unit 113 displays the input operation icon P1 on the display screen 13A in the second display mode.

[0054] For example, Figure 5As shown, when the input operation detection unit 112 detects the first gesture operation (the "paper" gesture) of the user, the display processing unit 113 displays the input operation icon P1 in the first size (size) and the first color (white). On the other hand, when the input operation detection unit 112 detects the above-mentioned second gesture operation (the "stone" gesture) of the user, the display processing unit 113 displays the input operation icon P1 in a second size smaller than the first size and in a second color (black). The display processing unit 113 is an example of the display processing unit of the present invention.

[0055] The acceptance processing unit 114 accepts input operations performed by the user through non-contact on the display screen 13A. Specifically, when the input operation detection unit 112 detects the first gesture operation, the acceptance processing unit 114 accepts a movement operation of moving the input operation icon P1 according to the first gesture operation, and when the input operation detection unit 112 detects the second gesture operation, the acceptance processing unit 114 accepts a selection operation of selecting a selection object through the input operation icon P1. In addition, when the input operation detection unit 112 detects the above-mentioned third gesture operation, the acceptance processing unit 114 accepts a movement operation of moving the input operation icon P1 according to the third gesture operation. For example, the acceptance processing unit 114 refers to the setting information stored in the storage unit 12 and accepts an input operation corresponding to a specified gesture operation. The acceptance processing unit 114 is an example of an acceptance processing unit of the present invention.

[0056] The input processing unit 115 performs input processing corresponding to the input operation accepted by the acceptance processing unit 114. Specifically, when the input receiving unit 115 receives the moving operation and the selecting operation, it performs input processing corresponding to the input operation of the user on the display screen 13A. For example, when the acceptance processing unit 114 accepts the moving operation, the input processing unit 115 moves the input operation icon P1 on the display screen 13A. In addition, for example, when the acceptance processing unit 114 accepts the selecting operation, the input processing unit 115 performs processing corresponding to the selected object on the display screen 13A. For example, when the acceptance processing unit 114 accepts the operation of selecting the selection button image of "Place an Order" on the display screen 13A, the input processing unit 115 performs order processing assigned to the selection button image of "Place an Order". In addition, when the acceptance processing unit 114 accepts the drawing operation, the input processing unit 115 performs drawing processing of drawing a handwritten image on the display screen 13A. The input processing unit 115 is an example of the input processing unit of the present invention.

[0057] When the input operation detection unit 112 detects the third gesture operation of the user, the calculation processing unit 116 calculates the similarity between the third gesture operation and the first gesture operation and the second gesture operation. The calculation processing unit 116 calculates the similarity based on the finger information. The similarity is an indicator indicating the degree of grip Gn of the user.

[0058] For example Figure 5 As shown, the grip degree Gn of the first gesture operation (operation of opening the hand) is set to "0" (or 0 to 0.1), and the grip degree Gn of the second gesture operation (operation of shaking hands) is set to "1" (or 0.9 to 1). The information of the grip degrees Gn corresponding to the first gesture operation and the second gesture operation is pre-stored in the storage unit 12. Here, when the input operation detection unit 112 detects the third gesture operation of the user, the calculation processing unit 116 calculates the grip degree Gn corresponding to the third gesture operation based on the finger information of at least any one of the skeleton, angle and position of the user's finger. The closer the grip degree Gn is to "0", the closer the third gesture operation is to the first gesture operation, and the closer the grip degree Gn is to "1", the closer the third gesture operation is to the second gesture operation. The calculation processing unit 116 calculates the similarity A1 between the first gesture operation and the third gesture operation and the similarity A2 between the second gesture operation and the third gesture operation based on the grip degree Gn corresponding to the third gesture operation.

[0059] The change processing unit 117 changes the display mode of the input operation icon P1 based on the approximation calculated by the calculation processing unit 116. In other words, the change processing unit 117 changes the display mode of the input operation icon P1 based on the gripping degree Gn calculated by the calculation processing unit 116. Specifically, the change processing unit 117 changes the size of the input operation icon P1 based on the approximation A1, A2 or the gripping degree Gn calculated by the calculation processing unit 116.

[0060] For example, the greater the similarity A1 between the first gesture operation and the third gesture operation, or the smaller the similarity A2 between the second gesture operation and the third gesture operation, the change processing unit 117 makes the size of the input operation icon P1 closer to the size of the input operation icon P1 corresponding to the first gesture operation (first display mode).

[0061] In addition, for example, the larger the approximation A2 between the second gesture operation and the third gesture operation, or the smaller the approximation A1 between the first gesture operation and the third gesture operation, the change processing unit 117 makes the size of the input operation icon P1 closer to the size of the input operation icon P1 corresponding to the second gesture operation (the second display mode). That is, the smaller the degree of grip Gn of the user's hand, the closer the size of the input operation icon P1 is to the first display mode, and the larger the degree of grip Gn of the user's hand, the closer the change processing unit 117 is to the second display mode. In this way, the change processing unit 117 changes the size of the input operation icon P1 to a size corresponding to the approximation A1, A2 or the degree of grip Gn.

[0062] As another embodiment, the change processing unit 117 may change the color of the input operation icon P1 based on the approximations A1 and A2 or the degree of gripping Gn calculated by the calculation processing unit 116 .

[0063] For example, the greater the similarity A1 between the first gesture operation and the third gesture operation, or the smaller the similarity A2 between the second gesture operation and the third gesture operation, the change processing unit 117 makes the color of the input operation icon P1 closer to the color of the input operation icon P1 corresponding to the first gesture operation (first display mode).

[0064] In addition, for example, the greater the similarity A2 between the second gesture operation and the third gesture operation, or the smaller the similarity A1 between the first gesture operation and the third gesture operation, the change processing unit 117 makes the color of the input operation icon P1 closer to the color of the input operation icon P1 corresponding to the second gesture operation (second display mode). That is, the smaller the degree of grip Gn of the user's hand, the closer the color of the input operation icon P1 by the change processing unit 117 is to the first display mode, and the greater the degree of grip Gn of the user's hand, the closer the change processing unit 117 is to the second display mode. In this way, the change processing unit 117 changes the color of the input operation icon P1 to the color corresponding to the similarity A1, A2 or the degree of grip Gn. For example, the greater the similarity A2, the change processing unit 117 can change the input operation icon P1 from black to red. In addition, in this case, the change processing unit 117 can share the size of the input operation icon P1 through the first gesture operation, the second gesture operation, and the third gesture operation.

[0065] As another embodiment, the change processing unit 117 may change the shape of the input operation icon P1 based on the approximations A1 and A2 or the degree of gripping Gn calculated by the calculation processing unit 116 .

[0066] For example, the greater the similarity A1 between the first gesture operation and the third gesture operation, or the smaller the similarity A2 between the second gesture operation and the third gesture operation, the change processing unit 117 makes the shape of the input operation icon P1 closer to the shape of the input operation icon P1 corresponding to the first gesture operation (first display mode).

[0067] In addition, for example, the greater the approximation A2 between the second gesture operation and the third gesture operation, or the smaller the approximation A1 between the first gesture operation and the third gesture operation, the change processing unit 117 makes the shape of the input operation icon P1 closer to the shape of the input operation icon P1 corresponding to the second gesture operation (the second display mode). That is, the smaller the degree of grip Gn of the user's hand, the closer the shape of the input operation icon P1 by the change processing unit 117 is to the first display mode, and the greater the degree of grip Gn of the user's hand, the closer the change processing unit 117 is to the second display mode. In this way, the change processing unit 117 changes the shape of the input operation icon P1 to a shape corresponding to the approximation A1, A2 or the degree of grip Gn. For example, the greater the approximation A2, the change processing unit 117 changes the shape of the input operation icon P1 from the "O" shape to the "□" shape.

[0068] Furthermore, the change processing unit 117 may simultaneously change at least two elements of the size, color, and shape of the input operation icon P1 based on the approximations A1 and A2 or the degree of gripping Gn calculated by the calculation processing unit 116 .

[0069] In addition, if Figure 5 As shown in FIG. 1 , the change processing unit 117 changes the display mode of the input operation icon P1 between the first display mode and the second display mode in a stepwise or continuous manner based on the approximations A1, A2 or the gripping degree Gn calculated by the calculation processing unit 116. Thus, the input operation icon P1 changes smoothly in appearance on the display screen 13A.

[0070] According to the above structure, for example Figure 6 As shown, when the third gesture operation corresponding to the user's right hand RH is detected, the size of the input operation icon P1 displayed on the display screen 13A changes to a size corresponding to the approximation A1, A2 of the third gesture operation. Therefore, the user can recognize that his or her right hand RH is approaching the shape (shape of a handshake) corresponding to the second gesture operation (selection operation) from the shape corresponding to the first gesture operation (shape of an open hand). As a result, the user can restore the right hand RH to the shape corresponding to the first gesture operation again, thereby preventing the user from inputting an operation that is not expected (here, the selection operation of "return") by mistake. And, as Figure 7As shown, the user moves the input operation icon P1 to the target position ("Place an order") while maintaining the shape of the right hand RH in the state of the first gesture operation, and performs the second gesture operation at the position where the input operation icon P1 and "Place an order" overlap, thereby enabling the target operation ("Place an order" selection operation).

[0071] [Display control processing]

[0072] Below, refer to Figure 8 , the display control process performed by the control unit 11 of the display device 1 is described.

[0073] In addition, the present invention can be understood as an invention of a display control method (an example of the display method of the present invention) that executes one or more steps included in the display control process, and one or more steps included in the display control process described herein may be appropriately omitted. In addition, it may also be that the execution order of the steps in the display control process is different within the range of producing the same effect. Moreover, the case where the control unit 11 executes the steps in the above-mentioned display control process is used as an example for explanation, but a display control method in which multiple processors execute the steps in the display control process in a distributed manner is also considered as another embodiment.

[0074] First, in step S11, the control unit 11 determines whether the user's input operation is detected. Specifically, the control unit 11 detects the user's input operation in the virtual operation surface R2. For example, the control unit 11 detects the detection coordinates in the virtual operation surface R2 based on the detection information obtained from the motion sensor 15, and calculates the input coordinates in the operation area R1 of the display screen 13A based on the detection coordinates. In the case where the control unit 11 detects the input operation (S11: Yes), the processing is transferred to step S12. In the case where the control unit 11 does not detect the input operation (S11: No), the processing is transferred to step S16.

[0075] In step S12, the control unit 11 determines whether the first gesture operation of the user is detected. For example, the control unit 11 determines whether an operation of opening the right hand RH of the user (an operation of making the right hand RH into the shape of "cloth") is detected based on the finger information. In the case where the control unit 11 detects the first gesture operation of the user (S12: Yes), the process transfers to step S13. On the other hand, in the case where the control unit 11 does not detect the first gesture operation of the user (S12: No), the process transfers to step S21. Steps S11 and S12 are an example of the input operation detection step of the present invention.

[0076] Next, in step S13, the control unit 11 displays the input operation icon P1 on the display screen 13A in the first display format. For example, the control unit 11 displays the input operation icon P1 on the display screen 13A in the first size and the first color (white) (see Figure 5 ). Step S13 is an example of the display step of the present invention.

[0077] Next, in step S14, the control unit 11 accepts the movement operation of the input operation icon P1. For example, when the user moves in the vertical and horizontal directions (X and Y directions) in the virtual operation surface R2 with the right hand RH open (the first gesture operation), the control unit 11 accepts the movement operation. Step S14 is an example of the acceptance step of the present invention.

[0078] Next, in step S15, the control unit 11 executes input processing corresponding to the user's movement operation on the display screen 13A. For example, the control unit 11 moves the input operation icon P1 displayed on the display screen 13A to follow the movement of the user's right hand RH.

[0079] Next, in step S16, the control unit 11 determines whether the input operation is completed. If the input operation is completed (S16: Yes), the control unit 11 ends the display control process. If the input operation is not completed (S16: No), the control unit 11 returns to step S11.

[0080] If the process returns to step S11, the control unit 11 executes the processes of steps S11 and S12 again. In step S12, if the control unit 11 does not detect the first gesture operation (S12: No), the process transfers to step S21. In step S21, the control unit 11 determines whether the second gesture operation (operation of closing the right hand RH) of the user is detected. If the control unit 11 does not detect the second gesture operation (S21: No), the process transfers to step S31.

[0081] Here, as an example, a case where the user performs the third gesture operation unintentionally after performing the first gesture operation will be described.

[0082] In step S31, the control unit 11 determines whether the user's third gesture operation (operation to make the right hand RH into a shape between "cloth" and "stone") is detected. For example, when the user moves the input operation icon P1 with the right hand RH open and gradually closes the right hand RH, the control unit 11 detects the third gesture operation based on the finger information (S31: Yes). Step S31 is an example of the input operation detection step of the present invention.

[0083] If the third gesture operation is detected (S31: Yes), the control unit 11 calculates the approximations A1 and A2 of the third gesture operation and the first and second gesture operations in step S32. For example, the control unit 11 calculates the approximations A1 and A2 based on the grip degree Gn of the user's hand.

[0084] Specifically, the control unit 11 calculates the similarity A1 between the first gesture operation and the third gesture operation and the similarity A2 between the second gesture operation and the third gesture operation according to the gripping degree Gn corresponding to the third gesture operation (refer to Figure 5 ). Step S32 is an example of a calculation step of the present invention.

[0085] Next, in step S33, the control unit 11 changes the display mode of the input operation icon P1 based on the calculated approximations A1 and A2. In other words, the control unit 11 changes the display mode of the input operation icon P1 based on the calculated grip degree Gn. Specifically, the control unit 11 changes at least one of the size, color, and shape of the input operation icon P1 between the first display mode and the second display mode in a phased or continuous manner based on the calculated approximations A1 and A2 or the grip degree Gn. Step S33 is an example of a change step of the present invention.

[0086] After that, the process transfers to step S14, and the control unit 11 accepts the movement operation of the input operation icon P1. That is, for example, when the user moves the right hand RH in the up, down, left, and right directions (X direction and Y direction) on the virtual operation surface R2 with the right hand RH half closed (the third gesture operation), the control unit 11 accepts the movement operation.

[0087] Thus, when the user performs the third gesture operation between the first gesture operation and the second gesture operation, the control unit 11 moves the input operation icon P1 displayed on the display screen 13A so as to follow the movement of the user's right hand RH (see Figure 6 ).

[0088] Thereafter, for example, when the user performs an operation of closing the right hand RH (the second gesture operation) (S21: Yes), the control unit 11 causes the input operation icon P1 to be displayed on the display screen 13A in the second display format. In step S22, for example, the control unit 11 causes the input operation icon P1 to be displayed on the display screen 13A in the second size and in the second color (black) (see Figure 5 ). Step S21 is an example of an input operation detection step of the present invention, and step S22 is an example of a display step of the present invention.

[0089] Next, in step S23, the control unit 11 accepts a selection operation of the input operation icon P1. For example, when the user closes the right hand RH on the virtual operation surface R2, the control unit 11 accepts the selection operation. If the control unit 11 accepts the selection operation, then in step S15, the control unit 11 executes a process corresponding to the selection object on the display screen 13A. For example, Figure 7 As shown, when the control unit 11 receives an operation of selecting the "order" selection button image on the display screen 13A, it executes the order processing assigned to the "order" selection button image. Step S23 is an example of the receiving step of the present invention.

[0090] In addition, in step S31, when the control unit 11 does not accept the third gesture operation (S31: No), in step S15, the control unit 11 executes the first gesture operation. Input processing corresponding to input operations other than the third gesture operation described above For example, when a drawing operation of drawing a handwritten image is received from the user, the control unit 11 executes a drawing process of drawing a handwritten image on the display screen 13A.

[0091] As described above, the control unit 11 executes the display control process.

[0092] As described above, the display device 1 according to the present embodiment accepts a non-contact input operation of the user on the display screen 13A. In addition, the display device 1 detects an input operation including a predetermined gesture operation of the user. In addition, when the first gesture operation of the user is detected, the display device 1 displays the display input operation icon P1 on the display screen 13A in the first display mode, and when the second gesture operation of the user is detected, the display input operation icon P1 is displayed on the display screen 13A in the second display mode. In addition, when the first gesture operation is detected, the display device 1 accepts a moving operation of moving the input operation icon P1 according to the first gesture operation, and when the second gesture operation is detected, the display device 1 accepts a selection operation of selecting a selection object through the input operation icon P1. In addition, when the third gesture operation of the user is detected, the display device 1 changes the display mode of the input operation icon P1. For example, the display mode of the input operation icon P1 is changed in stages or continuously between the first display mode and the second display mode. Furthermore, the display device 1 calculates the similarity between the third gesture operation and the first gesture operation and the second gesture operation, and changes the display form of the input operation icon P1 according to the calculated similarity.

[0093] Specifically, when the display device 1 detects an operation of opening the user's hand (first gesture operation), the input operation icon P1 is displayed in a larger size, and the input operation icon P1 moves to follow the movement of the user's hand. When the display device 1 detects an operation of closing the user's hand (second gesture operation), the input operation icon P1 is displayed in a smaller size, and a selection operation based on the input operation icon P1 is performed. In addition, when the display device 1 detects a state between the state of opening the user's hand and the state of closing the user's hand (third gesture operation), the display device 1 displays the input operation icon P1 in a size corresponding to the degree of clenching Gn (the approximation) of the user's hand, and the input operation icon P1 moves to follow the movement of the user's hand.

[0094] According to this structure, the user can recognize that his / her gesture operation has approached the shape (shape of handshake) corresponding to the second gesture operation (selection operation) from the shape corresponding to the first gesture operation (shape of open hand). Thus, the user can return to the shape corresponding to the first gesture operation again, thereby preventing erroneous input caused by unintentional operation by the user.

[0095] [How to set up the virtual operation surface]

[0096] The operation surface setting unit 111 can set the virtual operation surface R2 of a desired size to a desired position by gesture operation according to the user's predetermined setting.

[0097] Specifically, the operation surface setting unit 111 detects the user's setting gesture operation. Specifically, the operation surface setting unit 111 detects the setting gesture operation based on the detection information obtained from the motion sensor 15. For example, the operation surface setting unit 111 determines the shape of the user's hand based on the coordinate information included in the detection information, and determines the corresponding gesture operation among the multiple gesture operations pre-registered in the storage unit 12.

[0098] When the operation surface setting unit 111 detects a predetermined first setting gesture operation by the user, the area corresponding to the first setting gesture operation is set as the virtual operation surface R2 that accepts the input operation of the user on the display screen 13A. In addition, the operation surface setting unit 111 may set the virtual operation surface R2 when the first setting gesture operation is performed continuously for a predetermined time. The first setting gesture operation is, for example, an operation of covering the display screen 13A with the palms of the left hand LH and the right hand RH. That is, the first setting gesture operation is a setting operation for the user to set the virtual operation surface R2.

[0099] For example Fig. 9As shown, when the user covers the display screen 13A with the palm of the left hand LH at an arbitrary position on the upper left, and covers the display screen 13A with the palm of the right hand RH at an arbitrary position on the lower right, the operation surface setting unit 111 detects the coordinates Ph1 of the left hand LH, the coordinates Ph2 of the right hand RH, and the first setting gesture operation of the cover of the left hand LH and the right hand RH based on the detection information obtained from the motion sensor 15. When the operation surface setting unit 111 detects the first setting gesture operation, it sets the virtual operation surface R2 according to the coordinates Ph1 of the left hand LH and the coordinates Ph2 of the right hand RH.

[0100] For example, Fig. 9 As shown in FIG. 1 , the operation surface setting unit 111 sets a virtual operation surface R2 of a rectangle whose diagonal line is a line connecting the position of the left hand LH (coordinate Ph1) and the position of the right hand RH (coordinate Ph2). Specifically, the operation surface setting unit 111 calculates the coordinates of the corners of the rectangle based on the coordinates Ph1 of the left hand LH and the coordinates Ph2 of the right hand RH. (Refer to Figure 2 ) and set the virtual operation surface R2.

[0101] For example, operation surface setting unit 111 sets virtual operation surface R2 at a predetermined distance L1 from display screen 13A. Predetermined distance L1 corresponds to coordinate Ph1 (Z coordinate) of left hand LH and coordinate Ph2 (Z coordinate) of right hand RH.

[0102] In addition, for example, the operation surface setting unit 111 may set the virtual operation surface R2 having the same aspect ratio as the aspect ratio of the display screen 13A. Fig.10 As shown, operation surface setting unit 111 sets virtual operation surface R2 such that the aspect ratio (H1:W1) of display screen 13A and the aspect ratio (H2:W2) of virtual operation surface R2 are the same (H1:W1=H2:W2).

[0103] In this way, the size of the display screen 13A (operation area R1) and the size of the virtual operation surface R2 can be the same or different. Here, when the virtual operation surface R2 is smaller than the operation area R1, it is suitable for operating the large display panel 13 at the user's hand. On the other hand, when the virtual operation surface R2 is larger than the operation area R1, it is suitable for operating at a position separated from the small display panel 13.

[0104] In addition, if Fig.11As shown, the operation surface setting unit 111 can also set a virtual operation surface R2 with a predetermined angle d1 that is not parallel to the display screen 13A. That is, the virtual operation surface R2 can be set in a direction inclined relative to the display screen 13A. For example, the operation surface setting unit 111 sets the predetermined angle d1 based on the coordinate Ph1 (Z coordinate) of the left hand LH and the coordinate Ph2 (Z coordinate) of the right hand RH. Thus, the user can perform input operations on the display screen 13A from an inclined direction. In addition, the operation surface setting unit 111 can also display information of the specified angle d1 on the display screen 13A. Thus, the user can grasp the angle (degree of inclination) of the virtual operation surface R2 relative to the display screen 13A.

[0105] In addition, the operation surface setting unit 111 may set a virtual operation surface R2 corresponding to a part of the area on the display screen 13A. Fig.12 As shown in FIG. 1 , the operation surface setting unit 111 sets a virtual operation surface R2 corresponding to the operation area R1 of a portion (left area) of the display screen 13A. The position and size of the operation area R1 can be set by the user's setting operation. In addition, when setting the virtual operation surface R2, the operation surface setting unit 111 can also Fig.13 As shown, the display screen 13A is caused to display the object image T1 indicating the operation region R1 so that the user who sets the virtual operation plane R2 can easily understand the operation region R1.

[0106] Furthermore, the operation surface setting unit 111 can set the virtual operation surface R2 associated with the operation area R1 of the display screen 13A based on the coordinates corresponding to the first setting gesture operation using known coordinate transformation (projection transformation, affine transformation, etc.).

[0107] In addition, the operation surface setting unit 111 may also perform a process of adjusting the set virtual operation surface R2. Specifically, when a second setting gesture operation specified by the user is detected after the virtual operation surface R2 is set, the operation surface setting unit 111 changes at least one of the size and position of the virtual operation surface R2 based on the second setting gesture operation. The second setting gesture operation is, for example, an indication operation of the right hand RH (see Fig.14 ).

[0108] For example Fig.14As shown, after the virtual operation surface R2 is set, when the user performs an indication operation on the display screen 13A with the right hand RH, the operation surface setting unit 111 detects the coordinate Ph3 of the right hand RH and the second setting gesture operation of the indication operation performed by the right hand RH based on the detection information obtained from the motion sensor 15. When the second setting gesture operation is detected, the operation surface setting unit 111 sets the virtual operation surface R2 to be movable according to the coordinate Ph3 of the right hand RH, and accepts the movement operation of the virtual operation surface R2 performed by the user. For example, when the user maintains the indication state of the right hand RH and moves it to the left, the operation surface setting unit 111 moves the virtual operation surface R2 to the left by an amount corresponding to the movement amount of the right hand RH. That is, the operation surface setting unit 111 sets the virtual operation surface R2 to the coordinate Ph3 of the right hand RH after the movement.

[0109] In addition, for example Fig.15 As shown, after the virtual operation surface R2 is set, when the user performs an operation of making a fist with the right hand RH while the left hand LH is covered, the operation surface setting unit 111 detects the coordinate Ph1 of the left hand LH, the coordinate Ph2 of the right hand RH, and the second setting gesture operation in which the left hand LH is covered and the right hand RH is made into a fist based on the detection information obtained from the motion sensor 15. When the second setting gesture operation is detected, the operation surface setting unit 111 sets the size of the virtual operation surface R2 in a changeable manner according to the coordinate Ph2 of the right hand RH, and receives the size change operation of the virtual operation surface R2 performed by the user. For example, when the user moves to the lower right direction with the right hand RH made into a fist, the operation surface setting unit 111 expands the size (area) of the virtual operation surface R2 by an amount corresponding to the movement amount of the right hand RH. That is, the operation surface setting unit 111 sets the virtual operation surface R2 defined by the coordinate Ph1 of the left hand LH and the coordinate Ph2 of the right hand RH after the movement.

[0110] exist Fig.16In the example shown, an example is shown in which after the virtual operation surface R2 is set, the user performs a fist operation of the left hand LH in a state where the right hand RH is blocked. In this case, the operation surface setting unit 111 detects the coordinate Ph1 of the left hand LH, the coordinate Ph2 of the right hand RH, and the second setting gesture operation of blocking the right hand RH and making a fist with the left hand LH based on the detection information obtained from the motion sensor 15. When the second setting gesture operation is detected, the operation surface setting unit 111 sets the size of the virtual operation surface R2 in a changeable manner according to the coordinate Ph1 of the left hand LH, and accepts the user's operation to change the size of the virtual operation surface R2. For example, when the user moves the left hand LH in the lower right direction while keeping the fist, the operation surface setting unit 111 reduces the size (area) of the virtual operation surface R2 by an amount corresponding to the amount of movement of the left hand LH. That is, the operation surface setting unit 111 sets the virtual operation surface R2 specified by the coordinate Ph2 of the right hand RH and the coordinate Ph1 of the left hand LH after the movement.

[0111] Furthermore, when the second setting gesture operation is detected after the virtual operation plane R2 is set, the operation plane setting unit 111 may display the object image T2 representing the virtual operation plane R2 on the display screen 13A according to the second setting gesture operation. Fig.17 An example of the object image T2 is shown, which shows the virtual operation surface R2 after the size is changed. According to this configuration, the user can intuitively grasp the size, position, etc. of the virtual operation surface R2 after the change.

[0112] Furthermore, the display device of the present invention may be configured by freely combining the above-described embodiments within the scope of the invention described in the claims, or by appropriately omitting a modification or a part of the embodiments.

[0113] The scope of the present invention is not limited to the above content, but is defined by the description of the claims, so it can be considered that the embodiments described in this specification are only illustrative, not limiting. Therefore, all changes that do not depart from the scope and limits of the claims, as well as contents that are equivalent to the scope and limits of the claims, are included in the scope of the claims.

Claims

1. A display device that accepts a non-contact input operation of a user on a display screen, characterized in that: include: an input operation detection unit configured to detect an input operation including a predetermined gesture operation of the user; a display processing unit configured to display an input operation icon on the display screen in a first display mode when the input operation detection unit detects a first gesture operation of the user, and to display the input operation icon on the display screen in a second display mode when the input operation detection unit detects a second gesture operation of the user; an acceptance processing unit that accepts a move operation of moving the input operation icon according to the first gesture operation when the input operation detection unit detects the first gesture operation, and accepts a selection operation of selecting a selection object according to the input operation icon when the input operation detection unit detects the second gesture operation; as well as A change processing unit, wherein when the input operation detection unit detects a third gesture operation of the user, the change processing unit causes the display mode of the input operation icon to be changed in stages or continuously between the first display mode and the second display mode, wherein the third gesture operation is a gesture operation in the middle of the change from the first gesture operation to the second gesture operation.

2. The display device according to claim 1, wherein: The change processing unit changes the size of the input operation icon between the first display mode and the second display mode in a stepwise or continuous manner.

3. The display device according to claim 1 or 2, characterized in that: The change processing unit changes the color of the input operation icon between the first display mode and the second display mode in a stepwise or continuous manner.

4. The display device according to claim 1 or 2, characterized in that: The change processing unit changes the shape of the input operation icon between the first display mode and the second display mode in a stepwise or continuous manner.

5. The display device according to claim 1 or 2, characterized in that: When the third gesture operation is detected, the acceptance processing unit accepts a move operation of moving the input operation icon in accordance with the third gesture operation.

6. The display device according to claim 1 or 2, characterized in that: An input processing unit is further provided, and when the acceptance processing unit accepts the movement operation and the selection operation, the input processing unit executes input processing corresponding to the input operation of the user on the display screen.

7. The display device according to claim 1 or 2, characterized in that: The device further comprises a calculation processing unit, which, when the input operation detection unit detects the third gesture operation of the user, calculates the similarity between the third gesture operation and the first gesture operation and the second gesture operation, The change processing unit changes the display mode of the input operation icon between the first display mode and the second display mode stepwise or continuously based on the approximation calculated by the calculation processing unit.

8. The display device according to claim 7, characterized in that The input operation detection unit detects the first gesture operation, the second gesture operation, and the third gesture operation based on finger information of at least one of a skeleton, an angle, and a position of a finger of the user, The calculation processing unit calculates the degree of approximation based on the finger information.

9. The display device according to claim 7, wherein: The first gesture operation is an operation of opening the user's hand. The second gesture operation is an operation of the user clenching his hand. The approximation degree is an index indicating the degree of clenching of the user's hand.

10. A display method for accepting a non-contact input operation of a user on a display screen, characterized in that: in: One or more processors perform the following steps: An input operation detection step of detecting an input operation including a predetermined gesture operation of the user; A display step, when a first gesture operation of the user is detected, causing an input operation icon to be displayed on the display screen in a first display mode, and when a second gesture operation of the user is detected, causing the input operation icon to be displayed on the display screen in a second display mode; an accepting step of accepting a moving operation of moving the input operation icon according to the first gesture operation when the first gesture operation is detected, and accepting a selection operation of selecting a selection object according to the input operation icon when the second gesture operation is detected; as well as A change step, when detecting a third gesture operation of the user, causes the display mode of the input operation icon to be changed in stages or continuously between the first display mode and the second display mode, wherein the third gesture operation is a gesture operation in the middle of changing from the first gesture operation to the second gesture operation.

11. A recording medium having a display program recorded thereon, wherein the display program receives a non-contact input operation of a user on a display screen, characterized in that: The display program is used to enable one or more processors to execute the following steps: An input operation detection step of detecting an input operation including a predetermined gesture operation of the user; A display step, when a first gesture operation of the user is detected, causing an input operation icon to be displayed on the display screen in a first display mode, and when a second gesture operation of the user is detected, causing the input operation icon to be displayed on the display screen in a second display mode; an accepting step of accepting a moving operation of moving the input operation icon according to the first gesture operation when the first gesture operation is detected, and accepting a selection operation of selecting a selection object according to the input operation icon when the second gesture operation is detected; as well as A change step, when detecting a third gesture operation of the user, causes the display mode of the input operation icon to be changed in stages or continuously between the first display mode and the second display mode, wherein the third gesture operation is a gesture operation in the middle of changing from the first gesture operation to the second gesture operation.

Images