Indication device

Abstract

By using a virtual hand object, it is possible to accurately and easily manipulate virtual objects that appear to float in space. [Solution] The display unit 20 is provided on the glasses 10 and displays the source image that serves as the basis for a two-dimensional virtual object for operation, which the user uses to indicate what to do with their hands. The detection unit 72 of the control unit 70 detects the user's hand movements using image data captured by the imaging unit 30 and distance data measured by the camera unit (distance measuring unit) 31, and outputs information related to those hand movements. Based on the information about hand movements sent from the detection unit 72, the display control unit 71 of the control unit 70 controls the display of the virtual hand object, which is an image modeling a hand, so that the virtual hand object moves on the same plane as the two-dimensional virtual object for operation in accordance with the hand movements.

Description

Technical Field

[0004] , , , , , , , , ,

[0001] The present invention relates to a display device for displaying virtual images (videos) in an environment using Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), etc.

Background Art

[0002] In recent years, various head-mounted display devices that can experience Extended Reality (XR) including not only virtual reality and augmented reality but also mixed reality in which content is overlaid and displayed in the real space have been manufactured and sold. In these devices, for example, in an environment using mixed reality, a user can perform a key input operation on a keyboard (an operation virtual object) displayed as a computer graphics (CG) image / video using a virtual object of a hand in which the user's own hand is modeled (see, for example, Patent Documents 1 to 3).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with the conventional devices described above, for example, when performing input operations using a virtual hand object that appears to float in space on a keyboard displayed as if floating in space in the environment described above, it was difficult to grasp the front-to-back positional relationship between the front and back, making accurate key input impossible. For example, if the user tried to press a key on the keyboard (virtual object for operation) with their index finger and moved their finger in the depth direction of that key, it was sometimes judged that they had pressed one of the other keys located above, below, to the left, or to the right of that key. When this judgment occurred, the user had to cancel the key operation they had performed and perform the same key input operation again. Therefore, with conventional devices, when performing input work using the keyboard (virtual object for operation) displayed in the environment described above, not only were there many input errors, but re-entry was often required, resulting in a problem that prevented comfortable input operation.

[0005] The present invention has been made in accordance with the above circumstances, and aims to provide a display device that allows for accurate and easy operation of a virtual object for manipulation, which is displayed as if floating in space, using a virtual hand object that models the user's hand. [Means for solving the problem]

[0006] To achieve the above objective, the present invention provides a display device configured to allow a user to view a virtual image, comprising: a device worn on the user's head; a display unit provided on the device that displays a source image which is the source image of the virtual image; an imaging unit that captures the user's hand and outputs the captured image data; a detection unit that uses the image data obtained by the imaging unit to detect the user's hand movements and outputs information regarding those hand movements; and a display control unit that controls the display of the virtual image by controlling the display unit, wherein the display control unit controls the display unit to display a two-dimensional virtual object for operation to indicate the operation content with the user's hand and a virtual hand object modeled after the hand, respectively, as elements of the virtual image, and also controls the display of the virtual hand object so that the virtual hand object moves on the same plane as the two-dimensional virtual object for operation in accordance with the hand movements, based on information regarding the hand movements sent from the detection unit.

[0007] For example, virtual objects for operation can be models of a keyboard, a numeric keypad, or the control surface of a remote control.

[0008] In the display device of the present invention, the display control unit controls the display of the virtual hand object so that the virtual hand object moves on the same plane as the two-dimensional virtual operation object in accordance with the user's hand movements. As a result, the virtual hand object cannot move in the depth direction relative to the virtual operation object, and can only move on the same plane as the virtual operation object. Therefore, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the virtual operation object. Consequently, compared to conventional devices in which the user operates a keyboard (virtual operation object) that appears to float in space, for example, the display device of the present invention allows for accurate and easy operation of the virtual operation object.

[0009] Furthermore, it is desirable that the display device of the present invention further includes an action determination unit that determines whether the user has performed any of the specific actions, including tapping (clicking) and scrolling, using their hand, based on image data obtained by the imaging unit and information on hand movements sent from the detection unit, and outputs information on the content of the specific action performed by the hand when it is determined that either of those actions has been performed. In this case, the display control unit can recognize the content of the specific action performed by the user when information on the content of the specific action is sent from the action determination unit, and can control the display of the virtual operation object and / or the virtual hand object according to the recognized content of the specific action.

[0010] For example, when the display control unit receives information from the action determination unit indicating that a tap (click) operation has been performed as information regarding the content of a specific operation, it can control the display of the virtual hand object so that the portion of the virtual hand object corresponding to the finger that performed the tap (click) operation moves in the depth direction relative to the virtual operation object. Alternatively, when the display control unit receives information from the action determination unit indicating that a tap (click) operation has been performed as information regarding the content of a specific operation, it can control the display of the virtual operation object so that the portion of the virtual hand object corresponding to the finger that performed the tap (click) operation is located on the virtual operation object is highlighted, for example.

[0011] In the display device of the present invention, the imaging unit has the function of a distance measuring unit that measures the distance to the user's hand and outputs the measured distance data, and the detection unit may obtain three-dimensional position information for identifying the hand based on the image data and distance data obtained by the imaging unit, and acquire the obtained three-dimensional position information as information related to the movement of the hand. In this case, the display control unit can determine the display position of the virtual hand object on the virtual operation object based on the three-dimensional position information for identifying the hand sent from the detection unit.

[0012] Furthermore, the display device of the present invention may further include a position information measurement unit that measures the three-dimensional position information of the user's hand in an arbitrarily set three-dimensional coordinate system and acquires the measured three-dimensional position information as information related to hand movement. In this case, the display control unit can determine the display position of the virtual hand object on the virtual operation object based on the three-dimensional position information for identifying the hand sent from the position information measurement unit.

[0013] In the display device of the present invention, the detection unit determines whether the user is gripping part or all of their fingers based on image data obtained by the imaging unit, and when it determines that the user is gripping part or all of their fingers, it outputs information indicating that the user is gripping part or all of their fingers. The display control unit, upon receiving information from the detection unit indicating that the user is gripping part or all of their fingers, may, at the user's discretion, use a virtual mouse body modeled after the mouse body and a virtual mouse cursor modeled after the mouse cursor instead of a virtual hand object, display the virtual mouse body at a predetermined location on the virtual image, and control the display of the virtual mouse cursor object based on information about hand movements sent from the detection unit so that the virtual mouse cursor object moves on the same plane as the virtual operation object in accordance with the hand movements.

[0014] In this case, for example, if the detection unit receives information from the detection unit that the user is clenching their hand with one finger extended, the operation determination unit will determine, based on the hand movement information received from the detection unit, which of the specified operations was performed using that extended finger. The display control unit can then recognize, when the operation determination unit receives information from the detection unit that a click operation was performed using that one finger, that a mouse click operation was performed on the portion of the virtual operation object corresponding to the virtual mouse cursor object at the time the click operation was performed. Furthermore, if the detection unit receives information from the detection unit that the user is clenching their hand with two fingers extended, the operation determination unit will determine, based on the hand movement information received from the detection unit, which of the two extended fingers was used to perform which of the specified operations. The display control unit can then recognize, when the operation determination unit receives information from the detection unit that a click operation was performed using the left (or right) finger of the two fingers, that a left-click (or right-click) mouse operation was performed on the portion of the virtual operation object corresponding to the virtual mouse cursor object at the time the click operation was performed. Furthermore, when the detection unit receives information from the detection unit indicating that the user is clenching all of their fingers, the operation determination unit determines, based on the hand movement information received from the detection unit, which of the specified operations was performed with the hand that was clenching all of its fingers. The display control unit can recognize, when the operation determination unit receives information from the operation determination unit indicating that a click operation was performed with the hand that was clenching all of its fingers, that a mouse click operation was performed on the portion of the virtual operation object corresponding to the virtual object of the mouse cursor at the time the click operation was performed.

[0015] Furthermore, when the display control unit controls the display unit to display the virtual object of the mouse cursor, it may control the movement of the virtual object of the mouse cursor on the virtual image by determining the amount of movement of the virtual object of the mouse cursor on the virtual image based on the actual distance the hand moves, which is calculated based on the hand movement information sent from the detection unit.

[0016] Furthermore, in the display device of the present invention, the display control unit is configured to control the display unit to display a three-dimensional virtual operation object instead of a two-dimensional virtual operation object, and when the display unit is controlled to display the three-dimensional virtual operation object, the display of the virtual hand object may be controlled so that the virtual hand object moves on the surface of the three-dimensional virtual operation object or on a plane formed by its upper end surface in accordance with the user's hand movements.

[0017] Furthermore, in the display device of the present invention, the virtual object of the hand may be a two-dimensional virtual object or a three-dimensional virtual object. [Effects of the Invention]

[0018] In the display device of the present invention, the display control unit controls the display of the virtual hand object so that the virtual hand object moves on the same plane as the two-dimensional virtual operation object, or on the surface of the three-dimensional virtual operation object or on the plane formed by its upper end surface, in accordance with the user's hand movements. As a result, the virtual hand object cannot move in the depth direction relative to the virtual operation object and can only move on the same plane as the virtual operation object. Therefore, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the virtual operation object. Consequently, compared to conventional devices in which the user operates with their hand on a keyboard (virtual operation object) that appears to float in space, for example, the display device of the present invention allows for accurate and easy operation of the virtual operation object. [Brief explanation of the drawing]

[0019] [Figure 1] Figure 1 is a schematic perspective view of a display device according to the first embodiment of the present invention. [Figure 2] Figure 2 is a schematic block diagram of the display device according to the first embodiment. [Figure 3]FIG. 3(a) and (b) are diagrams for explaining the arrangement of the display unit when the virtual image is projected on both the left and right sides of the user's field of view. [Figure 4] FIG. 4 is a diagram showing an example of a two-dimensional operation virtual object. [Figure 5] FIG. 5 is a diagram showing an example of a character input screen as an operation virtual object. [Figure 6] FIG. 6 is a diagram showing an example of a search screen displayed on the character input screen. [Figure 7] FIG. 7 is a schematic diagram for explaining a method of correcting the movement distance of the hand. [Figure 8] FIG. 8 is a schematic diagram for explaining a method of correcting the movement distance of the hand. [Figure 9] FIG. 9 is a diagram showing the actual hand length h0 of the user and the hand length h1 seen by the camera when the plane of the user's hand is tilted backward. [Figure 10] FIG. 10 is a diagram showing the actual finger length f0 of the user and the finger length f1 seen by the camera when the user's finger is tilted significantly backward from the back of the hand. [Figure 11] FIG. 11 is a diagram showing the actual length g0 of the part of the user's finger from the second joint to the fingertip and the length g1 of the part of the finger from the second joint to the fingertip seen by the camera when the part of the user's finger from the second joint to the fingertip is tilted significantly backward from the part of the finger from the third joint to the second joint. [Figure 12] FIG. 12 is a diagram showing an example of a virtual hand object displayed on the same plane as the operation virtual object. [Figure 13] FIG. 13 is a diagram showing an example of a virtual hand object displayed on the same plane as the operation virtual object. [Figure 14] FIG. 14 is a flowchart for explaining the procedure of the display process of the operation virtual object and the virtual hand object by the display control unit in the display device of the first embodiment. [Figure 15]Figure 15 is a flowchart illustrating the processing procedure for controlling the display of a virtual image (including virtual objects) as shown in step S16 of the processing flow in Figure 14. [Figure 16] Figure 16 is a schematic block diagram of a display device according to a second embodiment of the present invention. [Figure 17] Figure 17 is a schematic block diagram of a display device according to a third embodiment of the present invention. [Figure 18] Figure 18(a) is a schematic perspective view of a display device according to the fourth embodiment of the present invention, and Figure 18(b) is a schematic cross-sectional view of the display device in the direction of arrow AA. [Figure 19] Figure 19 is a schematic block diagram of the display device according to the fourth embodiment. [Figure 20] Figure 20(a) is a schematic perspective view of a modified display device of the fourth embodiment, and Figure 20(b) is a schematic cross-sectional view of the same display device in the direction of arrow BB. [Figure 21] Figure 21 is a schematic block diagram of a display device according to the fifth embodiment of the present invention. [Figure 22] Figure 22 is a schematic block diagram of a display device according to the sixth embodiment of the present invention. [Figure 23] Figure 23 is a schematic front view of a display device according to the seventh embodiment of the present invention. [Figure 24] Figure 24 is a schematic block diagram of the display device according to the seventh embodiment. [Figure 25] Figure 25 is a schematic diagram of a display device according to the eighth embodiment of the present invention. [Figure 26] Figure 26 is a schematic block diagram of the display device according to the eighth embodiment. [Figure 27] Figure 27 is a schematic block diagram of a display device which is a modified example of the eighth embodiment. [Figure 28] Figure 28 is a schematic diagram of a display device according to the ninth embodiment of the present invention. [Figure 29] Figure 29 is a schematic block diagram of the display device according to the ninth embodiment. [Figure 30] Figure 30 is a schematic block diagram of a display device which is a modified example of the ninth embodiment. [Figure 31] Figure 31 shows an example of a virtual hand object displayed on a two-dimensional virtual object for manipulation in three-dimensional space. [Figure 32] Figure 32 shows an example of a virtual hand object displayed on a three-dimensional virtual object for manipulation in three-dimensional space. [Figure 33] Figure 33 shows an example of a plane formed by the upper surface of a three-dimensional keyboard image, i.e., the keytop surface. [Figure 34] Figure 34 shows an example of a virtual object representing the mouse body. [Figure 35] Figure 35 shows an example of a virtual object representing the mouse body. [Figure 36] Figure 36 shows examples of virtual objects for the mouse body and mouse cursor displayed on the same plane as a two-dimensional virtual object for manipulation. [Figure 37] Figure 37 shows examples of virtual objects for the mouse body and mouse cursor displayed on a three-dimensional virtual object for operation. [Figure 38] Figure 38 shows an example of a virtual object for operation that models the numeric keypad of a smartphone. [Figure 39] Figure 39 shows an example of a virtual object for operation that models the control surface of an air conditioner remote control. [Modes for carrying out the invention]

[0020] The following describes embodiments for carrying out the present invention with reference to the drawings.

[0021] [First Embodiment] The display device of the present invention displays virtual images (videos) in an environment using augmented reality (AR), virtual reality (VR), mixed reality (MR), etc. First, the display device according to the first embodiment of the present invention will be described. Figure 1 is a schematic perspective view of the display device according to the first embodiment of the present invention, and Figure 2 is a schematic block diagram of the display device according to the first embodiment.

[0022] In the first embodiment, the case in which the display device of the present invention is applied to a glasses-type terminal used by a user who wears it on their head like glasses will be described. The display device 1a of the first embodiment is AR or MR hardware configured to allow a user to view virtual images, and as shown in Figures 1 and 2, it comprises glasses 10 as a wearable device worn on the user's head, a display unit 20 provided on the glasses 10 that displays a source image which is the source image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a touch pad unit 40, a microphone unit 50, a speaker unit 60, a control unit 70, a communication unit 80, and a storage unit 90.

[0023] The display unit 20 is a transmissive prism display. Specifically, the display unit 20 comprises, for example, a small projector 21 having a liquid crystal panel (display device), an optical system 22, and a half mirror (not shown). As the optical system 22, for example, a lens, a reflector, a prism, a light guide plate, a waveguide, etc. may be used. Alternatively, other optical systems may be used instead of the optical system 22 described above. The half mirror acts as a projection unit onto which the image or video displayed on the display device of the small projector 21 is projected via the optical system 22. For example, the small projector 21 and the optical system 22 are arranged in a single housing H, and as shown in Figure 1, the housing H is attached to the right temple of the eyeglasses 10. In this case, the half mirror is embedded in, for example, a prism (not shown) positioned in front of the lens 11 for the right eye of the eyeglasses 10. The image or video displayed on the liquid crystal panel of the small projector 21 is projected onto the half mirror via the optical system 22. In fact, a very small image or video is displayed on this half mirror. When a user uses the display device 1a of the first embodiment, they can see a tiny image or video displayed on the half-mirror as if it were floating in space. This image or video that appears to be floating in space is a virtual image. Here, a virtual image representing an object is also called a virtual object of that object. A virtual object can be considered an element of the virtual image. In contrast, the tiny image or video displayed on the half-mirror is the "original image," which is the source image of the virtual image. In this way, by attaching the miniature projector 21 to the right temple of the glasses 10, the virtual image can be projected onto the right side of the user's field of vision. The miniature projector 21 can project multiple images or videos on top of each other. For example, the miniature projector 21 can project an image representing an operation screen for the user to indicate operations with their hands and a video representing the external environment (e.g., the exterior or interior of the room) on top of each other.

[0024] Furthermore, as described above, when the original image of a virtual image (including a virtual object) is displayed on the half-mirror of the display unit 20, the user will see the virtual image appearing to float in space through the display unit 20. In this respect, the display unit 20 can be considered to be for displaying virtual images. For this reason, in this specification, we will also simply state that "the display unit displays virtual images."

[0025] Furthermore, if the virtual image (including virtual objects) is to be displayed on the left side of the user's field of vision, the housing H described above can be attached to the left temple of the glasses 10, and a half-mirror can be attached in front of the lens for the left eye. It is also possible to display the virtual image (including virtual objects) on both the left and right sides of the user's field of vision. Figures 3(a) and (b) illustrate the arrangement of the display unit when the virtual image is displayed on both the left and right sides of the user's field of vision. In this case, the housings H1 and H2, including the miniature projector 21 and the optical system 22, are attached to the temples on both the left and right sides of the glasses 10, as shown in Figures 3(a) and (b), and a half-mirror (not shown) is also attached in front of the lens sections 11, 11 for both the left and right eyes. Different images or videos can be displayed on these left and right miniature projectors 21, 21. In particular, for example, by displaying the image for the left eye on the left miniature projector 21 and the image for the right eye on the right miniature projector 21, the display device 1a can display a three-dimensional virtual image (including virtual objects) that the user can view in 3D.

[0026] The display device 1a of the first embodiment can display a virtual object of a specific object. The virtual object of a specific object is, for example, an image modeling that object. In particular, the display device 1a of this embodiment can display an operation virtual object, which is a model of an object for the user to use to indicate what to do with their hands, as a two-dimensional virtual object. At this time, a miniature original image corresponding to the operation virtual object is displayed on the half mirror of the display unit 20. Figure 4 is a diagram showing an example of a two-dimensional operation virtual object. The display device 1a of this embodiment may generally also display the actual scenery or an image of the scenery superimposed on the two-dimensional operation virtual object, but in the example of Figure 4, the display device 1a displays only the operation virtual object as a two-dimensional virtual object. The operation virtual object shown in Figure 4 is a character input screen 200. Figure 5 is a diagram showing an example of a character input screen 200 as an operation virtual object. As shown in Figure 5, the character input screen 200 has a keyboard image 210 and a display area 220 for displaying entered characters, etc. The keyboard image 210 includes multiple character key images associated with each character (including symbols), and multiple function key images assigned to specific functions. The display area 220 displays, for example, a search screen. Figure 6 shows an example of a search screen displayed on the character input screen 200. This search screen 221 is for searching websites on the internet and includes a keyword input section 2211 and a search results display section 2212 that displays the search results.

[0027] In this specification, a "two-dimensional" virtual image (including virtual objects) refers to a virtual image that represents how an object is distributed in a two-dimensional space or on a two-dimensional plane in a three-dimensional space. In contrast, a "three-dimensional" virtual image (including virtual objects) refers to a virtual image that represents how a three-dimensional object is distributed in a three-dimensional space.

[0028] Furthermore, in the first embodiment, when the display device 1a is displaying an operation virtual object as a two-dimensional virtual object, a virtual hand object modeled after the user's hand can be displayed on the same plane as the two-dimensional operation virtual object (see, for example, Figure 12 described later). In this case, the original image corresponding to the virtual hand object is displayed on the half-mirror of the display unit 20. As the virtual hand object, for example, a two-dimensional virtual object representing the shape of a general hand is used. The user can move the virtual hand object on the operation virtual object by moving their own hand. In addition, when the part of the virtual hand object corresponding to the user's fingers is located on the desired part of the operation virtual object, the user can give various instructions related to the operation virtual object to the control unit 70 by performing specific actions (gestures) including tap (click), scroll, pinch-in, and pinch-out actions using their hand. Here, the tap (click) action is an action to select a key image, etc., of the operation virtual object. The scroll action is an action to instruct the operation virtual object to be moved up and down or left and right for display. A pinch-in gesture instructs the user to shrink the virtual object used for operation, and a pinch-out gesture instructs the user to enlarge the virtual object used for operation. The control unit 70 recognizes the content of these instructions and controls the display of the virtual object used for operation and the virtual hand object according to the recognized instructions. For example, when a user is looking at the virtual object used for operation, which is the character input screen 200, they can move their hand to move the virtual hand object on the keyboard image 210, and when the index finger of the virtual hand object is positioned on the desired key image on the keyboard image 210, they can perform a tap (click) gesture with their index finger to instruct the user to input the character corresponding to that key image. In this specification, expressions such as "performing a specific action using the hand" include both "performing a specific action using the hand" and "performing a specific action using the fingers of the hand."

[0029] The imaging unit 30 captures images of the user's hands. In the first embodiment, as shown in Figure 1, the imaging unit 30 is installed on the temple portion of the glasses 10 adjacent to the display unit 20. That is, the imaging unit 30 is arranged inside the housing H together with the small projector 21. However, as shown in Figure 3(b), the imaging unit 30 can also be installed on its own at the upper center of the glasses 10. As shown in Figure 2, the imaging unit 30 comprises a camera unit 31, an image processing unit 32, and a camera control unit 33. The camera unit 31 has a lens and an image sensor. This camera unit 31 is positioned to capture a wide range, including the area in front of the user. For example, the camera unit 31 can capture images of the user's hands placed on a table or on their lap while they are sitting with the glasses 10 on. The image processing unit 32 performs image processing such as color and gradation correction of the captured image and compression of the image data based on the image data captured by the camera unit 31. The camera control unit 33 controls the image processing unit 32 and controls the exchange of image data with the control unit 70. Image data captured by the imaging unit 30 is sent to the control unit 70 and stored in the storage unit 90 by the control unit 70. The imaging unit 30 also has functions for capturing still images and capturing moving images.

[0030] The camera unit 31 also functions as a distance measuring unit to measure the distance to the user's hand. In this case, for example, the camera control unit 33 is configured to include an autofocus control unit. This autofocus control unit controls the camera unit 31 to automatically focus on a subject at a predetermined position within the imaging range, and when it captures an image of the subject that has been automatically focused on, it calculates distance data to the captured subject. The distance data obtained by the camera unit (distance measuring unit) 31 is sent to the control unit 70, and the control unit 70 stores it in the storage unit 90. In addition, when the camera unit 31 measures the distance to the hand, it may also measure the distance to the tips of each finger. For this reason, expressions such as "distance to the hand" in this specification shall mean "distance to the fingers" when the distance measuring unit measures the distance to the fingers.

[0031] In the first embodiment, the imaging unit 30 is described as having one camera unit 31, which photographs the user's hand and measures the distance to the hand. However, generally, the imaging unit 30 may be configured to have two or more camera units. For example, if the imaging unit 30 has two camera units, one camera unit may be used to photograph the external environment including the user's hand, and the other camera unit may be used to measure the distance to the user's hand. Also, if the imaging unit 30 has three or more camera units, one camera unit may be used to photograph the external environment including the user's hand, and the remaining multiple camera units may be used to measure the distance to the user's hand. In this case, by measuring the distance to the user's hand with multiple camera units, the position of the user's hand can be obtained as three-dimensional position information, and the distance to the user's hand can be determined using this three-dimensional position information. Furthermore, multiple camera units may be used to photograph the external environment including the user's hand. In addition, in the first embodiment, the camera unit 31 for photographing the hand may be positioned downward or diagonally downward so that even when the user's hand is resting on a desk, lap, or the like, the hand is photographed from a direction that is perpendicular or nearly perpendicular to the optical axis of the camera unit 31. Furthermore, the camera unit 31 may be configured to be able to move its direction downward or diagonally downward.

[0032] Furthermore, the camera unit used to measure the distance to the user's hand may be not limited to a standard natural light camera, but may also use an infrared camera or a combination of a natural light camera and a depth sensor (such as an infrared sensor). In addition, the distance measuring unit used to measure the distance to the user's hand may be not limited to a camera, but may also use a distance sensor, laser, radar, etc. Furthermore, it is also possible to measure the distance to the subject (hand) using the "monocular camera distance measurement technology" software recently developed by Toshiba Information Systems Corporation. Moreover, by using these devices and technologies, it is possible to measure the three-dimensional position information (XYZ coordinate information) of the subject (hand or fingers) in an arbitrarily set three-dimensional coordinate system and determine the position of the subject (hand or fingers), and based on the determined three-dimensional position information of the subject (hand or fingers), the distance from a predetermined reference position (coordinate origin) to the subject (hand or fingers) can be determined.

[0033] The temple portion of the eyeglasses 10 is equipped with a touchpad 40, a microphone 50, a bone conduction speaker 60, various sensor units, and a battery unit. Note that in Figure 1, these units are omitted for the sake of simplification. The touchpad 40 allows the user to provide various instructions to the control unit 70 through touch operations. The microphone 50 receives the user's voice input in order to operate the display device 1a of the first embodiment using voice commands. The voice information input from the microphone 50 is sent to the control unit 70, which then analyzes the voice information. The speaker 60 transmits voice information to the user using bone vibrations. Generally, the speaker 60 is not limited to one that transmits voice information to the user using bone vibrations; a regular speaker, earphones, headphones, etc., may also be used.

[0034] The control unit 70 includes a central processing unit (CPU) and other components, and controls the entire display device 1a of the first embodiment. For example, the control unit 70 controls the display unit 20 to control the display unit 20 to display the original image of the virtual image (including virtual objects), and controls the imaging unit 30 to capture images. The control unit 70 also recognizes the content of the instructions given by the operation of the touchpad unit 40 when it is operated, and executes processing according to the recognized content, and recognizes the content of the input audio information when audio is input from the microphone unit 50, and executes processing according to the recognized content. Furthermore, the control unit 70 controls the audio information emitted by the speaker unit 60. Specifically, as shown in Figure 2, this control unit 70 includes a display control unit 71, a detection unit 72, and an operation determination unit 73.

[0035] The display control unit 71 controls the display of various virtual images, including virtual objects for operation, by controlling the display unit 20. Specifically, when a user gives a voice command using the microphone unit 50 or an instruction using the touchpad unit 40, the display control unit 71 selects the content of the virtual image (including virtual objects) that the display device 1a should display according to the content of the instruction, and controls the display of the virtual image by displaying the original image corresponding to the selected virtual image on the half-mirror of the display unit 20. As a result, the original image corresponding to the virtual image instructed by the user is displayed on the half-mirror of the display unit 20, and the user can see the virtual image as if it were floating in space. For example, when the original image of a virtual object for operation is displayed on the half-mirror of the display unit 20, the user can see that the two-dimensional virtual object for operation is displayed as if it were floating in space. Also, when the display unit 20 is displaying the original image of a virtual object for operation, and the user's hand is captured by the imaging unit 30, the display control unit 71 displays the original image corresponding to the virtual hand object, which is a model of the user's hand, on the half-mirror of the display unit 20 in order to display the virtual hand object, which is a model of the user's hand, as if it were floating in space. As a result, the half-mirror of the display unit 20 displays the original image corresponding to the virtual hand object, allowing the user to see the virtual hand object. Therefore, in this case, the user can see both the virtual hand object and the virtual control object simultaneously.

[0036] The detection unit 72 uses image data and distance data obtained by the imaging unit 30 to detect the user's hand movements (including finger movements) and outputs information related to those hand movements. For example, three-dimensional position information for identifying the hand can be used as information related to hand movements. Typically, three-dimensional position coordinates (XYZ coordinates) in a three-dimensional Cartesian coordinate system are used as the three-dimensional position information. The three-dimensional Cartesian coordinate system can be set arbitrarily, but here, we will set a three-dimensional Cartesian coordinate system in which the direction of the camera's optical axis is the Z-axis direction, and the two orthogonal directions in the plane perpendicular to the camera's optical axis are the X-axis direction and the Y-axis direction, respectively. In this case, the detection unit 72 first recognizes the hand in the image data based on the image data captured by the imaging unit 30 using a general image recognition method, and obtains two-dimensional position information (XY coordinate information) for identifying the recognized hand. That is, this two-dimensional position information is the two-dimensional position information for identifying the hand on a plane when the hand is projected onto a plane perpendicular to the optical axis of the camera unit 31. Next, the detection unit 72 obtains one-dimensional position information (Z coordinate information) for identifying the hand in the optical axis direction of the camera unit 31, based on the two-dimensional position coordinates (XY coordinate information) obtained and the distance data obtained by the imaging unit 30. Then, from this two-dimensional position information and one-dimensional position information, it constructs three-dimensional position information (XYZ coordinate information) in a three-dimensional orthogonal coordinate system for identifying the hand. The detection unit 72 acquires this three-dimensional position information for identifying the hand, for example, at regular time intervals, and sends the acquired three-dimensional position information to the motion determination unit 73 and the display control unit 71 as information related to the movement of the hand. As the three-dimensional position information for identifying the hand, for example, the three-dimensional position information of the center of the back of the hand can be used, but in addition to the three-dimensional position information of the center of the back of the hand, the three-dimensional position information of the tip of each finger may also be used. Furthermore, as the three-dimensional position information for identifying the hand, the three-dimensional position information of the fingertip or the three-dimensional position information of the first, second, or third joint of the finger may also be used. Therefore, the expression "three-dimensional positional information for identifying a hand" in this specification shall also include the meaning of "three-dimensional positional information for identifying fingers of a hand."

[0037] The motion determination unit 73 determines the content of the hand action based on the image data obtained by the imaging unit 30 and the hand movement information sent from the detection unit 72. For this determination, a general image recognition method is used, for example. Specifically, the motion determination unit 73 determines whether the user has performed any of the specific actions (gestures) using their hand, including tap (click) actions, scroll actions, pinch-in actions, and pinch-out actions. When the motion determination unit 73 determines that any of the specific actions have been performed, it generates information about the content of the specific action performed by the hand and transmits it to the display control unit 71. For example, the information generated about the content of the specific action performed by the hand may include information indicating that a tap (click) action was performed, information indicating that a scroll action was performed, information indicating that a pinch-in action was performed, information indicating that a pinch-out action was performed, etc. Here, this information includes information that identifies the hand and / or finger that performed the action (gesture).

[0038] As described above, the display control unit 71 controls the display unit 20 to display a two-dimensional virtual operation object for the user to indicate operations with their hands, and a virtual hand object modeled after the hand, as elements of a virtual image. Furthermore, when the display device 1a is displaying a two-dimensional virtual operation object, the display control unit 71 controls the display of the virtual hand object based on information about hand movements sent from the detection unit 72, so that the virtual hand object moves on the same plane as the two-dimensional virtual operation object in accordance with the user's hand movements. Specifically, when controlling the display of the virtual hand object, the display control unit 71 determines the display position of the virtual hand object on the plane containing the virtual operation object based on the two-dimensional position information (XY coordinate information) used to identify the hand on a plane perpendicular to the optical axis of the camera unit 31, which is part of the three-dimensional position information for identifying the hand sent from the detection unit 72. In other words, it assumes that the plane perpendicular to the optical axis of the camera unit 31 is parallel to the plane containing the virtual operation object. Furthermore, the display control unit 71 determines the amount of movement of the virtual hand object on the plane containing the virtual hand object based on the actual distance the hand moves, calculated based on the information about the hand movement sent from the detection unit 72. Here, the actual distance the hand moves can be determined using three-dimensional position information (XYZ coordinate information) before and after the hand movement. In this case, the three-dimensional position information (XYZ coordinate information) is not limited to the two-dimensional position information (XY coordinate information) for identifying the hand on a plane perpendicular to the optical axis of the camera unit 31 and the one-dimensional position information (Z coordinate information) for identifying the hand in the direction of the optical axis of the camera unit 31, as described above, but may also be used as three-dimensional position information of the hand in an arbitrary three-dimensional coordinate system. In addition, the three-dimensional position information (XYZ coordinate information) of the hand and fingers in an arbitrarily set three-dimensional coordinate system can be measured, and the distance the hand and fingers move can be determined from the difference in coordinates before and after the hand and fingers move.

[0039] Incidentally, the actual distance the hand moves can be determined using two-dimensional position information (XY coordinate information) to identify the hand on a plane perpendicular to the optical axis of the camera unit 31, without using one-dimensional position information (Z coordinate information) to identify the hand in the optical axis direction of the camera unit 31. Now, let me explain this point. First, if the user's hand plane is perpendicular to the optical axis of the camera, the display control unit 71 can easily calculate the actual distance the hand moves based on the two-dimensional position information (XY coordinate information) sent from the detection unit 72. However, if the user's hand is placed on a table or on their lap, for example, and the user's hand plane is tilted relative to the plane perpendicular to the optical axis of the camera, the user may move their hand along that tilted plane to indicate an operation. In such cases, the display control unit 71 can correct the hand's movement distance calculated based on the two-dimensional position information (XY coordinate information) to calculate the actual distance the hand moves.

[0040] Now, I will briefly explain the method for correcting the distance of hand movement. Figures 7 and 8 are schematic diagrams illustrating the method for correcting the distance of hand movement. As shown in Figures 7 and 8, assume that the user's hand plane does not coincide with the plane perpendicular to the camera's optical axis, but is tilted towards the back. Let θ be the angle between the user's hand plane and the plane perpendicular to the camera's optical axis. In Figure 7, the user's hand plane and the hand on that plane are represented by solid lines, while the hand plane as seen by the camera (the hand plane obtained by projecting the user's hand plane onto the plane perpendicular to the camera's optical axis) and the hand on that plane are represented by dashed lines. The positional relationship between the user's hand plane and the hand plane as seen by the camera, when viewed from the side, is shown in Figure 8. In Figure 8, the user's hand plane is also represented by a solid line, and the hand plane as seen by the camera is represented by a dashed line. In Figure 8, the hand plane as seen by the camera is drawn so that it intersects with the user's hand plane at its center. Here, the point where the plane of the hand as seen by the camera intersects with the plane of the user's hand can be set to any position on the hand or fingers, such as the base of the back of the hand or near the center of the fingers. Now, let h0 be the actual length of the user's hand, and h1 be the length of the hand as seen by the camera. Figure 9 shows the actual length of the user's hand h0 and the length of the hand as seen by the camera h1 when the plane of the user's hand is tilted towards the back. In this case, the length of the hand as seen by the camera is smaller than the actual length of the hand by the ratio cosθ = h1 / h0. Now, suppose the user moves their hand a distance △D along that hand plane. For simplicity of explanation, let's assume the user moves their hand upward or downward along the hand plane in Figure 8. In this case, the display control unit 71 first calculates the length of the hand h1 and the distance △Y the hand has moved in the vertical direction (Y axis direction) based on the image data obtained by the imaging unit 30 and the two-dimensional position information (XY coordinate information) sent from the detection unit 72. Next, by multiplying the vertical movement distance of the hand △Y by the factor 1 / cosθ = h0 / h1, the actual movement distance of the hand △D = △Y × 1 / cosθ = △Y × (h0 / h1) can be obtained. Note that the actual length of the hand h0 can be obtained in advance by positioning the user's hand so that the plane of the user's hand is perpendicular to the optical axis of the camera and taking a picture.

[0041] Furthermore, if the user's fingers (from the third joint to the fingertips) are significantly tilted inward relative to the back of the hand, the ratio of the finger length as seen by the camera to the actual finger length will be smaller than the ratio of the back of the hand length as seen by the camera to the actual back of the hand length. Figure 10 shows the user's actual finger length f0 and the finger length as seen by the camera f1 when the user's fingers are significantly tilted inward relative to the back of the hand. In this case, using the user's actual finger length in the vertical direction (Y-axis direction) f0 and the finger length in the vertical direction (Y-axis direction) f1 as seen by the camera allows for a more accurate calculation of the actual hand movement distance than using the user's actual hand length h0 and the hand length as seen by the camera h1 as described above. That is, the actual hand movement distance can be obtained by multiplying the vertical movement distance of the hand as seen by the camera by f0 / f1.

[0042] Furthermore, if the portion of the user's finger from the second joint to the fingertip is tilted more significantly inward than the portion from the third joint to the second joint, using the vertical length (Y-axis direction) g0 of the portion of the user's finger from the second joint to the fingertip and the vertical length (Y-axis direction) g1 of the portion of the finger from the second joint to the fingertip as seen by the camera can more accurately calculate the actual distance the hand moves than using the user's actual finger length f0 and the finger length f1 as seen by the camera. Here, Figure 11 shows the actual length g0 of the portion of the user's finger from the second joint to the fingertip and the length g1 of the portion of the finger from the second joint to the fingertip as seen by the camera when the portion of the user's finger from the second joint to the fingertip is tilted more significantly inward than the portion from the third joint to the second joint. In this case, the actual distance the hand moves can be obtained by multiplying the vertical distance the hand moves as seen by the camera by g0 / g1.

[0043] As can be seen from the above explanation, when the display control unit 71 controls the display of the virtual hand object so that the virtual hand object moves on the same plane as the two-dimensional operation virtual object in accordance with the user's hand movements, distance data to the user's hand is not essential. In this case, the detection unit 72 detects the user's hand movements using image data obtained by the imaging unit 30, and uses two-dimensional position information (XY coordinate information) to identify the hand as information regarding the user's hand movements. Generally, three-dimensional position information (XYZ coordinate information) of the hand and fingers in an arbitrarily set three-dimensional coordinate system can be measured, and this measured three-dimensional position information can be used as information regarding hand movements.

[0044] Figures 12 and 13 show examples of virtual hand objects displayed on the same plane as the virtual operation object. While the display device 1a generally displays actual scenery or images of the scenery overlaid on the virtual operation object (a two-dimensional virtual object) and the virtual hand object, in the examples of Figures 12 and 13, the display device 1a displays only the virtual operation object and the virtual hand object. Furthermore, only the keyboard image 210 is shown as the virtual operation object. In Figures 12 and 13, a two-dimensional virtual object representing a typical hand shape is used as the virtual hand object. In Figure 12, the detection unit 72 recognizes only the right hand in the image data, and the display control unit 71 controls the display unit 20 to display an image of the right hand as the virtual hand object. On the other hand, in Figure 13, the detection unit 72 recognizes both the left and right hands in the image data, and the display control unit 71 controls the display unit 20 to display images of both hands as the virtual hand objects. The display control unit 71 controls the display of the virtual hand object so that it moves on the same plane as the two-dimensional virtual operation object in accordance with the user's actual hand movements. As a result, the virtual hand object cannot move in the depth direction relative to the virtual operation object, and can only move on the same plane as the virtual operation object. Therefore, the user can move their hand to accurately and easily move a predetermined part of the virtual hand object (e.g., the index finger) to a desired location on the virtual operation object, for example, to a desired key image on the keyboard image 210.

[0045] Furthermore, in the first embodiment, when the display control unit 71 receives information regarding the content of a specific action from the action determination unit 73, it recognizes the content of the specific action performed by the user and controls the display of the virtual operation object and / or the virtual hand object according to the recognized content of the specific action.

[0046] Specifically, when the display control unit 71 receives information from the operation determination unit 73 indicating that a tap (click) operation has been performed, it controls the display of the virtual operation object so that the portion of the virtual hand corresponding to the finger that performed the tap (click) operation is highlighted. Therefore, for example, if a keyboard image 210 is displayed as the operation screen, and the index finger portion of the right hand in the virtual hand object is positioned on a desired key image, when the user performs a tap (click) operation (gesture) with their right index finger, the key image on which the index finger portion of the virtual hand object is positioned is highlighted. In the examples in Figures 12 and 13, the highlighted key image when the user performs a tap (click) operation is indicated by diagonal lines. Here, in Figure 12, the user performs a tap (click) operation with their right index finger, and in Figure 13, the user performs a tap (click) operation with their right middle finger. On the other hand, when the operation determination unit 73 sends information indicating that a tap (click) operation has been performed as information regarding the content of a specific operation, the display control unit 71 can also control the display of the virtual hand object so that the portion of the virtual hand object corresponding to the finger that performed the tap (click) operation moves in the depth direction relative to the virtual operation object. For example, if the user performs a tap (click) operation (gesture) with the index finger of their right hand, the portion of the virtual hand object corresponding to the index finger will move in the depth direction. When the operation determination unit 73 sends information indicating that a tap (click) operation has been performed as information regarding the content of a specific operation, the display control unit 71 can choose whether to perform the above-mentioned highlight display or depth-direction movement display, which can be set in advance on a predetermined settings screen. In the first embodiment, the above-mentioned highlight display control is set as the default. Note that the present invention is not limited to the above-mentioned highlight display, and any display that is different from the normal display may be used.

[0047] Furthermore, for example, suppose a keyboard image 210, as shown in Figures 12 and 13, is displayed as a virtual object for operation, and the index finger of a virtual hand object is positioned on a character key image on the keyboard image 210, and the user performs a tap (click) action (gesture) with their index finger. In this case, the display control unit 71 knows the configuration of the keyboard image 210 in advance based on the data about the keyboard image 210 stored in the storage unit 90, so when information is sent from the action determination unit 73 that a tap (click) action has been performed with the index finger, it can identify the character key image on which the part of the virtual hand object corresponding to the index finger that the user tapped (clicked) is located. The display control unit 71 then recognizes the input instruction for the character corresponding to the identified character key image and displays the instructed character at a predetermined location (for example, the display area 220) on the virtual object for operation. If the virtual object for operation does not have a display area 220, the instructed character may be displayed on a virtual image other than the virtual object for operation.

[0048] Furthermore, when the display control unit 71 receives information from the operation determination unit 73 indicating that a scroll operation has been performed, it recognizes that a scroll operation has been performed by the user and moves the virtual operation object up, down, left, or right according to the recognized scroll operation instruction and displays it. When the display control unit 71 receives information from the operation determination unit 73 indicating that a pinch-in operation has been performed, it recognizes that a pinch-in operation has been performed by the user and shrinks the virtual operation object according to the recognized pinch-in operation instruction and displays it. Then, when the display control unit 71 receives information from the operation determination unit 73 indicating that a pinch-out operation has been performed, it recognizes that a pinch-out operation has been performed by the user and enlarges the virtual operation object according to the recognized pinch-out operation instruction and displays it.

[0049] The communication unit 80 communicates information with the outside world. The storage unit 90 stores various programs and data. The programs stored in the storage unit 90 include programs for the display control unit 71, detection unit 72, and operation determination unit 73 to realize the processing functions described above. On the other hand, the data stored in the storage unit 90 includes, for example, image data of the original image corresponding to the virtual image (including the virtual object for operation and the virtual hand object), and data related to the virtual object for operation and the virtual hand object. Specifically, the data related to the virtual object for operation includes data indicating the size, shape, content, and structure of the virtual object for operation, and the data related to the virtual hand object includes data indicating the size, shape, and structure of the virtual hand object. Alternatively, these programs and data may be stored in an external server device, and the control unit 70 may access that server device via the communication unit 80 to obtain these programs and data.

[0050] Next, the display processing of the virtual operation object and the virtual hand object by the display control unit 71 in the display device 1a of the first embodiment will be described. Figure 14 is a flowchart illustrating the procedure for displaying the virtual operation object and the virtual hand object by the display control unit 71 in the display device 1a of the first embodiment.

[0051] The user instructs the display of a desired virtual object for operation (a two-dimensional virtual object) by voice through the microphone unit 50, or by using the touchpad unit 40 or other means. Upon receiving this instruction, the display control unit 71 of the control unit 70 performs processing according to the processing flow shown in Figure 14. Specifically, when the display control unit 71 receives an instruction to display a virtual object for operation (S11), it reads the image data of the original image corresponding to the instructed virtual object for operation and the data related to that virtual object for operation from the storage unit 90, and displays the original image corresponding to that virtual object for operation on the half-mirror of the display unit 20 (S12). As a result, the user can see the virtual object for operation as if it were floating in space to the right of their field of view.

[0052] Next, the control unit 70 sends an instruction to the imaging unit 30 to start imaging, and controls the imaging operation of the imaging unit 30. As a result, the imaging unit 30 images a wide area including the area in front of the user, and the image data obtained is sent to the control unit 70. Here, the image data obtained by the imaging unit 30 includes the user's hand. At this time, the camera unit 31 of the imaging unit 30 measures the distance to the user's hand, and the distance data obtained is also sent to the control unit 70. The detection unit 72 of the control unit 70 uses this image data and distance data to generate three-dimensional position information about the user's hand movement at regular time intervals, and outputs it to the motion determination unit 73 and the display control unit 71 each time. Here, the three-dimensional position information consists of two-dimensional position information (XY coordinate information) for identifying the hand on a plane perpendicular to the optical axis of the camera unit 31, and one-dimensional position information (Z coordinate information) for identifying the hand in the direction of the optical axis of the camera unit 31. The time interval at which the detection unit 72 generates the three-dimensional position information is very short.

[0053] When the display control unit 71 receives three-dimensional position information as information regarding hand movement from the detection unit 72 (S13), it determines the display position of the virtual hand object on the plane containing the virtual operation object based on the two-dimensional position information (XY coordinate information) used to identify the hand on a plane perpendicular to the optical axis of the camera unit 31, and controls the display of the virtual hand object so that it is positioned at the determined location (S14). When this display control of the virtual hand object is performed each time three-dimensional position information is sent from the detection unit 72, the user can see the virtual hand object moving on the same plane as the two-dimensional virtual operation object in accordance with the actual movement of their own hand. As a result, the virtual hand object cannot move in the depth direction relative to the virtual operation object and only moves on the same plane as the virtual operation object, so the user can move their own hand to accurately and easily move the virtual hand object to a desired location on the virtual operation object.

[0054] Meanwhile, when the motion determination unit 73 receives three-dimensional position information as information about hand movement sent from the detection unit 72, it determines, based on the sent three-dimensional position information, whether the user has performed any of the specific actions (gestures) including tapping (clicking), scrolling, pinching in, and pinching out using the fingers of their hand. If the motion determination unit 75 determines that any of the specific actions have been performed, it outputs information about the content of that specific action to the display control unit 71. When the display control unit 71 receives information about the content of the specific action from the motion determination unit 73 (S15), it recognizes the content of the specific action performed by the user based on the information about the content of the specific action sent, and controls the display of the virtual image, particularly the display of the virtual operation object and / or the virtual hand object, according to the recognized content of the specific action (S16). The content of the display processing in step S16 is shown in Figure 15. Note that if the display control unit 71 determines in step S15 that it has not received information about the content of the specific action from the motion determination unit 73, it proceeds to step S17.

[0055] Figure 15 is a flowchart illustrating the processing procedure for controlling the display of a virtual image (including virtual objects) as shown in step S16 of the processing flow in Figure 14. In explaining the processing flow in Figure 15, it is assumed that four specific operations are predetermined: tap (click), scroll, pinch-in, and pinch-out.

[0056] When the display control unit 71 determines that it has received information from the operation determination unit 73 indicating that a tap (click) operation has been performed (S21), it highlights the portion of the virtual hand object on the operation virtual object where the finger corresponding to the finger that performed the tap (click) operation is located (S22). In particular, if the virtual hand object is displayed on the keyboard image 210 which is the operation virtual object, the display control unit 71 highlights the key image on which the finger portion of the virtual hand object corresponding to the finger that performed the tap (click) operation is located, as shown in Figure 12 or Figure 13, and recognizes that there has been an input instruction for the character represented by that key image, and displays that character, for example, on the input character display unit 220 of the character input screen 200. After that, it terminates the processing according to the processing flow in Figure 15. On the other hand, if the determination in step S21 is negative, it proceeds to step S23.

[0057] In step S23, the display control unit 71 determines whether information indicating that a scroll operation has been performed has been sent from the operation determination unit 73 as information related to a specific operation. If the display control unit 71 determines that information indicating that a scroll operation has been performed has been sent, it controls the display of the currently displayed virtual operation object so that it moves up or down or left or right (S24). After that, it terminates the processing according to the processing flow in Figure 15. On the other hand, if the determination in step S23 is negative, the process proceeds to step S25.

[0058] In step S25, the display control unit 71 determines whether information indicating a pinch-in operation has been sent from the operation determination unit 73 as information regarding a specific operation. If the display control unit 71 determines that information indicating a pinch-in operation has been sent, it controls the display of the virtual operation object so that the currently displayed virtual operation object shrinks (S26). After that, it terminates the processing according to the processing flow in Figure 15. On the other hand, if the determination in step S25 is negative, the display control unit 71 determines that information indicating a pinch-out operation has been sent from the operation determination unit 73 as information regarding a specific operation, since the determinations in steps S21, S23, and S25 are all negative. As a result, the display control unit 71 controls the display of the virtual operation object so that the currently displayed virtual operation object enlarges (S27). After that, it terminates the processing according to the processing flow in Figure 15. Once the display control of the virtual image (including the virtual object) in step S16 is completed, the process moves to step S17 in Figure 14.

[0059] In step S17 of Figure 14, the display control unit 71 of the control unit 70 determines whether it has received an instruction to terminate the display of the virtual operation object. The user can instruct the termination of the display of the virtual operation object by voice from the microphone unit 50 or by operation using the touchpad unit 40. When the display control unit 71 receives an instruction to terminate the display of the virtual operation object, it hides the virtual operation object and the virtual hand object (S18). The control unit 70 also sends an instruction to the imaging unit 30 to terminate the imaging operation. This terminates the display process of the virtual operation object and the virtual hand object shown in Figure 14. On the other hand, if the display control unit 71 determines in step S17 that it has not received an instruction to terminate the display of the virtual operation object, the process proceeds to step S13.

[0060] In the display device of the first embodiment, the display control unit controls the display of the virtual hand object so that the virtual hand object moves on the same plane as the two-dimensional virtual operation object in accordance with the user's hand movements. As a result, the virtual hand object cannot move in the depth direction relative to the virtual operation object, and can only move on the same plane as the virtual operation object. Therefore, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the virtual operation object. Consequently, compared to conventional devices in which the user operates a keyboard (virtual operation object) that appears to float in space, for example, the display device of the first embodiment allows for accurate and easy operation of the virtual operation object.

[0061] [Second Embodiment] Next, a display device according to a second embodiment of the present invention will be described. Figure 16 is a schematic block diagram of the display device according to the second embodiment of the present invention. In the second embodiment, components having the same functions as those in the first embodiment described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0062] The display device 1b of the second embodiment is configured to allow the user to view virtual images, and as shown in Figure 16, it comprises glasses 10 as a wearable device worn on the user's head, a display unit 20 that displays the original image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, a speaker unit 60, a communication unit 80, and a terminal 100. The display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, speaker unit 60, and communication unit 80 are provided on the glasses 10. In contrast, the terminal 100 is configured separately from the glasses 10. In order to simplify the explanation of the terminal 100, only the parts directly related to the present invention will be described in detail, and the explanation of other parts will be omitted.

[0063] The communication unit 80 is for wireless communication between the various devices (display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60) provided on the glasses 10 and the terminal 100. The control of the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 is performed by the terminal 100 via wireless communication. For this reason, unlike the first embodiment, the second embodiment does not have a control unit or the like for controlling the display unit 20 on the glasses 10.

[0064] The main difference between the display device 1b of this second embodiment and the display device 1a of the first embodiment is that the display device 1b includes a terminal 100 configured separately from the glasses 10, and the terminal 100 controls the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. The other configurations of the display device 1b of the second embodiment are the same as those of the display device 1a of the first embodiment.

[0065] In the second embodiment, the terminal 100 controls the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. Here, an existing terminal such as a smartphone or tablet is used as the terminal 100. As shown in Figure 16, the terminal 100 includes a terminal display unit 110, a communication unit 120, a control unit 130, and a storage unit 140. The terminal display unit 110 is a liquid crystal display device provided on the surface of the terminal 100. The screen of the terminal display unit 110 is also equipped with a touch panel. Various screens such as a home screen, menu screen, application screen, and text input screen are displayed on the screen of the terminal display unit 110. The user can give various instructions to the terminal 100 by performing touch operations on these screens. Note that the terminal 100 is not limited to existing terminals such as smartphones or tablet devices; a dedicated terminal can also be used. In this case, the dedicated terminal does not need to be equipped with a terminal display unit.

[0066] Furthermore, terminal 100 has the function of wirelessly communicating with the outside world, and this function is realized by the communication unit 120. Of course, terminal 100 can wirelessly communicate with the display unit 20, imaging unit 30, etc., via the communication unit 120 and the communication unit 80 provided in the glasses 10. In this respect, terminal 100 is wirelessly connected to the display unit 20, imaging unit 30, etc. Here, for example, Bluetooth® can be used as the wireless communication method between terminal 100 and the display unit 20, imaging unit 30, etc.

[0067] The control unit 130 includes a central processing unit (CPU) and other components, and controls the entire terminal 100, as well as the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. In other words, this control unit 130 has the same functions as the control unit 70 in the display device 1a of the first embodiment, and as shown in Figure 16, it includes a display control unit 71, a detection unit 72, and an operation determination unit 73. In addition, the storage unit 140 of the terminal 100 stores various programs and data. These programs and data include the programs and data stored in the storage unit 90 of the display device 1a of the first embodiment.

[0068] The display device of the second embodiment provides the same effects as that of the first embodiment. That is, by using the display device of the second embodiment, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the two-dimensional virtual control object. This allows for more accurate and easier operation of the virtual control object compared to conventional devices where the user operates on a keyboard (virtual control object) that appears to float in space using their hand.

[0069] In particular, in the display device of the second embodiment, the terminal having a control unit that controls each part such as the display unit and imaging unit provided on the glasses, and the glasses themselves are configured as separate units. Therefore, existing terminals such as smartphones and tablet devices can be used as the terminal. By using existing terminals in this way, the number of parts in the glasses can be reduced, and the configuration of the glasses can be simplified. A dedicated terminal may also be used as the terminal, in which case the dedicated terminal does not need to be provided with a display unit for the terminal.

[0070] [Third Embodiment] Next, a display device according to a third embodiment of the present invention will be described. Figure 17 is a schematic block diagram of the display device according to the third embodiment of the present invention. In the third embodiment, components that have the same functions as those of the second embodiment described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0071] The display device 1c of the third embodiment is configured to allow the user to view virtual images, and as shown in Figure 17, comprises glasses 10 as a wearable device worn on the user's head, a display unit 20 that displays the original image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, a speaker unit 60, a terminal 100, and a cable 300 connecting the glasses 10 and the terminal 100. The display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 are provided on the glasses 10. In contrast, the terminal 100 is configured separately from the glasses 10. Regarding the terminal 100, in order to simplify its explanation, only the parts directly related to the present invention will be described in detail, and the explanation of other parts will be omitted.

[0072] The glasses 10 are provided with connection terminals (not shown) at designated locations for connecting the cable 300. In the third embodiment, the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 are controlled by the terminal 100 via wired communication using the cable 300. Therefore, unlike the first embodiment, the glasses 10 are not provided with a control unit or the like for controlling the display unit 20.

[0073] The main difference between the display device 1c of the third embodiment and the display device 1b of the second embodiment is that the glasses 10 and the terminal 100 are connected by a wire using a cable 300. The other configurations of the display device 1c of the third embodiment are the same as those of the display device 1b of the second embodiment.

[0074] As shown in Figure 17, terminal 100 includes a terminal display unit 110, a communication unit 120, a control unit 130, a storage unit 140, and a connection terminal (not shown) as an interface. Cable 300 is connected to the connection terminal of terminal 100. The glasses 10 and terminal 100 are connected by cable 300, and terminal 100 can communicate with the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 via this cable 300. For example, HDMI® terminals can be used as the connection terminals of terminal 100 and the connection terminals provided on the glasses 10, and an HDMI® cable can be used as the cable 300.

[0075] The control unit 130 is equipped with a central processing unit (CPU) and other components, and controls the entire terminal 100, as well as the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. In other words, this control unit 130 has the same functions as the control unit 70 in the display device 1a of the first embodiment, and as shown in Figure 17, it includes a display control unit 71, a detection unit 72, and an operation determination unit 73. In addition, the storage unit 140 of the terminal 100 stores various programs and data. These programs and data include the programs and data stored in the storage unit 90 of the display device 1a of the first embodiment.

[0076] The display device of the third embodiment provides the same effects as that of the second embodiment. That is, using the display device of the third embodiment, the user can easily move the virtual hand object to a desired location on the two-dimensional virtual operation object by moving their hand. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices where the user operates on a keyboard (virtual operation object) displayed as if floating in space. Furthermore, by configuring the terminal, which has a control unit that controls each part such as the display unit and imaging unit provided on the glasses, and the glasses as separate components, existing terminals such as smartphones and tablet devices can be used as the terminal, thus reducing the number of parts in the glasses and simplifying the glasses' structure. A dedicated terminal may also be used as the terminal, in which case the dedicated terminal does not need to be equipped with a terminal display unit.

[0077] [Fourth Embodiment] Next, a display device according to the fourth embodiment of the present invention will be described. Figure 18(a) is a schematic perspective view of the display device according to the fourth embodiment of the present invention, and Figure 18(b) is a schematic cross-sectional view of the display device in the direction of arrow AA. Figure 19 is a schematic block diagram of the display device according to the fourth embodiment. In the fourth embodiment, components having the same functions as those of the first embodiment described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0078] In the fourth embodiment, a case will be described in which the display device of the present invention is applied to a glasses-type terminal that is worn on the user's head like eyeglasses. The display device 1d of the fourth embodiment is hardware for MR or VR configured to allow the user to view virtual images, and as shown in Figures 18 and 19, it comprises eyeglasses 10 as a wearable device worn on the user's head, a display unit 20d provided on the eyeglasses 10 for displaying the original image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, a speaker unit 60, a control unit 70, a communication unit 80, and a storage unit 90. Note that in Figure 18(b), the imaging unit 30 and the temple portion of the eyeglasses 10 are not shown.

[0079] The display device 1d of this fourth embodiment differs from the display device 1a of the first embodiment in the configuration of the display unit 20d. The other configurations of the display device 1d of the fourth embodiment are the same as those of the display device 1a of the first embodiment.

[0080] As shown in Figure 18, the display unit 20d is built into the left and right lens units 11, 11 of the eyeglasses 10 and comprises display devices 25, 25 and lenses 26, 26 as an optical system. In each lens unit 11, the display device 25 is located on the outside and the lens 26 is located on the inside. As the display device 25, liquid crystal panels, organic EL panels, inorganic EL panels, macro displays, etc., can be used. A microdisplay is an ultra-small display device that can display clear images with high resolution. Microdisplays include microLED displays and microOLED displays that utilize organic EL technology. In the fourth embodiment, a display device 25 is located in each of the left and right lens units 11. Different images or videos can be displayed on these left and right display devices 25, 25. In particular, for example, by displaying an image for the left eye on the left display device 25 and an image for the right eye on the right display device 25, the display device 1d can display a three-dimensional virtual image that the user can view in 3D.

[0081] The display device 25 displays images or videos representing the external environment or VR, as well as images or videos (original images) corresponding to two-dimensional virtual images (such as virtual objects for manipulation or virtual hand objects). It is also possible to display the original images corresponding to virtual images on the display device 25, overlaid on the images or videos representing the external environment or VR.

[0082] The control unit 70 includes a central processing unit (CPU) and controls the entire display device 1d of the fourth embodiment. Specifically, the control unit 70 controls the display of virtual images (including virtual objects) and the imaging by the imaging unit 30 by controlling the display unit 20d. Specifically, as shown in Figure 19, the control unit 70 includes a display control unit 71, a detection unit 72, and an operation determination unit 73. The display control unit 71 controls the display of various virtual images, including virtual objects for operation, by controlling the display unit 20d. When the display control unit 71 controls the display device 25 of the display unit 20d to project the original image corresponding to the virtual image (including virtual object) onto the display device 25, the user perceives the virtual image as if floating in space by viewing the original image projected onto the display device 25 through the lens 26. Furthermore, when the display unit 20d is displaying a two-dimensional virtual object for operation, the display control unit 71 controls the display of the virtual hand object based on information about hand movements sent from the detection unit 72, so that the virtual hand object moves on the same plane as the virtual operation object in accordance with the user's hand movements. The display control unit 71 can also control the display of images or videos representing the external environment or VR by controlling the display unit 20d.

[0083] The display device of the fourth embodiment provides the same effects as that of the first embodiment. That is, by using the display device of the fourth embodiment, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the two-dimensional virtual operation object. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices where the user operates their hand on a keyboard that appears to float in space.

[0084] In the fourth embodiment described above, the case in which eyeglasses are used as the wearable device attached to the user's head was explained, but it is also possible to use a head-mounted display (HMD) instead of eyeglasses. In other words, the display device of the present invention can also be applied to a head-mounted display (HMD) type terminal. Figure 20(a) is a schematic perspective view of a display device which is a modified example of the fourth embodiment, and Figure 20(b) is a schematic cross-sectional view of the display device in the direction of arrow BB. The display device 1e, a modified example of the fourth embodiment, is a head-mounted display (HMD) type terminal configured to allow the user to view virtual images (including virtual objects). In this display device 1e, as in the fourth embodiment, the display unit 20d is built into the lens units 11, 11 of the goggles and comprises display devices 25, 25 and lenses 26, 26 as an optical system. In the display device 1e, a modified example of the fourth embodiment, the imaging unit 30 is attached to the front of the goggles, but the imaging unit 30 is omitted in Figure 20. Also, in Figure 20(b), the display of the goggle band is omitted. Furthermore, the display devices 25, 25 display images or videos representing the external environment or VR, as well as images or videos (original images) corresponding to two-dimensional virtual images (including virtual objects for manipulation and virtual hand objects, etc.). It is also possible to display the original images corresponding to virtual images on the display devices 25, 25, overlaid on the images or videos representing the external environment or VR.

[0085] [Fifth Embodiment] Next, a display device according to the fifth embodiment of the present invention will be described. Figure 21 is a schematic block diagram of the display device according to the fifth embodiment of the present invention. In the fifth embodiment, components having the same functions as those of the fourth embodiment described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0086] The display device 1f of the fifth embodiment is configured to allow the user to view virtual images, and as shown in Figure 21, it comprises glasses 10 as a wearable device worn on the user's head, a display unit 20d that displays the original image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, a speaker unit 60, a communication unit 80, and a terminal 100. The display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, speaker unit 60, and communication unit 80 are provided on the glasses 10. In contrast, the terminal 100 is configured separately from the glasses 10.

[0087] The communication unit 80 is for wireless communication between the various devices (display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60) installed on the glasses 10 and the terminal 100. The display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 are controlled by the terminal 100 via wireless communication.

[0088] The main difference between the display device 1f of this fifth embodiment and the display device 1e of the fourth embodiment is that the display device 1f includes a terminal 100 configured separately from the glasses 10, and the terminal 100 controls the display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. The other configurations of the display device 1f of the fifth embodiment are the same as those of the display device 1e of the fourth embodiment.

[0089] As shown in Figure 21, terminal 100 includes a terminal display unit 110, a communication unit 120, a control unit 130, and a storage unit 140. Terminal 100 has the function of performing wireless communication with the outside world, and this function is realized by the communication unit 120.

[0090] The control unit 130 is equipped with a central processing unit (CPU) and other components, and controls the entire terminal 100, as well as the display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. Specifically, this control unit 130 includes a display control unit 71, a detection unit 72, and an operation determination unit 73.

[0091] The display device of the fifth embodiment provides the same effects as that of the fourth embodiment. That is, by using the display device of the fifth embodiment, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the two-dimensional virtual operation object. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices in which the user operates with their hand on a keyboard (virtual operation object) that appears to float in space.

[0092] In particular, in the display device of the fifth embodiment, the terminal having a control unit that controls each part such as the display unit and imaging unit provided on the glasses, and the glasses themselves are configured as separate units. Therefore, existing terminals such as smartphones and tablet terminals can be used as the terminal. By using existing terminals in this way, the number of parts in the glasses can be reduced, and the configuration of the glasses can be simplified. A dedicated terminal may also be used as the terminal, in which case the dedicated terminal does not need to be provided with a display unit for the terminal. Furthermore, as in the modified example of the fourth embodiment, the glasses as the wearable device may be an HMD type terminal.

[0093] [Sixth Embodiment] Next, a display device according to the sixth embodiment of the present invention will be described. Figure 22 is a schematic block diagram of the display device according to the sixth embodiment of the present invention. In the sixth embodiment, components having the same functions as those of the fifth embodiment described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0094] The sixth embodiment of the display device 1g is configured to allow the user to view virtual images, and as shown in Figure 22, it comprises glasses 10 as a wearable device attached to the user's head, a display unit 20d that displays the original image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, a speaker unit 60, a terminal 100, and a cable 300 connecting the glasses 10 and the terminal 100. The display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 are provided on the glasses 10. In contrast, the terminal 100 is configured separately from the glasses 10.

[0095] A connection terminal (not shown) for connecting a cable 300 is provided at a predetermined location on the glasses 10. In the sixth embodiment, the display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 are controlled by the terminal 100 via wired communication using the cable 300.

[0096] The main difference between the display device 1g of the sixth embodiment and the display device 1f of the fifth embodiment is that the glasses 10 and the terminal 100 are connected by a wire using a cable 300. The other configurations of the display device 1g of the sixth embodiment are the same as those of the display device 1f of the fifth embodiment.

[0097] As shown in Figure 22, terminal 100 includes a terminal display unit 110, a communication unit 120, a control unit 130, a storage unit 140, and a connection terminal (not shown) as an interface. Cable 300 is connected to the connection terminal of terminal 100. The glasses 10 and terminal 100 are connected by cable 300, and terminal 100 can communicate with the display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60 via this cable 300.

[0098] The control unit 130 is equipped with a central processing unit (CPU) and other components, and controls the entire terminal 100, as well as the display unit 20d, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. Specifically, this control unit 130 includes a display control unit 71, a detection unit 72, and an operation determination unit 73.

[0099] The display device of the sixth embodiment provides the same effects as that of the fifth embodiment. That is, using the display device of the sixth embodiment, the user can easily move the virtual hand object to a desired location on the two-dimensional virtual operation object by moving their hand. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices where the user operates on a keyboard (virtual operation object) displayed as if floating in space. Furthermore, by configuring the terminal, which has a control unit that controls various parts such as the display unit and imaging unit provided on the glasses, and the glasses as separate components, existing terminals such as smartphones and tablet terminals can be used as the terminal, thus reducing the number of parts in the glasses and simplifying the glasses' configuration. A dedicated terminal may also be used as the terminal, in which case the dedicated terminal does not need to be provided with a terminal display unit. Also, as in the modified example of the fourth embodiment, the glasses as the wearable device may be an HMD type terminal.

[0100] [Seventh Embodiment] Next, a seventh embodiment of the present invention, a display device, will be described. Figure 23 is a schematic front view of the seventh embodiment of the present invention, and Figure 24 is a schematic block diagram of the seventh embodiment of the display device. In the seventh embodiment, components having the same functions as those in the first to third embodiments described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0101] The seventh embodiment describes the case in which the display device of the present invention is applied to a so-called smart contact lens for display, which is worn by the user on their eye. The display device 1h of the seventh embodiment is configured to allow the user to view virtual images, and as shown in Figures 23 and 24, it comprises a contact lens 400 worn on the user's eye, a display unit 20 that displays the original image of the virtual image, an imaging unit 30 for imaging the area in front of the user, a control unit 70, a communication unit 80, a storage unit 90, and an antenna unit 500. Here, since the head includes the eyes, the contact lens 400 can be considered a type of wearable device worn on the user's head. In the seventh embodiment, the display unit 20, imaging unit 30, control unit 70, communication unit 80, storage unit 90, and antenna unit 500 are all provided on the contact lens 400. Note that the imaging unit 30 and storage unit 90 are not shown in Figure 23.

[0102] As shown in Figure 23, the display unit 20 is located approximately in the center of the contact lens 400. This display unit 20 comprises a miniature projector, an optical system, and a half-mirror. For the optical system, for example, lenses, waveguides, etc., can be used. The half-mirror acts as a projection unit, onto which images or videos displayed on the miniature projector are projected via the optical system. The half-mirror of the display unit 20 can display ordinary characters, symbols, images, videos, etc. (original images of virtual images). Here, a small lens may be mounted between the display unit 20 and the eye to make it easier for the eye to focus on the image, etc. displayed on the half-mirror of the display unit 20. Alternatively, a hologram pattern may be displayed on the half-mirror of the display unit 20 to display the image, etc. in 3D at a distance from the eye, making it easier for the eye to focus on that image, etc. Furthermore, the display unit 20 may consist only of a miniature projector, and this miniature projector may be used to project characters, symbols, images, videos, etc., onto the retina of the user's eye.

[0103] The imaging unit 30 is used to image the user's hand, and in the seventh embodiment, it is provided at the peripheral edge of the contact lens 400. As shown in Figure 24, this imaging unit 30 comprises a camera unit 31, an image processing unit 32, and a camera control unit 33.

[0104] The control unit 70 includes a central processing unit (CPU) and controls the entire display device 1h of the seventh embodiment. Specifically, the control unit 70 controls the display of virtual images (including virtual objects) and controls imaging by the imaging unit 30. As shown in Figure 23, this control unit 70 is provided at the peripheral edge of the contact lens 400. Specifically, as shown in Figure 24, the control unit 70 includes a display control unit 71, a detection unit 72, and an operation determination unit 73.

[0105] The communication unit 80 communicates information with the outside world, and in the seventh embodiment, as shown in Figure 23, it is provided at the peripheral edge of the contact lens 400. The storage unit 90 stores various programs and data. This storage unit 90 is also provided at the peripheral edge of the contact lens 400.

[0106] The antenna unit 500 is for sending and receiving data, images, etc., to and from an external device. As shown in Figure 23, this antenna unit 500 is annular in shape and is provided at the peripheral end of the contact lens 400.

[0107] In the seventh embodiment, a miniature battery (not shown) for supplying power to the display unit 20, imaging unit 30, control unit 70, communication unit 80, and storage unit 90 is mounted on the contact lens 400. Alternatively, power may be supplied to each unit without mounting the miniature battery on the contact lens 400, by utilizing electromagnetic induction to receive power from an external power source via an external transmitting antenna to a receiving antenna mounted on the contact lens 400. Alternatively, the miniature battery may be mounted on the contact lens 400, and electromagnetic induction may be used to receive power from an external power source via an external transmitting antenna to a receiving antenna mounted on the contact lens 400.

[0108] The display device of the seventh embodiment provides the same effects as that of the first embodiment. That is, by using the display device of the seventh embodiment, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the two-dimensional virtual operation object. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices where the user operates their hand on a keyboard that appears to float in space.

[0109] [Eighth Embodiment] Next, a display device according to the eighth embodiment of the present invention will be described. Figure 25 is a schematic diagram of the display device according to the eighth embodiment of the present invention, and Figure 26 is a schematic block diagram of the display device according to the eighth embodiment. In the eighth embodiment, components that have the same functions as those in the first to seventh embodiments described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0110] In the eighth embodiment, similar to the seventh embodiment, we will describe the case in which the display device of the present invention is applied to a so-called smart contact lens for display, which is worn by the user on their eye. The display device 1m of the eighth embodiment is configured to allow the user to view virtual images, and as shown in Figures 25 and 26, it comprises a contact lens (wearable device) 400 worn on the user's eye, a display unit 20 that displays the original image of the virtual image, a communication unit 80, an antenna unit 500 (see Figure 23), and a terminal 600. In the eighth embodiment, the display unit 20, the communication unit 80, and the antenna unit 500 are provided on the contact lens 400. Note that the communication unit 80 and the antenna unit 500 are not shown in Figure 25.

[0111] The communication unit 80 is for the display unit 20, which is provided on the contact lens 400, to communicate wirelessly with the terminal 600. The display unit 20 is controlled by the terminal 600 via wireless communication through the antenna unit 500 and the communication unit 80. For this reason, unlike the seventh embodiment, the eighth embodiment does not have a control unit or the like for controlling the display unit 20 provided on the contact lens 400.

[0112] The terminal 600 is configured separately from the contact lens 400. In this embodiment, the terminal 600 is of a type that is worn on the user's ear, as shown in Figure 25. As shown in Figure 26, the terminal 600 includes a communication unit 120, a control unit 130, a storage unit 140, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, and a speaker unit 60. When the terminal 600 is worn on the user's ear, the lens of the camera unit 31 of the imaging unit 30 faces the direction of the user's line of sight. In the eighth embodiment, the terminal 600 does not have a terminal display unit.

[0113] Furthermore, the terminal 600 has the function of wirelessly communicating with the outside world, and this function is realized by the communication unit 120. Therefore, the terminal 600 can wirelessly communicate with the display unit 20 via the communication unit 120 and the communication unit 80 provided on the contact lens 400. In this respect, the terminal 600 is wirelessly connected to the display unit 20.

[0114] The control unit 130 is equipped with a central processing unit (CPU) and other components, and controls the entire terminal 600, as well as the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, and speaker unit 60. In other words, this control unit 130 has the same functions as the control unit 70 in the display device 1a of the first embodiment, and as shown in Figure 26, it includes a display control unit 71, a detection unit 72, and an operation determination unit 73. In addition, the storage unit 140 of the terminal 600 stores various programs and data. These programs and data include the programs and data stored in the storage unit 90 of the display device 1a of the first embodiment.

[0115] The display device of the eighth embodiment provides the same effects as that of the first embodiment. That is, by using the display device of the eighth embodiment, the user can move their hand to accurately and easily move the virtual hand object to a desired location on the two-dimensional virtual operation object. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices where the user operates their hand on a keyboard that appears to float in space.

[0116] In the eighth embodiment, the case in which the imaging unit 30 is provided on the terminal 600 was described, but the imaging unit 30 may also be provided on the contact lens 400. Figure 27 shows a schematic block diagram of a display device 1n, which is a modified version of the eighth embodiment configured to have the imaging unit 30 provided on the contact lens 400. In this modified display device 1n of the eighth embodiment, the terminal 600a controls the imaging unit 30 provided on the contact lens 400 via wireless communication.

[0117] [Ninth Embodiment] Next, a ninth embodiment of the present invention, a display device, will be described. Figure 28 is a schematic diagram of the ninth embodiment of the present invention, and Figure 29 is a schematic block diagram of the ninth embodiment of the display device. In the ninth embodiment, components that have the same functions as those in the first to eighth embodiments described above are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0118] In the ninth embodiment, similar to the eighth embodiment, we will describe a case in which the display device of the present invention is applied to a so-called smart contact lens for display, which is worn by the user on their eye. The display device 1p of the ninth embodiment is configured to allow the user to view virtual images, and as shown in Figures 28 and 29, it comprises a contact lens (wearable device) 400 worn on the user's eye, a display unit 20 that displays the original image of the virtual image, a communication unit 80, an antenna unit 500 (see Figure 23), and a terminal 700. In the ninth embodiment, similar to the eighth embodiment, the display unit 20, the communication unit 80, and the antenna unit 500 are provided on the contact lens 400.

[0119] The communication unit 80 is for the display unit 20, which is provided on the contact lens 400, to communicate wirelessly with the terminal 700. The control of the display unit 20 is performed by the terminal 700 via wireless communication through the antenna unit 500 and the communication unit 80. For this reason, in the ninth embodiment, as in the eighth embodiment, no control unit or the like for controlling the display unit 20 is provided on the contact lens 400.

[0120] The terminal 700 is configured separately from the contact lens 400. In the ninth embodiment, unlike the eighth embodiment, an existing terminal such as a smartphone is used as the terminal 700, as shown in Figure 28. As shown in Figure 29, this terminal 700 includes a terminal display unit 110, a communication unit 120, a control unit 130, a storage unit 140, an imaging unit 30 for imaging the area in front of the user, a touchpad unit 40, a microphone unit 50, and a speaker unit 60. Note that the terminal 700 is not limited to a smartphone; a tablet, smartwatch, digital audio player, personal computer, laptop computer, etc., can also be used.

[0121] For the imaging unit 30 on the terminal 700 to fully perform its role of imaging the user's hand, attention must be paid to the placement of the terminal 700. For example, a smartphone-type terminal 700 may be used by hanging it around the user's neck using a strap, or by placing it in the user's breast pocket. In this case, by positioning the camera unit 31 of the imaging unit 30 of the terminal 700 so that its lens faces the direction of the user's line of sight, the imaging unit 30 can image the area in front of the user.

[0122] The terminal 700 can communicate wirelessly with the outside world through the functions of the communication unit 120. Therefore, the terminal 700 can communicate wirelessly with the display unit 20 via the communication unit 120 and the communication unit 80 provided in the contact lens 400.

[0123] The control unit 130 is equipped with a central processing unit (CPU) and other components, and controls the entire terminal 700, as well as the display unit 20, imaging unit 30, touchpad unit 40, microphone unit 50, speaker unit 60, and terminal display unit 110. In other words, this control unit 130 has the same functions as the control unit 70 in the display device 1a of the first embodiment, and as shown in Figure 29, it includes a display control unit 71, a detection unit 72, and an operation determination unit 73. In addition, the storage unit 140 of the terminal 700 stores various programs and data.

[0124] The display device of the ninth embodiment provides the same effects as that of the first embodiment. That is, using the display device of the ninth embodiment, the user can move their hand to accurately and easily move a virtual hand object to a desired location on a two-dimensional virtual operation object. This allows for more accurate and easier operation of the virtual operation object compared to conventional devices where the user operates a keyboard displayed as if floating in space. Furthermore, by configuring the terminal having a control unit for controlling the display unit provided on the contact lens and the contact lens as separate components, and using an existing terminal such as a smartphone as the terminal, the number of parts provided on the contact lens can be reduced, simplifying the structure of the contact lens.

[0125] In the ninth embodiment, the case in which the imaging unit 30 is provided on the terminal 700 was described, but the imaging unit 30 may also be provided on the contact lens 400. Figure 30 shows a schematic block diagram of a display device 1q, which is a modified version of the ninth embodiment configured to have the imaging unit 30 provided on the contact lens 400. In this modified display device 1q of the ninth embodiment, the terminal 700a controls the imaging unit 30 provided on the contact lens 400 via wireless communication.

[0126] [Other embodiments] Furthermore, the present invention is not limited to the embodiments described above, and various modifications are possible within the scope of its essence.

[0127] In the first to third and seventh to ninth embodiments described above, the case in which the miniature projector 21 is disposed within a housing H attached to the temple portion of the eyeglasses 10 was explained, but the miniature projector 21 may also be built into the eyeglasses 10.

[0128] Furthermore, in the first to third and seventh to ninth embodiments described above, the display unit 20 was described as comprising a small projector 21 having a display device, an optical system 22, and a half-mirror as a projection unit onto which an image or video displayed on the display device of the small projector 21 is projected via the optical system 22. However, instead of a half-mirror, for example, a hologram sheet, a translucent screen, a transmissive screen, a hologram optical element, or a diffraction grating may be used.

[0129] Furthermore, in the first to third and seventh to ninth embodiments described above, the display unit 20 was described as having a small projector 21, an optical system 22, and a half mirror. However, the display unit may not include the small projector, optical system, and half mirror, and may consist only of a display device such as a liquid crystal panel, an organic EL panel, an inorganic EL panel, a micro OLED display, or a micro LED display. Alternatively, the display unit may consist of these display devices and a lens. Here, the display device may be transmissive, semi-transparent, transparent, or semi-transparent, or it may be opaque or non-transparent. This display device can be built into eyeglasses, contact lenses, or an HMD. When the user looks at the image or video (original image) displayed on the display device, it will appear as if the image or video is floating in space. This image or video that appears to be floating in space is a virtual image (including virtual objects). In addition, the display unit 20 may have multiple display devices. For example, multiple of the above display devices may be used, and these multiple display devices may be arranged on top of each other. These multiple display devices, arranged in a layered configuration, may be of different types (e.g., transparent and opaque) or different sizes. In this case, different images or videos can be displayed on each display device. For example, an image or video (original image) corresponding to a virtual object for operation (a two-dimensional virtual object) can be displayed on one display device, while an image or video of the external environment or VR can be displayed on another display device. Alternatively, an image or video of the external environment or VR can be displayed on one display device, while the image or video (original image) corresponding to the virtual object for operation is displayed on a display unit 20, which is positioned in front of this display device and includes a small projector 21, an optical system 22, and a projection unit. Furthermore, one of the above display devices may be used to overlay different images or videos onto that display device. For example, an image or video (original image) corresponding to a virtual object for operation and the external environment can be overlaid on a single display device.This configuration of the display unit 20 can, of course, be applied to each of the fourth to sixth embodiments.

[0130] In the first to third embodiments described above, the imaging unit 30 is described as being disposed within a housing H attached to the temple portion of the eyeglasses 10, but the imaging unit 30 may also be built into the eyeglasses 10. Furthermore, in the fourth to sixth embodiments described above (excluding the modification of the fourth embodiment), the imaging unit 30 may be attached to the temple portion of the eyeglasses 10.

[0131] In each of the embodiments described above, a three-dimensional Cartesian coordinate system was set up in which the direction of the camera's optical axis is the Z-axis direction, and the two orthogonal directions in the plane perpendicular to the camera's optical axis are the X-axis direction and the Y-axis direction, respectively, and three-dimensional position information for identifying the hand was obtained in the set three-dimensional Cartesian coordinate system. Specifically, the detection unit 72 obtained two-dimensional position information for identifying the hand on the plane perpendicular to the optical axis of the camera unit 31 included in the imaging unit 30 based on image data captured by the imaging unit 30, and obtained one-dimensional position information for identifying the hand in the optical axis direction of the camera unit 31 based on image data and distance data obtained by the imaging unit 30, and the three-dimensional position information composed of these two-dimensional and one-dimensional position information was obtained as information about the hand's movement. However, the three-dimensional position information for identifying the hand is not limited to the three-dimensional position information (XYZ coordinate information) of the hand in the above three-dimensional Cartesian coordinate system (XYZ coordinate system), but can be used for the three-dimensional position information of the hand in any three-dimensional Cartesian coordinate system, and the method by which the detection unit 72 obtains information about the hand's movement is not limited to the above method, but can use various methods. For example, the detection unit 72 may identify a two-dimensional plane on which the hand is located based on the image data and distance data obtained by the imaging unit 30, obtain two-dimensional position information to identify the hand on the identified two-dimensional plane, and obtain one-dimensional position information to identify the hand in a direction perpendicular to the identified two-dimensional plane, and acquire three-dimensional position information composed of these two-dimensional and one-dimensional position information as information related to the movement of the hand. In this case, the display control unit 71 can identify the two-dimensional plane on which the hand is located with the plane containing the two-dimensional virtual object for operation, and determine the display position of the virtual hand object on the virtual object for operation based on the two-dimensional position information for identifying the hand on the two-dimensional plane from the three-dimensional position information for identifying the hand sent from the detection unit 72. Furthermore, the display position of the virtual hand (and fingers) object on the virtual object for operation can be determined based on the three-dimensional position information (XYZ coordinate information) of the hand and fingers in an arbitrarily set three-dimensional coordinate system.Furthermore, the equation of a plane in three-dimensional space can be expressed as aX + bY + cZ = 0, where a, b, and c are constants. By determining the three-dimensional position coordinates of three arbitrary points on the hand and using them to define the three constants a, b, and c, the equation of the two-dimensional plane on which the hand exists in three-dimensional space can be determined.

[0132] In each of the embodiments described above, the imaging unit has the function of a distance measuring unit that measures the distance to the user's hand and outputs the measured distance data, and the detection unit obtains three-dimensional position information for identifying the hand based on the image data and distance data obtained by the imaging unit, and the obtained three-dimensional position information is acquired as information related to hand movement. However, the display device of the present invention may also include a position information measuring unit instead of a distance measuring unit and a detection unit that measures the three-dimensional position information of the user's hand in an arbitrarily set three-dimensional coordinate system and acquires the measured three-dimensional position information as information related to hand movement. In this case, the display control unit determines the display position of the virtual hand object on the virtual operation object based on the three-dimensional position information for identifying the hand sent from the position information measuring unit.

[0133] In the embodiments described above, the case where the virtual hand object is a two-dimensional virtual object was explained, but the virtual hand object may also be a three-dimensional virtual object.

[0134] In the above embodiments of the display device, the case in which the display control unit 71 controls the display of the operation virtual object as a two-dimensional virtual object has been described. However, the display control unit 71 can also control the display of a three-dimensional virtual object, and may be configured to control the display unit 20 so that the display device can display the operation virtual object as a three-dimensional virtual object. When the display device is displaying a three-dimensional operation virtual object, the display control unit 71 can control the display of the hand virtual object so that the hand virtual object moves on the surface of the three-dimensional operation virtual object (this surface includes a two-dimensional curved surface) or on a plane formed by its upper end surface in accordance with the movement of the user's hand. Here, the storage unit 90 stores image data of the original image corresponding to the three-dimensional virtual object (including the three-dimensional operation virtual object) and data related to the operation virtual object.

[0135] Figure 31 shows an example of a virtual hand object displayed on a two-dimensional virtual object for manipulation in three-dimensional space, and Figure 32 shows an example of a virtual hand object displayed on a three-dimensional virtual object for manipulation in three-dimensional space. In Figures 31 and 32, a virtual object representing a general hand shape is used as the virtual hand object. In Figure 31, the two-dimensional virtual object for manipulation is a planar keyboard image 210. This planar keyboard image 210 is a two-dimensional virtual object, and the surface of the keyboard represented by the planar keyboard image 210 is displayed as a two-dimensional plane in three-dimensional space, extending in the depth direction. In this case, the display control unit 71 controls the display of the virtual hand object so that it moves on the two-dimensional plane representing the surface of the planar keyboard image 210 in accordance with the user's hand movements, based on information (three-dimensional position information) about hand movements sent from the detection unit 72. In other words, the display control unit 71 can identify a two-dimensional plane representing the surface of the planar keyboard image 210 based on data relating to the planar keyboard image 210. Therefore, when a virtual hand object is on the planar keyboard image 210, the display control unit 71 controls the display of the virtual hand object so that it can move only on the two-dimensional plane representing the surface of the planar keyboard image 210. On the other hand, in Figure 32, the three-dimensional virtual operation object is an image of a cylindrical object. The image of this cylindrical object is a three-dimensional virtual object, and the surface of this cylindrical object is displayed as a two-dimensional curved surface (side) and a two-dimensional plane (top and bottom) in three-dimensional space. In this case, the display control unit 71 controls the display of the virtual hand object so that it moves on the two-dimensional curved surface and the two-dimensional plane representing the surface of the cylindrical object according to the user's hand movements, based on information (three-dimensional position information) regarding hand movements sent from the detection unit 72. Note that in the example of Figure 31, it is also possible to use a keyboard image representing a three-dimensional keyboard (a three-dimensional virtual operation object) as the virtual operation object instead of a planar keyboard image (a two-dimensional virtual operation object).In this case, for example, the display control unit 71 can identify the keytop surface (two-dimensional plane) formed by the upper end surface of the three-dimensional keyboard image (three-dimensional virtual object for operation) 210 based on data relating to the three-dimensional keyboard image 210. Therefore, when a virtual hand object is on the three-dimensional keyboard image 210, the display control unit controls the display of the virtual hand object so that the virtual hand object can move only on the keytop surface of the three-dimensional keyboard image 210. Here, Figure 33 shows an example of a plane formed by the upper end surface of the three-dimensional keyboard image, i.e., a keytop surface.

[0136] In addition, in the examples shown in Figures 31 and 32, the display control unit 71 may control the display of the virtual hand object so that it moves in three-dimensional space in accordance with the user's hand movements. Whether the display control unit 71 controls the movement display of the virtual object on its surface or its movement display in three-dimensional space can be set in advance on a predetermined settings screen.

[0137] In the embodiments described above, a two-dimensional virtual object representing a general hand shape was used as the virtual hand object. However, at the user's discretion, a two-dimensional or three-dimensional virtual object representing the shape of an object other than a hand can be used instead of a virtual hand object. For example, when a user is clenching some or all of their fingers, they can use that clenched hand as a mouse to give instructions to the virtual object for operation. In this case, the detection unit 72 determines whether the user is clenching some or all of their fingers based on the image data captured by the imaging unit 30, and when it determines that the user is clenching some or all of their fingers, it is configured to output information indicating that the user is clenching some or all of their fingers. The display control unit 71, upon receiving information from the detection unit 72 indicating that the user is gripping some or all of their fingers, uses a virtual mouse body modeled after the mouse body and a virtual mouse cursor modeled after the mouse cursor instead of a virtual hand object. The display control unit 71 displays the virtual mouse body at a predetermined location on the virtual image and controls the display of the virtual mouse cursor object so that it moves on the same plane as the operation virtual object in accordance with the hand movement, based on the hand movement information sent from the detection unit 72. The virtual mouse body and virtual mouse cursor objects are virtual mouse objects modeled after a mouse. When controlling the movement of the virtual mouse cursor object, the display control unit 71 can determine the amount of movement of the virtual mouse cursor object on the virtual image based on the actual hand movement distance calculated based on the hand movement information sent from the detection unit 71. The ratio between the actual hand movement distance and the movement distance of the virtual mouse cursor object on the virtual image may be changed via a settings screen or the like.

[0138] Figures 34 and 35 show examples of virtual objects for the mouse body. If the user is clenching all of their fingers, the detection unit 72 sends information to the display control unit 71 indicating that the user is clenching all of their fingers, and the display control unit 71 displays a virtual object representing only the outline of the mouse body at a predetermined location on the virtual image, as shown in Figure 34(a). If the user is clenching their hand with one finger extended, the detection unit 72 sends information to the display control unit 71 indicating that the user is clenching their hand with one finger extended, and the display control unit 71 displays a virtual object representing the shape of a mouse with only one click button at a predetermined location on the virtual image, as shown in Figure 34(b). Furthermore, if the user is clenching their fist with two fingers extended, the detection unit 72 sends information to the display control unit 71 indicating that the user is clenching their fist with two fingers extended. The display control unit 71 then displays a virtual object representing the shape of the mouse, with a right-click button and a left-click button drawn on it, at a predetermined location on the virtual image, as shown in Figures 35(a) and (b). Here, the two fingers extended by the user may be the thumb and index finger as shown in Figure 35(a), or the index finger and middle finger as shown in Figure 35(b).

[0139] Figure 36 shows an example of a virtual mouse body and virtual mouse cursor displayed on the same plane as a two-dimensional virtual object for operation. In Figure 36, a text input screen 200 is used as the virtual object for operation. Furthermore, since the detection unit 72 has recognized that the user is clenching their hand with two fingers extended, the display control unit 71 displays a virtual mouse body with a right-click button and a left-click button. When the operation determination unit 73 receives information from the detection unit 72 that the user is clenching their hand with two fingers extended, it determines, based on the information about hand movements received from the detection unit 72, which of the two extended fingers is being used to perform which of the specified operations. Then, when the display control unit 71 receives information from the operation determination unit 73 indicating that a click operation was performed using the left (or right) finger of the two fingers, it recognizes that a left-click operation (or right-click operation) of the mouse was performed on the portion of the operation virtual object corresponding to the virtual object of the mouse cursor at the time the click operation was performed, and controls the display of the operation virtual object and / or the display of the mouse virtual object according to the recognized left-click operation (or right-click operation).

[0140] On the other hand, even when the display control unit 71 displays a virtual object of the mouse body that has only one click button, the processing of the operation determination unit 73 and the display control unit 71 is the same as described above. That is, when the operation determination unit 73 receives information from the detection unit 72 that the user is holding their hand with one finger extended, it determines, based on the information about the hand movement received from the detection unit 72, which of the specific operations was performed using that extended finger. Then, when the display control unit 71 receives information from the operation determination unit 73 that a click operation was performed using one finger, it recognizes that a mouse click operation was performed on the portion of the operation virtual object corresponding to the virtual object of the mouse cursor at the time the click operation was performed, and controls the display of the operation virtual object and / or the display of the virtual mouse object according to the recognized click operation.

[0141] Furthermore, when the detection unit 72 sends information that the user is clenching all of their fingers, the operation determination unit 73 determines, based on the information about the hand movement sent from the detection unit 72, which of the specified operations was performed by the hand that is clenching all of its fingers. For example, if the user performs a knocking motion with the hand that is clenching all of its fingers, the operation determination unit 73 determines that the operation is a click operation. In other words, if the user is clenching all of their fingers, a knocking motion with that clenched hand becomes a click operation. When the display control unit 71 receives information from the operation determination unit 73 that a click operation was performed by the hand that is clenching all of its fingers, it recognizes that a mouse click operation was performed on the portion of the virtual operation object corresponding to the virtual mouse cursor object at the time the click operation was performed, and controls the display of the virtual operation object and / or the virtual mouse object according to the recognized click operation. Also, if the user performs a scrolling motion with the hand that is clenching all of its fingers, the operation determination unit 73 determines that the operation is a scroll operation.

[0142] In the example shown in Figure 36, the virtual mouse object is described as being composed of a virtual mouse body and a virtual mouse cursor. However, it is also possible to use only the virtual mouse cursor as the virtual mouse object without displaying the virtual mouse body. Furthermore, if the user is holding their hand with one finger extended, they may be able to switch between mouse operation and keyboard input so that they can use that extended finger to input on the keyboard.

[0143] Figure 37 shows an example of a virtual object of the mouse body and a virtual mouse cursor displayed on a three-dimensional virtual object for operation. In Figure 37, an image of a cylindrical object is used as the three-dimensional virtual object for operation. Furthermore, because the detection unit 72 recognizes that the user is holding their hand with two fingers extended, the display control unit 71 displays a virtual object of the mouse body that has a right-click button and a left-click button. In this case as well, when the operation determination unit 73 sends information that a click operation was performed using the left (or right) finger of the two fingers, the display control unit 71 recognizes that a left-click operation (or right-click operation) of the mouse was performed on the portion of the virtual object for operation corresponding to the virtual mouse cursor at the time the click operation was performed, and controls the display of the virtual object for operation and / or the virtual mouse object according to the recognized left-click operation (or right-click operation).

[0144] In the embodiments described above, the case where the virtual object for operation is modeled after a keyboard and mainly includes a character input screen with a keyboard image has been explained. However, the virtual object for operation is not limited to a character input screen and may be, for example, modeled after the numeric keypad of a smartphone or the operating surface of a remote control for a home appliance. In this case, after the display device displays the virtual object for operation, when the user moves their hand to move a virtual hand object over the virtual object for operation, the control unit 70 of the display device generates a command signal indicating the content of the operation and wirelessly transmits the generated command signal to the smartphone or home appliance via the communication unit 80. This allows the smartphone or home appliance to be operated remotely. Figure 38 shows an example of a virtual object for operation modeled after the numeric keypad of a smartphone. When the user enters a desired telephone number into the virtual object for operation as shown in Figure 38(a) or (b), the control unit 70 generates a command to make a call to that telephone number and transmits the command to the smartphone via the communication unit 80, allowing the user to make a call without holding the smartphone. Figure 39 shows an example of a virtual control object that models the operating surface of an air conditioner remote control. For example, when a user moves their hand to move a virtual hand object over the virtual control object shown in Figure 39, and then taps (clicks) on a button to lower the set temperature on that virtual control object, the control unit 70 generates a command to lower the set temperature and sends that command to the air conditioner via the communication unit 80, thereby allowing the user to easily lower the set temperature of the air conditioner.

[0145] In the embodiments described above, the display device of the present invention was described in the case where it is applied to a glasses-type terminal worn on the user's head like eyeglasses, or to a smart contact lens for display worn on the user's eyes. However, the applications of the display device of the present invention are not limited to these. The display device of the present invention can be applied to various display devices that display virtual images (videos) in environments using augmented reality (AR), virtual reality (VR), mixed reality (MR), etc. Specifically, it can be applied to glasses-type terminals, head-mounted display (HMD) type terminals, smart contact lenses for display, etc.

[0146] Furthermore, in each of the above embodiments, when a user performs key input operations (typing operations) with both hands while viewing a two-dimensional or three-dimensional virtual object for operation that models a keyboard, it is possible to identify the key being input using AI (Artificial Intelligence) technology. This will be explained in a little more detail. When a user performs key input operations using a normal keyboard, the relative positions of both hands, the relative positions of adjacent fingers, the shape of the hand, the position of each finger, etc., change. Therefore, by analyzing the content of these changes in the hand and fingers, it is possible to identify the key that is being input. From this, the generation of a trained model for identifying input keys is performed as follows. First, video data (image data) of a person using each finger of their hand to hit each key on an actual keyboard is acquired. Then, information about the key is added to the acquired video data of each key being hit. The video data with the key information added in this way becomes the training data. Next, a trained model can be generated by subjecting this training data to machine learning such as deep learning. This trained model takes image data captured by the imaging unit as input while the user performs key input actions with their hands while viewing a two-dimensional or three-dimensional virtual object that models the keyboard, and outputs the keyboard key corresponding to the key input action and the user's finger. The trained model thus generated is stored in the memory unit. In this case, the control unit is also provided with a discrimination unit that uses the trained model to determine the input key. That is, when image data captured while the user performs key input actions with their hands is sent from the imaging unit, this discrimination unit can use the trained model stored in the memory unit to determine which finger of the user is using to press which key based on the content of the key input action contained in the image data. Furthermore, the user may pre-configure which keys on the keyboard are pressed with each finger of their left and right hands on a settings screen. The information set on this settings screen is stored in the memory unit as information on the correspondence between fingers and keys.In this case, the discrimination unit can quickly determine which finger the user used to press which key by referring to the finger-key correspondence information in addition to the trained model stored in the memory unit. Of course, if the user inputs keys in a way that does not conform to the finger-key correspondence information, the AI ​​technology described above can be used to determine which finger the user used to press which key. Furthermore, the content of the finger-key correspondence information can be changed or deleted as appropriate on the settings screen described above. In addition, even when the display control unit identifies keys using AI technology as described above, it may control the display unit so that when a three-dimensional virtual keyboard image, which is a virtual object for operation, is displayed, the virtual hand object can only move on the keytop surface (two-dimensional plane) formed by the upper end surface of the three-dimensional keyboard image.

[0147] Furthermore, in the second, third, fifth, sixth, eighth, and ninth embodiments described above, a terminal configured separately from the wearable eyewear, HMD, or contact lenses has all the functions of the display control unit, detection unit, and operation determination unit. However, the terminal may have only some of the functions of the display control unit, detection unit, and operation determination unit.

[0148] Furthermore, while the seventh, eighth, and ninth embodiments described above describe the case where the smart contact lens is worn on one eye of the user, the smart contact lens may also be worn on both eyes of the user. In this case, in the seventh embodiment, the two smart contact lenses can communicate with each other. Also, in the eighth embodiment, one terminal 600 may control the display units 20 provided on the two smart contact lenses, or two terminals 600 may be provided, and each of the two terminals 600 may control the display unit 20 provided on one smart contact lens. If two terminals 600 are provided, the two terminals 600 can also communicate with each other. Furthermore, in the ninth embodiment, one terminal 700 may control the display units 20 provided on the two smart contact lenses, or two terminals 700 may be provided, and each of the two terminals 700 may control the display unit 20 provided on one smart contact lens. If two terminals 700 are provided, the two terminals 700 can also communicate with each other.

[0149] Furthermore, in the seventh, eighth, and ninth embodiments described above, when smart contact lenses are worn on both eyes of the user, different images or videos can be displayed on the display unit of the smart contact lens worn on the left eye and the display unit of the smart contact lens worn on the right eye. For example, by displaying a left-eye image on the display unit of the smart contact lens worn on the left eye and a right-eye image on the display unit of the smart contact lens worn on the right eye, the display device can display a three-dimensional virtual image (including virtual objects) that the user can view in 3D. [Industrial applicability]

[0150] As described above, in the display device of the present invention, the display control unit controls the display of the virtual hand object so that the virtual hand object moves on the same plane as the two-dimensional virtual operation object, or on the surface of the three-dimensional virtual operation object or on the plane formed by its upper end surface, in accordance with the user's hand movements. As a result, the virtual hand object cannot move in the depth direction relative to the virtual operation object and can only move on the same plane as the virtual operation object. This allows the user to move their hand to accurately and easily move the virtual hand object to a desired location on the virtual operation object. Therefore, compared to conventional devices in which the user operates a keyboard (virtual operation object) that appears to float in space, for example, the display device of the present invention allows for accurate and easy operation of the virtual operation object. Accordingly, the present invention can be applied to various display devices that display virtual images (videos) in environments using augmented reality (AR), virtual reality (VR), mixed reality (MR), etc. [Explanation of symbols]

[0151] 1a,1b,1c,1d,1e,1f,1g,1h,1m,1n,1p,1q Display device 10. Eyeglasses (wearable item) 11 Lens section 20,20d Display section 21. Small projector 22 Optical system 25 Display Devices 26 lenses 30 Imaging Unit 31 Camera unit (rangefinder unit) 32 Image Processing Unit 33 Camera Control Unit 40 Touchpad section 50 Microphone section 60 Speaker section 70 Control Unit 71 Display Control Unit 72 Detection unit 73 Operation judgment section 80 Communications Department 90 Memory section 100, 600, 600a, 700, 700a terminals 110 Display unit for terminals 120 Communications Department 130 Control Unit 140 Storage section 200-character input screen 210 Keyboard Images 220 display area 221 Search screen 2211 Keyword Input Section 2212 Search Results Display Section 300 Cable 400 contact lenses 500 Antenna section H, H1, H2 enclosure

Claims

1. A display device configured to allow a user to view virtual images, A device to be attached to the user's head, The aforementioned attachment includes a display unit that displays the original image, which is the image from which the virtual image is based, An imaging unit that captures the user's hand and outputs the captured image data, A detection unit that uses the image data obtained by the imaging unit to detect the user's hand movements and outputs information related to those hand movements, A display control unit that controls the display of the virtual image by controlling the display unit, It is equipped with, The display control unit controls the display unit to display a two-dimensional virtual object for operation and a virtual hand object modeled after a hand as elements of the virtual image, respectively, and controls the display of the virtual hand object so that it moves on the same plane as the two-dimensional virtual operation object in accordance with the hand movement, based on information about the hand movement sent from the detection unit.

2. The system further includes an action determination unit that determines whether the user performed any of the following specific actions, including tapping (clicking) and scrolling, using the hand, based on the image data obtained by the imaging unit and the information regarding the hand movement sent from the detection unit, and outputs information regarding the content of the specific action performed by the hand when it is determined that either of the actions has been performed. The display device according to claim 1, characterized in that the display control unit recognizes the content of the specific operation performed by the user when information regarding the content of the specific operation is sent from the operation determination unit, and controls the display of the virtual operation object and / or the virtual hand object according to the recognized content of the specific operation.

3. The display device according to claim 2, characterized in that when the display control unit receives information from the operation determination unit indicating that a tap (click) operation has been performed as information regarding the content of the specific operation, it controls the display of the virtual hand object so that the portion of the virtual hand object corresponding to the finger that performed the tap (click) operation moves in the depth direction relative to the virtual operation object.

4. The aforementioned virtual object for operation is a model of a keyboard, a model of a numeric keypad, or a model of the operating surface of a remote control. The display device according to claim 2, characterized in that when the display control unit receives information from the operation determination unit indicating that a tap (click) operation has been performed as information regarding the content of the specific operation, it controls the display of the virtual hand object so that the portion of the virtual hand object corresponding to the finger that performed the tap (click) operation moves in the depth direction relative to the virtual operation object.

5. The display device according to claim 2, characterized in that when the display control unit receives information from the operation determination unit indicating that a tap (click) operation has been performed as information regarding the content of the specific operation, it controls the display of the operation virtual object such that the portion of the operation virtual object on which the portion of the hand virtual object corresponding to the finger that performed the tap (click) operation is located is displayed differently from the normal display, including a highlighted display.

6. The aforementioned virtual object for operation is a model of a keyboard, a model of a numeric keypad, or a model of the operating surface of a remote control. The display device according to claim 2, characterized in that when the display control unit receives information from the operation determination unit indicating that a tap (click) operation has been performed as information regarding the content of the specific operation, it controls the display of the operation virtual object such that the portion of the operation virtual object on which the portion of the hand virtual object corresponding to the finger that performed the tap (click) operation is located is displayed differently from the normal display, including a highlighted display.

7. The imaging unit is equipped with a distance measuring unit function that measures the distance to the user's hand and outputs the measured distance data. The detection unit obtains three-dimensional position information for identifying the hand based on the image data and distance data obtained by the imaging unit, and acquires the obtained three-dimensional position information as information regarding the movement of the hand. The display device according to claim 1, characterized in that the display control unit determines the display position of the virtual hand object on the virtual operation object based on the three-dimensional position information for identifying the hand sent from the detection unit.

8. The system further includes a position information measurement unit that measures the three-dimensional position information of the user's hand in an arbitrarily set three-dimensional coordinate system and acquires the measured three-dimensional position information as information related to the movement of the hand. The display device according to claim 1, characterized in that the display control unit determines the display position of the virtual hand object on the virtual operation object based on the three-dimensional position information for identifying the hand sent from the position information measurement unit.

9. The detection unit determines, based on the image data obtained by the imaging unit, whether the user is gripping some or all of their fingers, and when it determines that the user is gripping some or all of their fingers, it outputs information indicating that the user is gripping some or all of their fingers. The display device according to claim 2, characterized in that when the detection unit receives information that the user is gripping part or all of their fingers, the display control unit, at the user's selection, uses a virtual object of a mouse body modeled after the mouse body and a virtual object of a mouse cursor modeled after the mouse cursor instead of the virtual object of the hand, displays the virtual object of the mouse body at a predetermined location on the virtual image, and controls the display of the virtual mouse cursor object based on information about the hand movement sent from the detection unit so that the virtual mouse cursor object moves on the same plane as the operation virtual object in accordance with the hand movement.

10. When the detection unit receives information from the detection unit indicating that the user is clenching their hand with one finger extended, the motion determination unit determines, based on the information regarding the hand movement received from the detection unit, which of the specified actions was performed using the extended finger. The display device according to claim 9, characterized in that when the display control unit receives information from the operation determination unit indicating that the click operation was performed using one finger, it recognizes that a mouse click operation was performed on the portion of the virtual operation object corresponding to the virtual object of the mouse cursor at the time the click operation was performed.

11. When the detection unit receives information from the detection unit indicating that the user is clenching their hand with two fingers extended, the operation determination unit determines, based on the information regarding the hand movement received from the detection unit, which of the two extended fingers was used to perform which of the specified operations. The display device according to claim 9, characterized in that when the display control unit receives information from the operation determination unit that the click operation was performed using the left (or right) finger of the two fingers, it recognizes that a left-click operation (or right-click operation) of the mouse was performed on the portion of the virtual object for operation corresponding to the virtual object of the mouse cursor at the time the click operation was performed.

12. When the detection unit receives information from the detection unit indicating that the user is gripping with all of their fingers, the operation determination unit determines, based on the information regarding the hand movements received from the detection unit, which of the specified operations was performed with the hand gripping with all of its fingers. The display device according to claim 9, characterized in that when the display control unit receives information from the operation determination unit that the click operation was performed with a hand that was gripping all of the fingers, the display control unit recognizes that the mouse click operation was performed on the portion of the virtual operation object corresponding to the virtual object of the mouse cursor at the time the click operation was performed.

13. The display device according to claim 9, 10, 11, or 12, characterized in that when the display control unit controls the display unit to display the virtual object of the mouse cursor, it controls the movement of the virtual object of the mouse cursor on the virtual image by determining the amount of movement of the virtual object of the mouse cursor on the virtual image based on the actual distance of movement of the hand calculated based on the information on the movement of the hand sent from the detection unit.

14. The display control unit is configured to control the display unit to display a three-dimensional virtual operation object instead of a two-dimensional virtual operation object, and when the display unit is controlled to display the three-dimensional virtual operation object, the display of the virtual hand object is controlled so that the virtual hand object moves on the surface of the three-dimensional virtual operation object or on a plane formed by its upper end surface in accordance with the movement of the user's hand, as described in claim 1.

15. The display device according to claim 1, characterized in that the virtual hand object is a two-dimensional virtual object or a three-dimensional virtual object.

16. A storage unit that stores a trained model created by training with video data of a person using each finger of their hand to type on each key of a keyboard as training data, wherein the input is image data captured by the imaging unit when the user performs key input actions with their hands while looking at the two-dimensional or three-dimensional virtual object for operation that models the keyboard, and the output is the keyboard key corresponding to the key input action and the user's finger. When the image data captured while the user is performing the key input action with their own hands is sent from the imaging unit, the discrimination unit uses the trained model stored in the storage unit to determine which finger the user is using to press which key based on the content of the key input action contained in the image data, The display device according to claim 1, further comprising the above.

17. The display device according to claim 16, wherein the memory unit stores information on the correspondence between fingers and keys, which is predetermined for each finger of the user's left and right hands to press on the keyboard, and the discrimination unit uses the learned model and the finger-to-key correspondence information stored in the memory unit to determine which finger of the user is using to press which key.

18. The device further comprises a terminal configured separately from the aforementioned attachment, The display device according to claim 2, characterized in that the terminal is connected to the display unit wirelessly or by wire, and has all or some of the functions of the display control unit, the detection unit, and the operation determination unit.

19. The display device according to claim 1, characterized in that the attachment is worn over the user's eye.

20. The display device according to claim 19, characterized in that the attachment portion is attached to one or both of the user's eyes when it is attached to the user's eyes.

Images