Display device and its control method

Abstract

The present invention provides a highly convenient display device that allows users to easily perform support operations while holding it with both hands, even without having any physical operating components. [Solution] The display device includes a gripping part that the user holds, an imaging unit 101 that photographs the user's fingers holding the gripping part, a detection unit that detects the position of the fingers based on the image captured by the imaging unit 101, and a display control unit that displays a setting GUI for finger operation within the display area based on the detection result of the detection unit.

Description

Technical Field

[0001] The present disclosure relates to a display device and a control method thereof.

Background Art

[0002] There is a Mixed Reality (MR) technology that fuses the real space and the virtual space so that an experiencer can interact with virtual objects. In the MR technology, interaction is realized by synthesizing and presenting computer graphics (CG) representing virtual objects with respect to a real landscape, or by expressing the contact between a real object and a virtual object. In the MR technology, it is assumed that a virtual object displayed on a real landscape is operated by a gesture operation with one's own hand. In a gesture operation, operations such as moving and operating an object image created by CG can be performed without a controller.

[0003] In recent years, as a display device to which the MR technology is applied, a device that is worn on a user's head and displays an image in front of the user's eyes (Head Mount Display: HMD) is used. Some HMDs can be used not only by wearing them on the user's head but also by holding the handle part.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] When using a handheld HMD for gesture control, the shape of the HMD requires holding it with one hand, meaning that gestures must be performed with the other hand released. However, holding the HMD with only one hand reduces its stability, which presents a problem.

[0006] Patent Document 1 discloses a method for enabling functional operation using a touchpad in situations where the user's hand movements are restricted, such as when driving a car. For example, touchpads are positioned on the left and right sides of the steering wheel of a car, and by performing specific operations on these touchpads, predetermined functions can be executed. In the technology of Patent Document 1, although the range of motion of the hand is restricted, the touchpad is fixed in a position that is easy to operate relative to the hand position.

[0007] Patent Document 1 only discloses cases where a touchpad is actually provided in a location accessible to the user. However, it makes no mention of cases where there is no physical operating element such as a touchpad.

[0008] This disclosure has been made in view of the above-mentioned problems, and aims to provide a highly convenient display device that allows the user to easily perform instruction operations on an operating image while holding it with both hands, even without having a physical operating member. [Means for solving the problem]

[0009] The display device of this disclosure comprises a gripping unit, an imaging unit, a detection unit, and a display control unit. The gripping unit is the part of the display device that is gripped by the user. The imaging unit captures images of the fingers of the user's hand that are gripping the gripping unit. The detection unit detects the position of the fingers based on the image captured by the imaging unit. The display control unit displays an operation image for operation with the fingers within the display area based on the detection result of the detection unit. [Effects of the Invention]

[0010] According to this disclosure, a highly convenient display device can be realized that allows the user to easily perform instruction operations on an operating image while holding it with both hands, even without having a physical operating component. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the functions of the HMD unit according to the first embodiment. [Figure 2] This is a hardware configuration diagram of the HMD unit according to the first embodiment. [Figure 3] This is a schematic diagram illustrating an example in which a user performs gesture operations using their fingers while holding a handheld HMD in both hands according to the first embodiment. [Figure 4] This flowchart shows, step by step, how to determine the mode in which the handheld HMD in the first embodiment is operated using the fingers while holding the handle portion. [Figure 5] This is a schematic diagram showing the operable area within the display area. [Figure 6] This is a flowchart showing the operation of the finger gesture mode in the HMD unit according to the first embodiment, in step order. [Figure 7] This is a flowchart showing, in step order, the actions taken by the user to set the range of motion of their fingers in a handheld HMD unit according to the second embodiment. [Figure 8] This is a schematic diagram showing the settings GUI displayed on a handheld HMD unit according to the third embodiment. [Figure 9] This is a schematic diagram illustrating the field of view in a fourth embodiment where a camera for displaying video and a camera for tracking objects are provided separately. [Figure 10] The fourth embodiment is a schematic diagram illustrating an example of performing GUI operations in finger gesture mode when the field of view differs between the camera for video display and the camera for object tracking. [Figure 11]A flowchart showing the operation of the finger gesture mode in the HMD unit according to the fourth embodiment in step order. [Figure 12] A schematic diagram illustrating a horizontal viewing angle narrower than that of a general user and a horizontal viewing angle wider than that of a general user. [Figure 13] A block configuration diagram showing the functions of the HMD unit according to the fifth embodiment. [Figure 14] A flowchart showing the operation of the finger gesture mode in the HMD unit according to the fifth embodiment in step order. [Figure 15] A block configuration diagram showing the functions of the HMD unit according to the sixth embodiment. [Figure 16] A flowchart showing the operation of the finger gesture mode in the HMD unit according to the sixth embodiment in step order.

Modes for Carrying Out the Invention

[0012] -Basic Configuration of the Display Device in the Embodiments- Specifically, when disclosing the embodiments, the basic configuration of the display device in the embodiments will be described.

[0013] The display device of the present disclosure includes a photographing unit that photographs the fingers of the user's hand holding the holding unit, a detection unit that detects the position of the finger based on the photographed image of the photographing unit, and a display control unit that displays an operation image for operation with the finger within the display area based on the detection result of the detection unit. The operation image is, for example, a GUI for the user to operate with the finger to set a predetermined item. In this display device, it is assumed that the user sets a predetermined item while holding the holding unit. The position of the finger is detected from the image of the user's finger, and the operation image for operation with the finger is displayed as appropriate. In order for the user to operate the operation image with their own finger within the display area, the display control unit displays the operation image in an operable area (for example, the lower left and right ends of the display area) that the user's finger can reach within the display area. With this configuration, even without a physical operation member, the user can easily perform an instruction operation on the operation image while holding it with both hands.

[0014] Depending on the type of the operation image, the angle of view of the photographing unit, etc., if the operation image is displayed in an operable area such as the lower left and right ends of the display area, it may be difficult for the user to visually recognize. In the present disclosure, in consideration of this point, in a predetermined case, the operation image is enlarged and displayed not in an operable area such as the lower left and right ends of the display area but, for example, in the central area. In this case, it is preferable to display an image of the user's finger created by, for example, CG near the operation image corresponding to the operation image in the central area. By doing so, further improvement in the convenience of the display device is realized.

[0015] Furthermore, it may be possible to photograph the user's eyes, detect the user's gaze position in the display area based on the eye image, and display the operation image at the detected gaze position. Also in this case, it is preferable to display an image of the user's finger created by, for example, CG near the operation image corresponding to the operation image displayed at the gaze position. By doing so, further improvement in the convenience of the display device is realized.

[0016] -Specific Description of Embodiments- The embodiments of this disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not limit the invention as defined in the claims. While multiple features are described in the embodiments, not all of these features are essential, and the features may be combined in any way. Furthermore, in the drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0017] [First Embodiment] The first embodiment of this disclosure will be described below. In this embodiment, an HMD unit is disclosed that includes a handheld type HMD (Head Mount Display), which is an image display device worn on the user's head, as the display device. Figure 1 is a block diagram showing the functions of the HMD unit according to this embodiment. Figure 2 is a hardware configuration diagram of the HMD unit.

[0018] The CPU (Central Processing Unit) 200 controls the entire system by executing applications that operate the mixed reality system in the HMD 100. Each processing unit shown in Figure 1 is implemented by the CPU 200. The CPU 200 performs the processing of each flowchart described later by executing control programs stored in ROM 202, etc. However, some of these processing units may be implemented by a computer using hardware independent of the CPU 200.

[0019] The HMD unit of this embodiment includes an HMD 100 and an information processing device 103. The HMD100 is a head-mounted display device consisting of an image capture unit 101 and an image display unit 102. The HMD100 uses a video see-through system in which the image capture unit 101 is positioned at the user's viewpoint, and the live-action video captured by the image capture unit 101 is displayed on the image display unit 102. In the case of a video see-through HMD, it is common to operate the image capture unit 101 at the user's viewpoint and orientation. Although the HMD100 in this embodiment uses a video see-through system, it is not limited to this. For example, this embodiment can also be applied to a virtual reality HMD that does not display live-action video on the image display unit 102. In that case, the image capture unit 101 can be used not as a see-through camera, but as a camera to capture the user's fingers, which are the object to be measured.

[0020] The information processing device 103 creates data for display on the image display unit 102 using the image captured by the image capture unit 101. The information processing device 103 includes a detection unit that detects the position of the user's fingers based on the image captured by the image capture unit 101, and a display control unit that displays an operation image for the user to operate with their fingers within the display area (display screen) based on the detection result of the detection unit. The operation image is an image for the user to set predetermined items, for example, a GUI (Graphical User Interface). Hereinafter, this GUI will be referred to as the setting GUI. The detection unit includes a joint point extraction unit 105, a three-dimensional position calculation unit 106, and a position and orientation calculation unit 107. The display control unit includes an image drawing unit 108 and a processing calculation unit 109. The information processing device 103 may be a separate piece of hardware as shown in Figure 1, or it may be included in the HMD 100.

[0021] The image capture unit 101 is fixed to the housing of the HMD 100 and captures images of the real world at a predetermined frame rate (e.g., 30 frames / second). Images captured by the image capture unit 101 are stored in the captured image storage unit 104. The captured image storage unit 104 stores the images acquired from the image capture unit 101 in the RAM 201. The image capture unit 101 consists of a pair of real cameras, with a real camera for the left eye and a real camera for the right eye positioned close to both eyes of the HMD wearer. Hereafter, the pair of left and right images obtained by the image capture unit 101 will be referred to as stereo camera images. It is also possible to separate the camera used for video display (background video) from the camera used for object tracking. In this embodiment, we will describe the case where the image capture unit 101 consists of two cameras, a real camera for the right eye and a real camera for the left eye.

[0022] The joint point extraction unit 105 extracts the joint points of the user's fingers from each of the captured images stored by the captured image storage unit 104 using image recognition. The three-dimensional position calculation unit 106 calculates the three-dimensional position of the joint points extracted by the joint point extraction unit 105. The x and y coordinates of the joint points are obtained using the lower left corner of the VRAM (GPU memory) area of ​​the image captured by the image capture unit 101 of the HMD 100 as the origin coordinate (0,0). The z coordinate in the depth direction is obtained from the stereo camera image. The reference coordinate can be freely determined, not just the lower left corner.

[0023] To correctly display the depth of the detected hand, one method involves extracting the hand region from images of the fingers captured by a stereo camera and calculating the depth value of the fingers relative to the stereo camera. This method uses triangulation to determine the depth value for all corresponding points in the stereo image of the extracted finger contour lines (see, for example, Patent Document 2). Patent Document 2 utilizes a stereo camera mounted on a video see-through type HMD, which is a display device for presenting augmented reality.

[0024] In this embodiment, the three-dimensional position calculation unit 106 selects an arbitrary point from the joint points extracted by the joint point extraction unit 105 and calculates the three-dimensional position of that point. The point selected here is stored as a reference position during the calculation process, and the three-dimensional position information is periodically stored in the RAM 201 in accordance with the movement of the fingers.

[0025] The position and orientation calculation unit 107 calculates the position and orientation of each captured image stored by the captured image storage unit 104. Position and orientation can be calculated using deep learning or by using existing publicly available libraries. The processing calculation unit 109 determines the movement of the hand based on the change in coordinate values ​​calculated by the three-dimensional position calculation unit 106. Generally, coordinates acquired by sensors have errors, so a certain error rate is allowed, for example, determining that coordinates are the same up to an error rate of 5 percent. The error rate can be freely determined. Based on the processing result of the processing calculation unit 109, the image drawing unit 108 draws an image to be displayed on the image display unit 102 of the HMD 100.

[0026] Figure 3 is a schematic diagram showing an example of gesture operation using fingers while the user holds the handheld HMD100 in both hands in this embodiment. Since the index finger can be moved relatively freely while holding the handheld HMD100, this embodiment shows an example of operation using the index finger.

[0027] Figure 3(a) is a schematic diagram showing the user operating the settings GUI menu with their left index finger. The user holds the handle portion of the handheld HMD 100 and moves their index finger to operate the settings GUI 301 menu displayed on the image display unit 102. Either the left or right hand can be used, and the processing unit 109 recognizes the user's finger within the display area of ​​the image display unit 102 of the HMD 100 and displays the settings GUI 301 on the side where the user's finger is recognized, allowing the user to operate it with their finger. Figure 3(b) is a schematic diagram showing the user operating the settings GUI 301 menu with their right index finger.

[0028] Figure 3(c) is a schematic diagram showing the operation of the setting GUI 302 by moving the scale with the index finger of the left hand. Since the operation is performed near the finger, the setting GUI is displayed at the lower left (right) edge of the display area, which may be difficult for some users to see. Sometimes, detailed values ​​are set by moving the scale. In such cases, it may be difficult to operate while looking at the setting GUI displayed at the lower left (right) edge of the display area. Therefore, when operating menus that set fine values ​​by moving the scale, the setting GUI 303 may be enlarged and displayed in a more visible area in the center of the screen, either together with or instead of the setting GUI 302, as shown in Figure 3(c). This makes it easier to perform detailed operations on the setting GUI. In this case, the setting GUI may only display the position indicating the current setting value, as shown in Figure 3(c), but considering the ease of understanding for the user, it may also be possible to create and display a CG image of the user's fingers pointing to the vicinity of the setting value.

[0029] If CG other than the settings GUI is displayed in the display area of ​​the image display unit 102, the background may be grayed out, for example, to make the settings GUI easier to see. Also, although Figure 3(c) shows a speech bubble design as an example, it is not limited to this.

[0030] Figure 4 is a flowchart showing, in step order, how to determine the mode in which the handheld HMD100 in this embodiment is operated using the fingers while holding the handle (hereinafter referred to as the finger gesture mode).

[0031] First, in S400, the processing unit 109 sets an area within the display area of ​​the HMD 100 where the user can operate the setting GUI with their fingers while holding the handle (hereinafter referred to as the operable area), which is used to determine whether to transition to the finger gesture mode. When a finger is displayed in this operable area, the system operates in finger gesture mode. In this embodiment, since the operation is performed while holding the handle of the HMD 100, as shown in Figure 5, the operable areas 501 and 502, indicated by dashed lines in the figure and located at the lower left and right ends of the display area, become the areas for finger gesture mode. In this embodiment, the operable areas 501 and 502 are pre-set as areas corresponding to the fingers of a typical user. After the processing unit 109 sets the operable areas 501 and 502, the image drawing unit 108 may display the operable areas 501 and 502 within the display area of ​​the image display unit 102, but the operable areas 501 and 502 may be not displayed to improve the visibility of the display area. Since the range in which fingers can be extended while gripping the handle of the HMD100 can be predicted, the processing unit 109 sets the operable areas 501 and 502 based on data on typical hand movements and finger lengths.

[0032] In S401, the detection unit determines whether or not a finger is visible within the designated operable areas 501 and 502. If it is determined that a finger is displayed within the operable areas 501 and 502, in S402, the detection unit calculates the distance between the finger displayed within the operable areas 501 and 502 and the HMD 100. Since the operation is performed while gripping the handle portion of the HMD 100, the general distance from the user's finger to the HMD 100 can be determined in advance. Therefore, in order to decide whether or not to operate in finger gesture mode, in S403, the processing calculation unit 109 compares the distance measured in S402 with a pre-set general distance to determine whether or not to operate in finger gesture mode.

[0033] In S404, if the distance from the HMD100 to the user's finger is close to a predetermined general distance, the finger gesture mode is started (S404). The acceptable difference between the distance measured in S402 and the general distance can be freely set. If the distance from the HMD100 to the finger is far from the predetermined distance, it is highly likely that it is not the user's finger, and therefore the finger gesture mode is not started.

[0034] Figure 6 is a flowchart showing the operation of the finger gesture mode in the HMD unit of this embodiment in step order. First, in S600, the processing unit 109 determines whether or not the user is in finger gesture mode. If it is determined in S600 that the user is in finger gesture mode, the detection unit checks the position of the user's fingers in S601. If it is determined in S600 that the user is not in finger gesture mode, nothing is done. The position of the user's fingers determines the display position of the setting GUI in the display area of ​​the HMD 100.

[0035] In S602, the processing unit 109 determines the type of setting GUI to be displayed on the image display unit 102. Setting GUIs can be broadly classified into the following two forms: a first form in which items are listed, like the first level of a setting menu, and the user selects from there (for example, Figures 3(a) and (b)); and a second form in which detailed numerical values ​​are set, such as by displaying a setting bar (Figure 3(c)).

[0036] In the HMD unit of this embodiment, for example, if there are multiple setting items in the setting GUI, it is stipulated that the first form, a setting menu of the type that can be selected with buttons, is displayed first. Subsequently, depending on the setting item selected, the menu is divided into the first form, the second form, and so on at the second level and below. The way the setting GUI is represented differs depending on the setting item. Therefore, information on how the setting GUI is represented for each setting item is stored in advance, for example, in the ROM 202, such as the first form (button type) when selecting from several setting items, and the second form (bar type) when setting values ​​in fine units from multiple setting values. The processing unit 109 uses this information to determine the type of setting GUI to display. For example, the type of setting GUI may be determined for each individual setting, such as a bar type for brightness settings and a button type for high, medium, and low settings, or the form to be determined based on the characteristics of the setting, and the determination is made according to that rule.

[0037] If it is determined in S602 that the setting GUI is in its first form, the following occurs (S603). Based on the processing result of the processing calculation unit 109, the image drawing unit 108 displays a predetermined setting GUI in its first form near the user's finger, as confirmed in S601, as shown in Figures 3(a) and (b) (S604). The position where the setting GUI is displayed is, for example, the operable area 501 or operable area 502 shown in Figure 5.

[0038] If it is determined in S602 that the setting GUI is in the second form, the following occurs (S603). Based on the processing result of the processing calculation unit 109, the image drawing unit 108 displays the predetermined setting GUI in the second form enlarged in the center of the display area of ​​the image display unit 102 of the HMD 100, for example as shown in Figure 3(c) (S605). For example, if there is only one setting item, it is conceivable to display the second form without going through the first form.

[0039] In this embodiment, the finger gesture mode was determined by whether or not the user's finger was displayed in a predetermined area of ​​the display area on the image display unit 102 of the HMD 100, but it is not limited to this. For example, the HMD 100 could be equipped with a voice microphone, allowing the user to initiate the finger gesture mode using their voice, and the finger gesture mode could be enabled or disabled using a setting menu displayed on the image display unit 102. Furthermore, if the HMD 100 is used as a handheld type, it is conceivable that it could be configured to operate in finger gesture mode when, for example, the gripping part (handle) is attached to a predetermined part of the HMD 100.

[0040] As described above, this embodiment provides a highly convenient handheld HMD unit that allows the user to easily perform instruction operations on the setting GUI while holding it with both hands, even without having any physical operating members.

[0041] [Second Embodiment] A second embodiment of this disclosure will be described below. In this embodiment, a handheld HMD unit will be described, similar to the first embodiment, but a detailed explanation of the same configuration as in the first embodiment will be omitted.

[0042] In the first embodiment, the range of the user's finger position for determining the finger gesture mode was pre-set to an area where the fingers are likely to be displayed, based on the characteristics of the HMD100. However, due to differences in the shape of the HMD100 and the length of the user's fingers, the range that the user's fingers can reach may differ. In the second embodiment, a configuration is disclosed in which the operable area is pre-set according to each user. With this configuration, the finger gesture mode can be determined with high accuracy.

[0043] Figure 7 is a flowchart showing, step by step, the actions a user takes to set the range of motion of their fingers in a handheld HMD unit according to this embodiment. First, in the S700, the image display unit 102 displays a screen for registering the user's finger range of motion from the settings menu. Until it operates in finger gesture mode, the settings menu can be accessed by holding the gripping part of the HMD100 with one hand and performing gesture operations with the other hand. It is also conceivable that the settings menu can be displayed via remote operation.

[0044] In S701, the user moves their fingers up, down, left, and right while gripping the gripping part of the HMD100, and the detection unit detects the range of motion of the user's fingers. The processing unit 109 acquires information on the range of motion of the user's fingers. Generally, the index finger is considered to be the finger that can be freely moved while gripping the gripping part of the HMD100, but any finger may be used, not just the index finger. In S702, the information on the range of motion of the user's fingers acquired in S701 is recorded in the ROM 202. The processing unit 109 determines the operable area corresponding to the user's fingers based on the information on the range of motion of the user's fingers recorded in S702. After the processing unit 109 sets the operable area, the image drawing unit 108 may display the operable area within the display area of ​​the image display unit 102, but the operable area may be not displayed to improve the visibility of the display area.

[0045] In this embodiment, a region is set to determine a finger gesture mode suitable for the user, according to the user's finger range of motion. This enables highly accurate determination of the finger gesture mode.

[0046] As described above, this embodiment provides a highly convenient handheld HMD unit that allows the user to easily perform instruction operations on the setting GUI while holding it with both hands, even without having any physical operating members.

[0047] [Third Embodiment] A third embodiment of this disclosure will be described below. In this embodiment, a handheld HMD unit will be described, similar to the first embodiment, but a detailed explanation of the same configuration as in the first embodiment will be omitted.

[0048] When a user of an HMD unit operates the on-screen settings GUI with their fingers, they must operate it while gripping the HMD's gripping part, which limits the range of motion of the user's fingers. In view of this, this embodiment incorporates a design that makes the settings GUI easier to use in finger gesture mode.

[0049] Figure 8 is a schematic diagram showing the settings GUI displayed on the handheld HMD unit according to this embodiment. Figure 8(a) illustrates a component of the basic settings GUI in finger gesture mode. When operating the settings GUI while holding the gripping part of the handheld HMD100 with both hands, a natural movement is to push from the outside to the inside within the display area with the fingers 801. Therefore, in this embodiment, a component 802 is used that has a shape that makes it easy to see that the action is being pushed inward, for example, a component that looks like a horizontally placed rod in three dimensions.

[0050] When the user holds the gripping part of the handheld HMD100 with both hands and moves their fingers, the range of motion of the fingers becomes limited. Therefore, to enable operation even in a narrow area during use, a setting GUI 803 may be displayed in which each component 802 corresponding to multiple selection items rotates as shown by the arrows, for example, as shown in Figure 8(b). Here, the setting items of each component 802, such as menus 1, 2, etc., include camera-related items such as brightness, diopter adjustment, exposure, and white balance, as well as HMD-related items such as main unit settings.

[0051] Thus, in this embodiment, by using a settings GUI that is adapted to the manner in which finger gestures are performed within the display area, users can perform natural GUI operations without feeling any discomfort.

[0052] As described above, this embodiment provides a highly convenient handheld HMD unit that allows the user to easily perform instruction operations on the setting GUI while holding it with both hands, even without having any physical operating members.

[0053] [Fourth Embodiment] A fourth embodiment of this disclosure will be described below. In this embodiment, a handheld HMD unit will be described, similar to the first embodiment, but a detailed explanation of the same configuration as in the first embodiment will be omitted.

[0054] In the first embodiment, the case in which the image capture unit 101 of the HMD 100 is composed of a pair of real cameras was described. A real camera for the left eye and a real camera for the right eye were placed close to both eyes of the user of the HMD 100, and the pair of left and right images were referred to as stereo camera images. These stereo camera images were used for calculating the depth value of the hand and for the background image. As mentioned above, it is also possible to separate the camera used for image display (background image) from the camera used for object tracking. In this embodiment, the image capture unit 101 is divided into a first capture unit and a second capture unit, with the first capture unit being the camera used for object tracking and the second capture unit being the camera used for image display.

[0055] The camera used for displaying video is used by the user while viewing the video on the HMD100's image display unit 102, so it needs to use high-resolution video to improve the user experience. High-resolution video results in a huge amount of data and takes a long time to process. On the other hand, video used for tracking objects does not need to be high resolution as it is used to determine the position of objects, thus reducing the overall amount of data.

[0056] Using separate cameras for video display and object tracking allows for tracking with a wider field of view. However, because two types of cameras are used, there may be situations where the user's finger is recognized by only one type of camera.

[0057] Figure 9 is a schematic diagram showing the field of view in this embodiment when the camera for displaying the image and the camera used for tracking objects are provided separately. In Figure 9, (X) represents the field of view captured by the camera used for tracking objects, and (Y) represents the field of view captured by the camera for displaying the image. Here, considering that a wider field of view (X) is secured to perform tracking with a wide field of view, the field of view (Y) is narrower than the field of view (X). Therefore, if the user's finger is in a position that is only visible to the camera used for tracking objects, the user's finger may be recognized, but it may not be visible in the display area of ​​the image display unit 102 of the HMD 100.

[0058] Therefore, in this embodiment, if the user's finger 901 is only visible within the tracking area of ​​the camera used for object tracking, as shown in Figure 9, the display area of ​​the video display camera will be used to create and display the user's finger 902 using computer graphics to correspond to finger 901. The finger movements in finger gesture mode are the same as in the first embodiment, except that a computer-generated finger 902 is displayed instead of the actual user's finger 901.

[0059] Figure 10 is a schematic diagram illustrating an example of GUI operation in finger gesture mode when the field of view differs between the camera for video display and the camera for object tracking in this embodiment. Figure 10(a) shows an example of menu operation in the settings GUI, and Figure 10(b) shows an example of detailed setting operation in the settings GUI. Similar to Figure 9, (X) represents the field of view captured by the camera used for object tracking, and (Y) represents the field of view captured by the camera for video display.

[0060] In Figure 10(a), the display of the setting GUI 1001 for menu operation in the first form is the same as in Figure 3(a), but the user's finger 1002 is not within the field of view (Y) of the camera for video display. Therefore, a finger 1003 corresponding to finger 1002 is created using computer graphics and displayed in the display area of ​​the camera for video display.

[0061] In Figure 10(b), the display of the setting GUI 1004 for detailed setting operations in the second form is the same as in Figure 3(b), but the user's finger 1002 is not within the field of view (Y) of the camera for video display. Therefore, a finger 1003 corresponding to finger 1002 is created using computer graphics and displayed in the display area. Furthermore, when operating menus that require setting fine numerical values ​​such as moving memory, the setting GUI 1005 is enlarged and displayed in a clearly visible area in the center of the display area. This makes detailed operation of the setting GUI easier. Note that in Figure 10(b), the setting GUI 1004 and the user's finger 1002 may be displayed without being shown in the field of view (Y) of the camera for video display, and instead the enlarged setting GUI 1005 and the CG-generated finger 1006 may be displayed.

[0062] Since the CG displayed in the background is not expected to be manipulated while the settings GUI is being operated, the settings GUI 1005 may be grayed out to make it stand out, as shown in Figure 10(b). Alternatively, to make the setting positions of the scales in the settings GUI 1005 easier to understand, a CG representation of the user's finger 1006 may be created and placed.

[0063] Figure 11 is a flowchart showing the operation of the finger gesture mode in the HMD unit of this embodiment in step order. First, in S1100, the processing unit 109 determines whether or not the system is in finger gesture mode. The determination of finger gesture mode is as shown in Figure 4 in the first embodiment. In this embodiment, the operable area for determining whether or not the system is in finger gesture mode is set within the field of view of the camera used for object tracking. The operable area set at this time may overlap with the field of view of the camera used for video display and the camera used for object tracking, or it may be limited to the field of view of the camera used for object tracking.

[0064] If it is determined in S1100 that the system is in finger gesture mode, in S1101 the detection unit detects the area where the user's finger is recognized. In S1102, the processing unit 109 determines whether the user's finger is recognized within the display area of ​​the camera's field of view. If it is determined that the finger is recognized within the display area of ​​the camera, in S1103 the operation of normal finger gesture mode, i.e., the same operation as S601 in Figure 6 of the first embodiment, is performed. Subsequently, in S1104, the same operation as S602 to S605 in Figure 6 is performed. That is, the processing unit 109 determines the display position of the setting GUI according to the type of setting GUI (first form or second form) to be displayed on the image display unit 102. Based on the processing result of the processing unit 109, the image drawing unit 108 displays the setting GUI in the area where the user's finger is located or in the center of the display area.

[0065] In S1102, if it is determined that the user's finger is within the tracking area of ​​the camera's field of view used for tracking objects, the following occurs: Based on the processing result of the processing unit 109, the image drawing unit 108 displays the setting GUI on the side of the display area of ​​the video camera where the user's finger is displayed (S1105). At this time, based on the processing result of the processing unit 109, the image drawing unit 108 displays the user's finger created with computer graphics near the setting GUI displayed in S1105 (S1106). The user operates the GUI by moving their own finger using the computer graphics finger as a guide. In S1107, the processing unit 109 determines whether the user's finger has moved within the tracking area of ​​the camera's field of view used for tracking objects. If it is determined that the user's finger has moved, the image drawing unit 108 moves the computer graphics finger to follow the user's finger movement and displays it in the display area (S1108).

[0066] In this embodiment, a separate camera is provided for displaying video and for tracking objects. When the user's finger is only visible in the tracking area of ​​the latter camera, the user's finger, created using GUI operation and computer graphics, is appropriately displayed in the display area of ​​the former camera. This reduces the overall amount of video data while allowing the user to easily operate the settings GUI with their own finger without any inconvenience.

[0067] As described above, this embodiment provides a highly convenient handheld HMD unit that allows the user to easily perform instruction operations on the setting GUI while holding it with both hands, even without having any physical operating members.

[0068] [Fifth Embodiment] The fifth embodiment of this disclosure will now be described. In this embodiment, a handheld HMD unit will be described, similar to the first embodiment, but a detailed explanation of the same configuration as in the first embodiment will be omitted.

[0069] In HMDs, the image capture unit used may have a horizontal field of view wider than the typical user's field of view. In this case, the lower left and right edges of the display area become areas that are difficult for the user to see. In this embodiment, taking this into consideration, as shown in Figure 12, an HMD unit is disclosed having an image capture unit that can change between a horizontal field of view narrower than the typical user's field of view, for example, 100 degrees, and a horizontal field of view wider than the typical user's field of view, for example, 120 degrees.

[0070] Figure 13 is a block diagram showing the functions of the HMD unit according to this embodiment. In addition to the configuration shown in Figure 1 of the first embodiment, the HMD unit of this embodiment includes a field of view setting unit 1301 in the HMD 100 and a field of view detection unit 1302 in the information processing device 103. The field of view setting unit 1301 sets the horizontal field of view in the image capture unit 101. For example, the field of view setting unit 1301 can change and set at least two fields of view: a first field of view A, which is narrower than the commonly used field of view, and a second field of view B, which is wider than the commonly used field of view. The field of view detection unit 1302 detects whether the field of view in the image capture unit 101 set by the field of view setting unit 1301 is field of view A or field of view B. Based on the detection result from the field of view detection unit 1302, the processing calculation unit 109 determines the display position of the setting GUI displayed in the display area of ​​the image display unit 102.

[0071] Figure 14 is a flowchart showing the operation of the finger gesture mode in the HMD unit of this embodiment in step order. First, in S1400, the processing unit 109 determines whether or not the system is in finger gesture mode. The determination of finger gesture mode is as shown in Figure 4 in the first embodiment.

[0072] If it is determined in S1400 that the device is in finger gesture mode, the field of view detection unit 1302 detects whether the field of view of the image capture unit 101, as set in the field of view setting unit 1301, is field of view A (S1401). If it is determined in S1400 that the device is not in finger gesture mode, nothing is done.

[0073] If S1401 determines that the field of view A is narrower than the typical user's field of view, it is assumed that a normal user can easily see the lower left and right edges of the display area. In this case, S1402 performs the operation of the normal finger gesture mode, i.e., the same operation as S601 in Figure 6 in the first embodiment. Subsequently, S1403 performs the same operation as S602 to S605 in Figure 6. That is, the processing calculation unit 109 determines the display position of the setting GUI according to the type of setting GUI (first form or second form) to be displayed on the image display unit 102. Based on the processing result of the processing calculation unit 109, the image drawing unit 108 displays the setting GUI in the area where the user's finger is located or in the center of the display area.

[0074] If S1401 determines that the field of view B is wider than the typical user's viewing angle, it is considered that a normal user will have difficulty seeing the lower left and right edges of the display area. In this case, in S1404, the processing unit 109 determines the type of setting GUI to be displayed on the image display unit 102.

[0075] In S1404, if it is determined that the setting GUI to be displayed is the first form, S1405 is performed as follows: Based on the processing result of the processing calculation unit 109, the image drawing unit 108 displays the setting GUI of the first form for menu operation in the center of the display area of ​​the image display unit 102. In this way, when the field of view is B, even if the setting GUI to be displayed is the first form, by displaying it in the center of the display area rather than at the lower left or right edges of the display area which are difficult for the user to see, the user can easily see the setting GUI and perform the appropriate operation.

[0076] If, in S1404, it is determined that the setting GUI to be displayed is the second form, then S1406 is performed as follows: Based on the processing result of the processing calculation unit 109, the image drawing unit 108 displays the setting GUI of the second form for performing menu operations, for example, enlarged, in the center of the display area of ​​the image display unit 102.

[0077] In the subsequent S1407, the image drawing unit 108 displays the user's fingers, created with computer graphics (CG), near the setting GUI displayed on the screen in S1403, S1405, or S1406, based on the processing result of the processing unit 109. The user operates the GUI by moving their own fingers using the CG-created fingers as a guide. In S1408, the processing unit 109 determines whether or not the user's fingers have moved within the display area. If it is determined that the user's fingers have moved, the image drawing unit 108 moves the CG fingers to follow the movement of the user's fingers and displays them in the display area, based on the processing result of the processing unit 109 (S1409).

[0078] As described above, this embodiment provides a highly convenient handheld HMD unit that allows the user to easily perform instruction operations on the setting GUI while holding it with both hands, even without having any physical operating members.

[0079] [Sixth Embodiment] A sixth embodiment of this disclosure will be described below. In this embodiment, a handheld HMD unit will be described, similar to the first embodiment, but a detailed explanation of the same configuration as in the first embodiment will be omitted.

[0080] Figure 15 is a block diagram showing the functions of the HMD unit according to this embodiment. In addition to the configuration shown in Figure 1 of the first embodiment, the HMD unit of this embodiment includes a gaze image capture unit 1501 in the HMD 100 and a gaze position detection unit 1502 in the information processing device 103. The gaze image capture unit 1501 captures the user's eyes, including their pupils. The gaze position detection unit 1502 detects the user's gaze position within the display area of ​​the image display unit 102 from the image of the user's eyes, including their pupils, captured by the gaze image capture unit 1501. The processing calculation unit 109 displays a setting GUI at the user's gaze position within the display area based on the detection result in the gaze position detection unit 1502. The setting GUI moves within the display area as the user's gaze position moves.

[0081] Figure 16 is a flowchart showing the operation of the finger gesture mode in the HMD unit of this embodiment in step order. First, in S1600, the processing unit 109 determines whether or not the system is in finger gesture mode. The determination of finger gesture mode is as shown in Figure 4 in the first embodiment.

[0082] If it is determined in S1600 that the user is in finger gesture mode, the detection unit checks the position of the user's fingers in S1601. If it is determined in S1600 that the user is not in finger gesture mode, nothing is done.

[0083] In S1602, the gaze image capture unit 1501 captures the user's eye area, including the pupil. In S1603, the gaze position detection unit 1502 detects the user's gaze position within the display area of ​​the image display unit 102 from the image of the user's eye area, including the pupil, captured by the gaze image capture unit 1501. In S1604, the processing calculation unit 109 determines the type of setting GUI (first form or second form) to be displayed on the image display unit 102.

[0084] If it is determined in S1604 that the setting GUI is in its first form (S1605), the following occurs: In S1606, the image drawing unit 108 displays a predetermined setting GUI in its first form at the gaze position detected in S1603, based on the processing result of the processing calculation unit 109.

[0085] If it is determined in S1604 that the setting GUI is in its second form (S1605), the following occurs: In S1607, the image drawing unit 108 displays the predetermined setting GUI, which is in its second form, at the gaze position detected in S1603, for example, by enlarging it, based on the processing result of the processing calculation unit 109.

[0086] In the subsequent S1608, the image drawing unit 108 displays the user's fingers, created with computer graphics (CG), near the setting GUI displayed in the display area in S1606 or S1607, based on the processing result of the processing unit 109. The user operates the GUI by moving their own fingers using the CG-created fingers as a guide. In S1609, the processing unit 109 determines whether or not the user's fingers have moved within the display area. If it is determined that the user's fingers have moved, the image drawing unit 108 moves the CG fingers to follow the user's finger movements and displays them in the display area, based on the processing result of the processing unit 109 (S1610). At this time, the CG fingers move within the display area in accordance with the movement of the user's gaze position along with the setting GUI, and their relative position to the setting GUI moves in accordance with the user's finger movements.

[0087] In this embodiment, the settings GUI is displayed in accordance with the user's line of sight within the display area of ​​the image display unit 102. Since the settings GUI is always displayed to follow the user's line of sight, operations using the settings GUI can be performed easily and user-friendly.

[0088] As described above, this embodiment provides a highly convenient handheld HMD unit that allows the user to easily perform instruction operations on the setting GUI while holding it with both hands, even without having any physical operating members.

[0089] While preferred embodiments have been described above, this disclosure is not limited to these embodiments, and various modifications and changes are possible within the scope of its essence.

[0090] The disclosure of various embodiments includes the following configurations and methods. (Composition 1) The gripping part that the user holds, A camera unit for photographing the fingers of the user's hand that are gripping the gripping part, A detection unit detects the position of the finger based on the image captured by the aforementioned imaging unit, A display control unit that displays an operation image for operation with the finger within the display area based on the detection result of the detection unit, Having, Display device. (Configuration 2) The display control unit displays the operation image in the operable area accessible by the fingers while the user is holding the gripping part. The display device described in Configuration 1. (Composition 3) The display control unit determines the operation image based on the information regarding the range of motion of the user's finger detected by the detection unit. The display device described in Configuration 2. (Composition 4) The aforementioned operation image is a GUI for the user to operate with their finger to set predetermined items. A display device as described in any one of configurations 1 to 3. (Composition 5) The display control unit enlarges and displays the operation image in the center of the display area according to the settings in the operation image. A display device as described in any one of configurations 1 to 4. (Composition 6) The aforementioned operating image shows a component whose shape is operated by pressing it with the finger from the outside to the inside within the display area. A display device as described in any one of configurations 1 to 4. (Composition 7) The aforementioned operating image is designed so that multiple components are rotated by the user's fingers during operation. The display device described in configuration 6. (Composition 8) The aforementioned imaging unit is The first imaging unit for tracking, A second shooting unit for displaying images, It has, The display control unit, when the field of view of the second imaging unit is narrower than the field of view of the first imaging unit, and when the user's finger is recognized by the first imaging unit but not by the second imaging unit, creates an image of the finger and displays it near the operation image within the display area of ​​the second imaging unit. A display device as described in any one of configurations 1 to 7. (Composition 9) The imaging unit further includes a field of view setting unit that sets the horizontal field of view in the imaging unit to either a first field of view or a second field of view that is wider than the first field of view. The display control unit, when the horizontal field of view is set to the second field of view, displays the operation image in the center of the display area. A display device as described in any one of configurations 1 to 4. (Composition 10) The display control unit creates an image of the user's finger and displays it near the operation image. The display device described in configuration 9. (Composition 11) The aforementioned eye-tracking image capture unit captures the user's eye area, A gaze position detection unit detects the user's gaze position in the display area based on the eye image captured by the gaze image capture unit, It also has, The display control unit displays the operation image at the gaze position detected by the gaze position detection unit. A display device as described in any one of configurations 1 to 4. (Composition 12) The display control unit creates an image of the user's finger and displays it near the operation image. The display device described in configuration 11. (Composition 13) Equipped with a handheld HMD, A display device as described in any one of configurations 1 to 12. (Method 1) The steps include: photographing the fingers of the user's hand while the user is holding the gripping part of the display device; The steps include detecting the position of the finger based on the captured image of the finger, The steps include: displaying an operation image for operation with the finger within the display area based on the detection result of the finger position; Having, A method for controlling a display device. [Explanation of symbols]

[0091] 100 HMD 101 Image Acquisition Department 102 Image display section 103 Information Processing Device 104 Image acquisition unit 105 Articular point extraction area 106 3D position calculation section 107 Position and Attitude Measurement Unit 108 Image rendering section 109 Processing Unit 200 CPU 201 RAM 202 ROM 1301 Angle of View Setting Section 1302 Field of View Detection Unit 1501 Eye-tracking imaging unit 1502 Eye-line position detection unit

Claims

1. The gripping part that the user holds, A camera unit for photographing the fingers of the user's hand that are gripping the gripping part, A detection unit detects the position of the finger based on the image captured by the aforementioned imaging unit, A display control unit that displays an operation image for operation with the finger within the display area based on the detection result of the detection unit, Having, Display device.

2. The display control unit displays the operation image in the operable area accessible by the fingers while the user is holding the gripping part. The display device according to claim 1.

3. The display control unit determines the operation image based on the information regarding the range of motion of the user's finger detected by the detection unit. The display device according to claim 2.

4. The aforementioned operation image is a GUI for the user to operate with their finger to set predetermined items. The display device according to claim 1.

5. The display control unit enlarges and displays the operation image in the center of the display area according to the settings in the operation image. The display device according to claim 1.

6. The aforementioned operating image shows a component whose shape is operated by pressing it with the finger from the outside to the inside within the display area. The display device according to claim 1.

7. The aforementioned operating image is designed so that multiple components are rotated by the user's fingers during operation. The display device according to claim 6.

8. The aforementioned imaging unit is The first shooting unit for tracking, A second shooting unit for displaying images, It has, The display control unit, when the field of view of the second imaging unit is narrower than the field of view of the first imaging unit, and when the user's finger is recognized by the first imaging unit but not by the second imaging unit, creates an image of the finger and displays it near the operation image within the display area of ​​the second imaging unit. The display device according to claim 1.

9. The imaging unit further includes a field of view setting unit that sets the horizontal field of view in the imaging unit to either a first field of view or a second field of view that is wider than the first field of view. The display control unit, when the horizontal field of view is set to the second field of view, displays the operation image in the center of the display area. The display device according to claim 1.

10. The display control unit creates an image of the user's finger and displays it near the operation image. The display device according to claim 9.

11. The aforementioned eye-tracking image capture unit captures the user's eye area, A gaze position detection unit detects the user's gaze position in the display area based on the eye image captured by the gaze image capture unit, It also has, The display control unit displays the operation image at the gaze position detected by the gaze position detection unit. The display device according to claim 1.

12. The display control unit creates an image of the user's finger and displays it near the operation image. The display device according to claim 11.

13. Equipped with a handheld HMD, The display device according to claim 1.

14. The steps include: photographing the fingers of the user's hand while the user is holding the gripping part of the display device; The steps include detecting the position of the finger based on the captured image of the finger, The steps include: displaying an operation image for operation with the finger within the display area based on the detection result of the finger position; Having, A method for controlling a display device.

Images