HMD control device and HMD control method

Abstract

To provide a technology that enables simple and secure authentication of the wearer's hand from among the hands included in the HMD's field of view. [Solution] The HMD's control device gives the wearer an action instruction prompting them to perform an action using their hand or fingers, and when it determines that the hand detected from the captured image is performing an action in accordance with the action instruction, it authenticates that hand as the wearer's hand.

Description

Technical Field

[0001] The present invention relates to a technique for authenticating the hand of an HMD wearer included within the field of view of the HMD.

Background Art

[0002] There is a Mixed Reality (MR) technology that fuses the real space and the virtual space to enable an experiencer to interact with virtual objects. In MR technology, interaction is realized by synthesizing and presenting computer graphics (CG) representing virtual objects with respect to a real landscape, or by expressing contact between a real object and a virtual object.

[0003] In MR technology, it is assumed that gesture operations are performed with one's own hand to move virtual objects within a real landscape. In gesture operations, movement, operation, etc. of CG objects can be performed without a controller. However, when there are multiple people other than the experiencer in the same space, it may not be possible to distinguish one's own hand from others' hands, and there is a possibility that the experiencer's HMD (Head Mounted Display) may be unintentionally operated by a gesture operation of another person's hand.

[0004] Techniques capable of preventing HMD operation by others are disclosed in Patent Documents 1 and 2. Patent Document 1 discloses a method of authenticating an HMD wearer by operating the HMD based on a determination pattern input by the HMD wearer. Patent Document 2 discloses a method of recognizing the hand of an HMD wearer based on the relationship between the position of a hand detected from an image captured by the HMD and the position of a sensor device worn on the hand by the HMD wearer.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] The prior art disclosed in Patent Document 1 mentioned above can authenticate that the HMD wearer is an authorized user of the HMD. However, since this technology does not grant control of the HMD to the hand used for authentication, it is possible for someone else to operate the HMD after authentication.

[0007] Furthermore, the prior art disclosed in Patent Document 2 allows for the authentication of the HMD wearer by having the wearer attach a sensor device. However, sensor devices that accurately record and transmit the device position relative to the HMD are expensive and must be carried together with the HMD, making them difficult for users to use.

[0008] This invention has been made in view of the above circumstances, and its purpose is to provide a technology that enables the simple and secure authentication of the wearer's own hand from among the hands included in the field of view of the HMD. [Means for solving the problem]

[0009] One aspect of the present invention is an HMD (Head Mount) comprising an imaging means configured to capture the wearer's field of view and a display means configured to show the image to the wearer. The control device for a Display (d) is configured to give an action instruction to the wearer prompting them to perform an action using their hand or finger, and to authenticate the hand as the wearer's hand when it is determined from the captured image obtained by the imaging means that the hand is performing an action in accordance with the action instruction. [Effects of the Invention]

[0010] According to the present invention, it is possible to easily and securely authenticate the hand of the HMD wearer from among the hands included in the field of view of the HMD.

Brief Description of the Drawings

[0011] [Figure 1] Configuration of the HMD in the First Embodiment [Figure 2] Usage Mode of the HMD [Figure 3] Flowchart of Hand Authentication Process for the HMD Wearer in the First Embodiment [Figure 4] Composite Image When a Static Gesture Instruction is Displayed [Figure 5] Characteristic Shape of the Operation Instruction in FIG. 4 [Figure 6] Gesture Execution Position of the Operation Instruction in FIG. 4 [Figure 7] Composite Image with Mask Processing Performed after Gesture Authentication [Figure 8] Composite Image When a Static Gesture Instruction Executed with Both Hands is Displayed [Figure 9] Composite Image When a Moving Gesture Instruction is Displayed [Figure 10] Schematic Diagram of the Position on the Virtual Space When the Operation Instruction in FIG. 9 is Executed [Figure 11] Flowchart of Hand Authentication Process for the HMD Wearer in the Third Embodiment [Figure 12] Composite Image at S1102 in the Third Embodiment [Figure 13] Flowchart of Hand Authentication Process for the HMD Wearer in the Fourth Embodiment [Figure 14] User Estimation Based on Detection Information [Figure 15] Configuration of the HMD in the Fifth Embodiment [Figure 16] Composite Image When a Moving Gesture Instruction Using Voice Output is Displayed [Figure 17] Composite Image When a Moving Gesture Instruction Using Gaze Detection is Displayed

Modes for Carrying Out the Invention

[0012] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings. The embodiments described below are examples of the implementation means of the present invention, and may be appropriately modified or changed depending on the configuration of the apparatus to which the present invention is applied and various conditions. Also, it is possible to appropriately combine each embodiment.

[0013] (First Embodiment) Referring to FIGS. 1 to 2, the configuration of the HMD according to the first embodiment will be described. FIG. 1 is a block diagram showing the configuration of the HMD according to the first embodiment. FIG. 2 is a diagram showing the usage mode of the HMD.

[0014] The HMD 10 is a device worn on the head of a user, and is composed of a goggle device 11 and a control device 12. FIG. 2 schematically shows an example of the state in which the user wears the HMD 10. The goggle device 11 has at least an imaging unit 101 and a display unit 102. The control device 12 has at least a detection unit 103, an instruction unit 104, a comparison unit 105, an identification unit 106, a control unit 107, a photographed image storage unit 108, a hand information storage unit 109, and a composite image drawing unit 110. The control device 12 is a small computer (information processing device) equipped with a CPU (processor), memory, storage, communication device, etc. as hardware resources. The functions and processes of the control device 12 described later are realized by expanding a program non-temporarily stored in the storage into the memory and the CPU executing the program. Note that part or all of the functions and processes of the control device 12 may be replaced with a dedicated chip such as an FPGA or ASIC, or external resources such as a cloud server or a smartphone may be used.

[0015] The goggle device 11 of the HMD10 is a display device worn on the head. As shown in Figure 2, the HMD10 of this embodiment is a video see-through system in which the imaging unit 101 is positioned at the user's viewpoint, and the live-action video captured by the imaging unit 101 is displayed on the display unit 102. In the case of a video see-through HMD10, it is common to treat the imaging unit 101 as the user's viewpoint position and posture. Note that although the HMD10 of this embodiment is a video see-through system, this should be understood as merely an example. For example, the present invention can also be applied to a virtual reality HMD that does not display live-action video on the display unit 102. In that case, the imaging unit 101 can be used not as a see-through camera, but as a camera for capturing the hand, which is the object to be authenticated.

[0016] The control device 12 of the HMD 10 uses the image captured by the imaging unit 101 to create composite image data for display on the display unit 102. The control device 12 may be built into the same housing as the goggle device 11, or it may be configured in a separate housing from the goggle device 11. If the control device 12 and the goggle device 11 are configured in separate housings, the control device 12 and the goggle device 11 are connected to each other via wired or wireless means so that they can communicate with each other.

[0017] The imaging unit 101 is fixed to the housing of the goggle device 11 and consists of a sensor such as a CCD or CMOS and a lens, and outputs the image obtained by photographing the subject to the captured image storage unit 108. The imaging unit 101 can change the shooting conditions (number of pixels to be captured, shooting frequency (frame rate), exposure settings) through settings.

[0018] The display unit 102 is fixed to the housing of the goggle device 11 and is composed of, for example, an organic EL display or a liquid crystal display, and displays the composite image generated by the composite image rendering unit 110. This composite image is, for example, an image for MR (Mixed Reality) in which 3DCG digital content (virtual objects) is combined with a captured image acquired by the imaging unit 101. The display unit 102 has a structure that covers the field of view of the user wearing the goggle device 11, and can provide the user viewing the image with a highly immersive MR experience.

[0019] The detection unit 103 detects a hand from the captured image taken by the imaging unit 101 using image recognition and calculates the three-dimensional position and orientation of the hand and fingers as detection information. The detection information is stored in the hand information storage unit 109. In addition to the hand and fingers, the detection information may also include information about the arm, clothing, and accessories. Furthermore, in addition to the three-dimensional position and orientation, the detection information may also include information such as angle, size, color, and shape.

[0020] The instruction unit 104 generates an action instruction based on the authentication request received from the control unit 107 and transmits the action instruction to the composite image rendering unit 110. An action instruction is an instruction that prompts the HMD wearer to perform an action using their hands or fingers. The instruction unit 104 also generates a judgment criterion (first feature quantity), which is a feature quantity that quantifies the action instruction, and transmits the judgment criterion to the comparison unit 105.

[0021] The comparison unit 105 compares the judgment criteria (first feature quantity) of the operation instruction received from the instruction unit 104 with the detection information (second feature quantity) obtained from the hand information storage unit 109. At this time, the comparison unit 105 obtains the detection information corresponding to the judgment criteria (detection information to be compared with the judgment criteria) from among the multiple pieces of detection information stored in the hand information storage unit 109. The comparison result is output to the identification unit 106.

[0022] The identification unit 106 assigns an identifier to each piece of detection information stored in the hand information storage unit 109 based on the comparison results obtained from the comparison unit 105. The identifiers include a user identifier indicating that the user is wearing an HMD, as well as a non-user identifier indicating that the user is not wearing an HMD. You may grant it a child.

[0023] The control unit 107 refers to the hand information storage unit 109 and grants control rights to operate the user interface in the virtual space generated by the HMD 10 to the hand to which a user identifier has been assigned. The control unit 107 accepts the operation of the user interface by the hand to which control rights have been granted and controls each part of the HMD 10 according to the operation. The control unit 107 can also set shooting settings for the imaging unit 101, send authentication requests to the instruction unit 104, and set display settings for the composite image rendering unit 110.

[0024] The captured image storage unit 108 stores the captured image acquired by the imaging unit 101. The hand information storage unit 109 stores the detection information calculated by the detection unit 103 and the identifier generated by the identification unit 106.

[0025] The composite image rendering unit 110 generates a composite image by combining the captured image acquired from the captured image storage unit 108 with the operation instructions generated by the instruction unit 104. Then, the composite image rendering unit 110 applies the display settings made by the control unit 107 to the composite image and outputs the composite image to the display unit 102.

[0026] Figure 3 is a flowchart of the hand authentication process for the HMD wearer using a comparison of judgment criteria and detection information in the first embodiment.

[0027] In step S301, the instruction unit 104 determines whether it has received an authentication request. If it has received an authentication request, it proceeds to step S302; otherwise, it terminates without continuing the flow. The instruction unit 104 receives an authentication request from the control unit 107 when the HMD wearer or the software within the HMD 10 requires gesture authentication. This example shows the case where the instruction unit 104 receives an authentication request when the HMD 10 is started up.

[0028] In step S302, the instruction unit 104 generates an image of an action instruction prompting the HMD wearer to perform a predetermined action, and the composite image rendering unit 110 combines the image of the action instruction with the captured image and displays it on the display unit 102.

[0029] Action instructions include, for example, instructions for the HMD wearer to make a specified shape with their hand or fingers, instructions to move their hand or fingers to a specified position, and instructions to make a specified shape at a specified position. Since all of these instructions are gestures that stop the hand or fingers in a specified shape or position, these action instructions will henceforth be referred to as static gesture instructions. Figure 4 is an example of a composite image when static gesture instructions are displayed, in which text instructions 401 and image instructions 402 are composited into the captured image as action instructions. The captured image shows the HMD wearer's hand 403 and another person's hand 404. Image instructions 402 show the hand or finger movements or postures to be performed by the HMD wearer using an image. The example in Figure 4 is an image that prompts the wearer to move their right hand to a position that overlaps with the silhouette and assume a posture with all fingers spread out.

[0030] The instruction unit 104 generates an action instruction and also generates a judgment criterion for that action instruction, and passes the judgment criterion to the comparison unit 105. In the action instruction in Figure 4, for example, the two points that serve as the judgment criterion are the characteristic shape represented by the HMD wearer's hand and the execution position. Possible characteristic shapes to serve as the criterion include the point cloud 501 (also called contour information) drawn along the contour of the hand and fingers, as shown in Figure 5A, and the point cloud 502 (also called skeletal information) which abstracts the shape of the hand and fingers, as shown in Figure 5B. In this case, the correctness of the action can be determined from the amount of error by comparing the point cloud with the point cloud calculated from the hand detected in step S303, which will be described later. Alternatively, correct and incorrect images may be prepared, and a classification model may be created using classical machine learning. As for the location, as shown in Figure 6, a possible location is between the HMD 10 and the image instruction 402 (region 601), where, from the perspective of the HMD wearer, the hand performing the gesture and the instructed silhouette can be considered to overlap.

[0031] In step S303, the control unit 107 authorizes some of the operations of the HMD 10. If the HMD 10 has functions that should take priority over the risk of accidental operation, such as an emergency call function or a volume control function, it may be possible to authorize only those functions. After authorizing some of the operations, the process proceeds to step S304.

[0032] In step S304, the detection unit 103 uses the captured image received from the imaging unit 101 to detect a hand and the position and orientation of the hand and fingers. Once the detection unit 103 detects a hand from the captured image, the process proceeds to step S305. As an algorithm for detecting a hand from an image, classical machine learning such as support vector machines may be used, or deep learning-based algorithms such as R-CNN, YOLO, SSD, DCN, or rule-based detection algorithms may be used. In order to obtain detection information suitable for the comparison process in step S305, the control unit 107 may change the shooting conditions of the imaging unit 101 when capturing an image for hand detection (e.g., number of pixels, frame rate, exposure settings, etc.) depending on the type of action instruction. For example, in the case of a static gesture instruction, it is advisable to set the frame rate, which is the shooting frequency of the imaging unit 101, to a low frame rate and the number of pixels to a high resolution in order to accurately detect the position and orientation of the hand and fingers. The position and orientation of the hand and fingers can be measured using LeapMotion from LeapMotion Inc. There are also methods using deep learning or using existing publicly available libraries. The acquired images of each hand, as well as detection information including the position and orientation of the hands and fingers, are stored in the hand information storage unit 109. If multiple hands are detected in the captured image (within the field of view of the HMD 10), as in the example in Figure 4, detection information is acquired for each hand.

[0033] In step S305, the comparison unit 105 obtains the hand detection information detected from the captured image by the detection unit 103 from the hand information storage unit 109 and determines whether it matches the criteria for the operation instruction. If multiple hands are detected from the captured image, the comparison unit 105 simply compares the detection information of each hand with the criteria. If a hand detection information that matches the criteria for the operation instruction is found (i.e., a hand that performed an operation according to the operation instruction is detected), the process proceeds to step S306; otherwise, the process returns to step S304.

[0034] In comparing the judgment criteria for the motion instruction with the detected hand information, the comparison unit 105 may modify the detected information data or extract data from the detected information to match the judgment criteria. That is, the comparison unit 105 may, for example, extract contour information of the hand and fingers from the captured image for comparison with the point cloud 501, or perform position and orientation detection (skeletal detection) of the hand and fingers for comparison with the point cloud 502. For static gesture instructions, it is possible to compare the judgment criteria with the detected hand information with at least one image, but accuracy may be improved by comparing multiple images. The timing of starting the comparison should be delayed by a certain amount of time after the static gesture instruction is displayed on the display unit 102 (the time that is expected to take for the HMD wearer to recognize the static gesture instruction and perform the action. For example, tens to hundreds of milliseconds). Alternatively, for example, the comparison may be started as a trigger when any hand detected by the detection unit 103 enters the region 601.

[0035] In step S306, the identification unit 106, based on the comparison result in step S305, considers the hand that performed the action closest to (most matching) the action instruction to be the HMD wearer's hand, and assigns a user identifier to the detection information of that hand. This process corresponds to the process of authenticating the HMD wearer's hand. The user identifier assigned here is maintained until the detection unit 103 can no longer determine that the hand to which the user identifier was assigned is the same hand, either because the hand moves out of the field of view of the imaging unit 101 or is obstructed by an object. The user identifier is stored in the hand information storage unit 109 as part of the detection information. Furthermore, any hand other than the one identified as belonging to the HMD wearer in step S305 (i.e., a hand that did not perform an action matching the action instruction) is identified as belonging to another person, and a non-user identifier is assigned to its detection information. This assigned non-user identifier is used in step S308.

[0036] In step S307, the control unit 107 grants control of the HMD 10 to the hand that has been assigned a user identifier. From this point onward, the control unit 107 detects or tracks the movements and posture of the hand that has been assigned a user identifier (the authenticated hand) and accepts this as input for operations and commands to the HMD 10. This allows only the HMD wearer to operate the user interface in the virtual space generated by the HMD 10. Gesture authentication is now complete, and the HMD control rights remain valid until the hand that has been assigned a user identifier can no longer be detected. While these HMD control rights are valid, exclusive control is implemented to prevent operation by other hands. Note that if the HMD wearer's hand moves out of the field of view of the imaging unit 101 or is hidden by an obstruction, and the hand that has been assigned a user identifier can no longer be detected in the captured image (i.e., the HMD 10 loses sight of the hand that has been assigned a user identifier), the HMD control rights are invalidated.

[0037] In step S308, the control unit 107 performs a masking process that distinguishes each hand by assigning a different color based on the user identifier and non-user identifier. Figure 7 is an example of a composite image displayed on the display unit 102 after the masking process. Because hands 701 to 703 are each assigned a different color, the HMD wearer can recognize that their own hands and the hands of others are distinguishable within the HMD 10. Alternatively, the HMD wearer may be assigned a specific color to indicate that they have been authenticated as the HMD wearer. In this way, by displaying an image in which the hands authenticated as the HMD wearer's hands are rendered in a manner that distinguishes them from the hands of others, the HMD wearer can easily understand that their hands have been correctly authenticated.

[0038] By performing the series of processes described in steps S301 to S308 above, the HMD wearer's hands can be authenticated when the HMD 10 is started. According to the method of this embodiment, the HMD wearer only needs to manipulate the position and orientation of their hands and fingers according to the instructions, thus enabling simple authentication. Moreover, since the gesture instructions are not visible to anyone other than the HMD wearer, no one other than the HMD wearer can perform gesture authentication. Therefore, secure authentication can be achieved. Furthermore, since there is no need to use sensor devices or make prior preparations for personal authentication as in conventional methods, the burden on the user is reduced.

[0039] (Second embodiment) In the first embodiment, the instruction unit 104 issued a static gesture instruction in step S302, but the operation instruction is not limited to this. In the second embodiment, two other types of operation instructions that the instruction unit 104 can generate are given as examples. The flowchart is the same as that of the first embodiment.

[0040] The first action instruction is a static gesture instruction performed with both hands. Figure 8 shows a static gesture instruction in which a triangle is displayed on the screen of the display unit 102 and the HMD wearer is instructed to make a triangle using both hands, and a composite image of the hands performing this action captured by the imaging unit 101. Here, similar to the action instruction exemplified in Figure 4 shown in the first embodiment, the action instruction in Figure 8 also uses two points as criteria for determination: characteristic shape and execution position. In this case, the characteristic shape that serves as the criterion refers to a state in which the angular relationship of the figure formed by the thumb and index finger is close to a triangle. The execution position that serves as the criterion refers to a state in which, for example, the center position of the figure formed by both hands is located inside the figure instruction 801 as seen from the HMD 10. Both hands that the comparison unit 105 has determined to have performed this action instruction are each assigned a user identifier to the identification unit 106, and the control unit 107 gives each hand HMD control rights. According to this action instruction, since the HMD wearer is assigned a user identifier to both hands, it becomes possible to control the HMD 10 with either the left or right hand.

[0041] The second type of motion instruction is a movement gesture instruction, in which the HMD wearer moves their hand or finger along a specified trajectory. In movement gesture instructions, authentication is performed using multiple consecutive images rather than a single image, as the movement trajectory of the hand or finger is compared with the specified trajectory. Therefore, in movement gesture instructions, it is preferable for the control unit 107 to set shooting conditions for the imaging unit 101 and increase the video frame rate of the imaging unit 101 in order to increase the amount of data on the position of the hand or finger over time and improve the comparison accuracy. Figure 9 shows an example of a movement gesture instruction in which an arrow is displayed on the display unit 102 and the wearer is instructed to trace the arrow from its starting point P1 to its ending point P2 with their index finger, and a composite image of the HMD wearer's hand. In movement gesture instructions like the one in Figure 9, a characteristic shape is not required as a judgment criterion, as in a static gesture instruction; only the execution position is required. For example, the judgment criterion could be that any hand on the screen detected by the detection unit 103 touches the starting point P1 of the arrow, maintains a certain distance from the line segment P1-P2, and then touches the ending point P2 of the arrow. The contact detection performed by the comparison unit 105 requires determining the position and time. For example, one possible method of detection is to determine if contact has occurred if the position of the index finger remains within a radius of 3 cm from P1 in the virtual space for 0.1 seconds (3 frames at 30 frames / second). This is schematically represented in Figure 10. The black circles represent the starting point P1 and the ending point P2, and the white circles represent the group of positions Qn=Q1,Q2,Q3,... of the HMD wearer's index finger recorded for each frame. Furthermore, Qn from contact with P1 to contact with P2 is represented by a white circle with a thick border. In Figure 10, among the positions of the HMD wearer's index finger from contact with P1 to contact with P2, Q10 is the position furthest from the line segment P1-P2. At this time, the distance between position Q10 and the line segment P1-P2 is calculated, and by determining whether this is, for example, within 3 cm, the matching of the detection criteria and the detected information can be determined. Here, we used the trajectory of a line segment composed of P1 and P2 as an example, but the instruction could also be to perform authentication by simply touching point P1. Alternatively, the instruction could be to perform authentication by adding points such as P3 and P4 to create a polygon, curve, or trajectory resembling an alphabet. Furthermore, the instruction could be to perform an action of touching multiple points P1, P2, ... in a specified order with the hand or fingers.In this case, there is no need to evaluate the distance between position Qn and the line segment; instead, it is sufficient to evaluate whether contact with the next point Pm+1 occurred within a predetermined time after contact with point Pm. The simpler the authentication, the easier it is to implement; the more complex it is, the more secure the authentication becomes. According to these instructions, the HMD wearer does not need to represent a feature shape, only move their hand, allowing for more intuitive authentication. Movement gestures are not only less burdensome for the HMD wearer, but also have the advantage of requiring less effort to define judgment criteria because they do not involve comparing feature shapes.

[0042] (Third embodiment) In the first embodiment, gesture authentication was performed when the HMD 10 was started. While performing gesture authentication every time the HMD is started is secure, it also increases the frequency of authentication. In the third embodiment, control rights to the HMD 10 are granted by a different method when the HMD 10 is started, and gesture authentication is performed only for some operations. Figure 11 is a flowchart of the hand authentication process for the HMD wearer in the third embodiment. Below, detailed explanations of parts that are the same as in the first embodiment will be omitted, and the distinctive features of the third embodiment will be mainly described.

[0043] In this embodiment, operations on the HMD10 are pre-classified into multiple categories (levels) according to the level of security risk (in other words, the criticality (fatality) in the event of an error). For example, in the following example, operations are classified into Category 1, which has a low security risk (also called "unspecified operations"), and Category 2, which has a higher security risk than Category 1 (also called "specified operations"). Then, the necessary HMD control rights are separated for each category (level), and the requirement for authentication is set for each HMD control right.

[0044] In step S1101, the detection unit 103 detects a hand from the image captured by the imaging unit 101 and grants the first HMD control right to the first hand detected. The first HMD control right is This right allows the user to perform only Category 1 operations (non-specific operations) with low security risk. By granting Category 1 HMD control rights without hand authentication, the HMD wearer can perform non-specific operations without cumbersome authentication, resulting in superior usability. While there is a risk of accidental operation by another person, non-specific operations are low-security-risk operations, so accidental operation by another person is not a critical problem.

[0045] In step S301, the instruction unit 104 determines whether it has received an authentication request. If the instruction unit 104 has received an authentication request, it proceeds to step S302; otherwise, it terminates without continuing the flow. Here, we assume a case where an authentication request is issued when an HMD wearer attempts to conduct a financial transaction using a user interface in a virtual space within an application on the HMD 10. Because financial transactions require high security, operations related to financial transactions are set as specific operations and can only be performed by the hand with the second HMD control right. The second HMD control right is the right that allows the performer to perform specific operations.

[0046] In step S302, the HMD 10 issues an operation instruction to the HMD wearer. Here, the same operation instruction as in the first embodiment is used.

[0047] In step S1102, the control unit 107 permits only non-specific operations of the HMD 10 and prohibits the input of specific operations. Figure 12 is an example of a composite image displayed on the HMD 10 in step S1102 of the third embodiment. The back button 1201 is a virtual space user interface that cancels a financial transaction when touched by the hand with control of the first HMD. The amount input button 1202 is a virtual space user interface that inputs an amount when touched by the hand with control of the second HMD. The confirmation button 1203 is a virtual space user interface that completes the amount input and executes the transaction when touched by the hand with control of the second HMD. In other words, the hand with control of the first HMD can operate the back button 1201, but cannot operate the amount input button 1202 or the confirmation button 1203.

[0048] In step S304, the detection unit 103 detects a hand from the image captured by the imaging unit 101.

[0049] In step S1103, the control unit 107 determines whether an interruption operation has occurred for gesture authentication. Here, an interruption operation is determined when the back button 1201 is pressed by the hand that has control of the first HMD. If an interruption operation has occurred, the flow is terminated without continuing; otherwise, the process proceeds to step S305.

[0050] In steps S305 to S306, the HMD 10 performs gesture authentication and assigns an identifier. In step S1104, the control unit 107 grants the second HMD control rights to the HMD wearer's hand. This allows the HMD wearer to perform specific operations, namely the amount input button 1202 and the confirmation button 1203, enabling them to conduct financial transactions.

[0051] By performing the above series of processes, when the HMD10 is started, a first HMD control right is granted that is only permitted for non-specific operations, and gesture authentication can be performed only when a specific operation is required. According to this embodiment, gesture authentication is not performed for basic operations with low security risks, while it is performed only in situations where there is a high risk of erroneous operation by others, such as financial transactions. This reduces the burden on the user of performing gesture authentication while ensuring security.

[0052] (Fourth embodiment) In the first to third embodiments, when the instruction unit 104 receives an authentication request, gesture authentication is performed. This is performed. However, when the HMD wearer's hand moves out of the field of view of the imaging unit 101 or is hidden by an obstruction and the HMD 10 can no longer detect the hand (when it loses sight of the hand), authentication is also required, which may result in frequent gesture authentication. In the fourth embodiment, an example is shown in which gesture authentication is omitted by using past authentication results. This makes it possible to skip gesture authentication when the HMD wearer's hand re-enters the field of view of the imaging unit 101 after the HMD 10 has lost sight of the hand.

[0053] Figure 13 is a flowchart of the hand authentication process for the HMD wearer in the fourth embodiment. Hereinafter, detailed explanations of parts identical to the first embodiment will be omitted, and the distinctive features of the fourth embodiment will be primarily described.

[0054] Here, we assume a scenario where, during the previous authentication process, multiple hands were detected, one of which was assigned a user identifier, and the remaining hands were assigned non-user identifiers. However, after that, the hand assigned the user identifier was no longer detected, and control of the HMD was invalidated.

[0055] In step S1301, the detection unit 103 performs hand feature detection using the captured image received from the imaging unit 101. Here, features refer to, for example, the color, size, length, thickness, shape, etc., of the HMD wearer's hand or fingers. The hand features acquired by the detection unit 103 are stored in the hand information storage unit 109, and the process proceeds to step S1302.

[0056] In step S1302, if any of the hands detected from the captured image have been assigned a non-user identifier in the previous authentication process, the hands with the non-user identifier are excluded from the comparison in the next step S1303. In other words, any hands that are known to belong to another person are excluded from the authentication process. For example, if N hands are detected from the captured image and M of them have been assigned a non-user identifier, only the (NM) hands with unknown identifiers will be used for the authentication process from step S1303 onward.

[0057] In step S1303, the detection information stored in the hand information storage unit 109, which was stored in step S1301, is used to determine whether any of the detected hands have similar features to hands that have previously been assigned a user identifier. If similar hands are found, the process proceeds to step S306. If no similar hands are found, the process proceeds to step S1304.

[0058] The comparison between hand features to which user identifiers have been previously assigned and hand features currently being detected can be performed in any way. Generally, features are defined as scalars or multidimensional vectors, and the similarity between features is defined as the reciprocal of the difference between scalars or the reciprocal of the distance between vectors. For example, if the similarity of the hand or finger color is above a threshold, it may be determined that the hands are similar. Alternatively, as shown in Figure 14, the finger ratios (L1 / L2, L1' / L2') of the first and second finger lengths of hand 1401 and hand 1402 may be calculated, and if the similarity of the finger ratios is above a threshold, the hands may be determined to be similar. In this case, the stricter the determination of similarity, i.e., the higher the similarity threshold, the higher the authentication frequency and the higher the security. Conversely, if the determination is lenient, i.e., the lower the similarity threshold, the lower the security.

[0059] In steps S306 to S308, a user identifier is assigned to hands that have similar characteristics to the hand with the user identifier, and a non-user identifier is assigned to all other hands. Then, HMD control is re-granted to the hands to which the user identifier was assigned, and masking is performed.

[0060] On the other hand, in step S1304, the control unit 107 issues an authentication request, and the flow corresponding to the first embodiment is executed. That is, if no hand similar to the hand with the user identifier is found, gesture authentication is performed.

[0061] By performing the above series of processes and using the results of past gesture authentication, the flow can be simplified. According to this embodiment, even if the HMD 10 loses track of the HMD wearer's hand, re-authentication by gesture can be omitted. In addition, by excluding hands with non-user identifiers from the comparison, unnecessary authentication processing can be reduced while lowering the risk of authentication errors.

[0062] In step S1303, a comparison is made with past data stored in the hand information storage unit 109. The extent to which past data is included in the comparison can be arbitrarily designed depending on the purpose. For example, if the purpose is to ignore (compensate for) the HMD wearer's hands temporarily moving out of the field of view or being hidden by an obstruction, the time can be set to a short period of several tens of milliseconds to several tens of seconds. The shorter the time, the higher the security. Alternatively, if it is known in advance that a particular user will continue to use the HMD 10 for a predetermined period of time (for example, when providing a customer with a 10-minute XR experience), the time can be set to approximately the same length as the planned usage time. This allows the user to continue using the HMD with valid control rights for a predetermined period of time (regardless of whether their hands move out of the field of view or are hidden) simply by performing gesture authentication once when putting on the HMD. Alternatively, it is possible to not set a time limit. That is, all the hand feature quantities and user identifiers of users who have performed gesture authentication in the past are registered and stored. Such settings are useful when the uses and users of the HMD10 are somewhat limited, and it is expected that the same user will use it repeatedly (for example, when it is used only by family members at home). The extent to which past data should be used for comparison (in other words, how long the stored user identifier should be retained before being discarded) can be pre-configured in the HMD10, or it can be changed by the user.

[0063] (Fifth embodiment) The first to fourth embodiments describe a method of authenticating the HMD wearer using the imaging unit 101 and the display unit 102. However, by installing other input / output means, the HMD 10 can perform operation instructions and authentication methods with greater flexibility. The fifth embodiment shows a case where additional configurations are made as shown in Figure 1. Figure 15 is a block diagram showing the configuration of the HMD according to the fifth embodiment. The flowchart is the same as that of the first embodiment.

[0064] The audio output unit 1501 is fixed to the housing of the goggle device 11 and outputs sound that can be heard by the HMD wearer. Preferably, the sound is output at a low volume from a position close to the HMD wearer's ears, like earphones or headphones, or it is directed or bone conduction is used so that it can only be heard by the HMD wearer.

[0065] The vibration output unit 1502 is fixed to the housing of the goggle device 11 and outputs vibrations that can be perceived by the HMD wearer. It is preferable that the vibrations are so quiet that they do not produce sound, so that only the HMD wearer can perceive them.

[0066] The gaze detection unit 1503 is fixed to the housing of the goggle device 11 and detects the gaze after photographing the eyes and surrounding area of ​​the HMD wearer. However, the eye photography and gaze detection may be performed in separate structures, with the detection processing carried out by the control device 12.

[0067] By using the above configuration, it is possible to give more flexible operation instructions in step S302 of Figure 3.

[0068] For example, Figure 16 shows a movement gesture instruction that instructs the user to move their hand or finger in time with the audio output. In this case, a possible criterion for judgment is that the change in the position of the index finger is close to the timing of the audio output. Specifically, the amount of change in the position of the index finger with respect to time is calculated, By comparing whether the period of the position change is close to the timing of the audio output, it is possible to determine whether the judgment criteria and the detection information match. If the audio output unit 1501 is not audible to people in the vicinity, the action instructions cannot be executed by anyone other than the HMD wearer, similar to displaying action instructions on a screen. Therefore, the hand that executes the action instruction can be authenticated as the hand of the HMD wearer. Note that this example also works if the audio is replaced with vibrations output by the vibration output unit 1502.

[0069] According to these instructions, using voice and vibration to give instructions means that the instructions occupy less of the screen compared to image-based instructions, making it possible to authenticate the HMD wearer's hands while maintaining a clear field of view.

[0070] Figure 17 also shows an example of a movement gesture instruction in which the hand or finger is moved in conjunction with the movement of the gaze. For example, the position of the hand or finger to focus the gaze on is instructed, such as "Look at your index finger," and authentication is performed by checking whether the position of the hand or finger and the gaze match within a predetermined time. At this time, the position coordinates of the gaze trajectory detected by the gaze detection unit 1503 become the judgment criterion, so by comparing these position coordinates with the position coordinates of the index finger, it is possible to determine whether the judgment criterion and the detected information match. At this time, the gaze of the HMD wearer cannot be seen from outside the HMD, that is, the action instruction cannot be performed by anyone other than the HMD wearer, so the hand that performs the action instruction can be authenticated as the hand of the HMD wearer.

[0071] According to this method of giving instructions, by using gaze detection as the instruction, the HMD wearer themselves will give the instructions and move their hand or finger in accordance with those instructions, making it possible to authenticate the HMD wearer's hand more intuitively. In the example in Figure 17, the location to focus the gaze on is instructed by message, but instead, the instruction may be output by voice. Also, the location to focus the gaze on does not have to be the fingertips. For example, the intersection of the gaze with the palm or the entire hand may be determined, or the action of focusing the gaze on multiple locations on the hand or fingers may be instructed, such as "Look at your right hand, then look at your left hand." In any of these methods, the HMD wearer themselves coordinates their gaze with the movement of their hand or finger, enabling authentication through intuitive movement.

[0072] (others) The embodiments described above are merely illustrative of preferred configurations of the present invention, and the scope of the present invention is not limited to the configurations of these embodiments. For example, the configurations of the first to fifth embodiments may be combined with each other insofar as there is no technical inconsistency.

[0073] For example, the following could be used as an action instruction. • A shape is drawn on the image displayed on the display unit 102, and the user touches the shape with their finger or hand. - Multiple shapes are drawn on the image displayed on the display unit 102, and the user is instructed to touch these shapes simultaneously with multiple fingers. - Multiple shapes are drawn on the image displayed on the display unit 102, and the user is instructed to touch these shapes sequentially with their hand or finger. • Shapes are drawn sequentially at different positions on the image displayed on the display unit 102, and the user is instructed to touch them sequentially with their hand or finger. For example, it can output instructions such as "Make a thumbs-up gesture" or "Extend only your index and little fingers" as images or audio, and change the hand or fingers to the specified shape. - The user registers an authentication action in the HMD10 in advance (assuming only the user knows the authentication action). Then, the user is prompted to perform an action without specifying the action, for example, by saying "Please perform the authentication action," and if the HMD wearer is able to perform the same action as the authentication action, the HMD wearer's hand is authenticated as the hand of a legitimate user.

[0074] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0075] The disclosures herein include the following configurations, methods, and programs:

[0076] [Configuration 1] A control device for an HMD (Head Mounted Display) comprising: an imaging means configured to capture the field of view of the wearer; and a display means configured to show the image to the wearer, The system is configured to give the wearer an action instruction prompting them to perform an action using their hand or fingers, and to authenticate the hand as the wearer's hand when it is determined from the captured image obtained by the imaging means that the hand is performing an action in accordance with the action instruction. Control device.

[0077] [Configuration 2] The control device is configured to grant control rights to the HMD to the hand that has been authenticated as the wearer's hand. The control device described in Configuration 1.

[0078] [Configuration 3] When multiple hands are detected from the captured image obtained by the imaging means, the control device is configured to authenticate the hand performing the action closest to the action instruction as the wearer's hand, and to determine that the other hands are the hands of others. The control device described in configuration 1 or configuration 2.

[0079] [Structure 4] The control device is configured to display an image on the display means in which the hand authenticated as the wearer's hand is depicted in a manner that makes it distinguishable from the hand of another person. A control device as described in any of Configurations 1 to 3.

[0080] [Composition 5] The aforementioned action instructions include instructions to make a specified shape using the hand or fingers. A control device as described in any of Configurations 1 to 4.

[0081] [Composition 6] The aforementioned action instruction includes an instruction to move the hand or finger to a specified position. A control device as described in any of Configurations 1 to 5.

[0082] [Composition 7] The aforementioned action instructions include instructions to move the hand or finger according to a specified trajectory or specified order. A control device as described in any of Configurations 1 to 6.

[0083] [Structure 8] The aforementioned action instructions include instructions to move the hand or fingers in conjunction with the movement of the eyes. A control device as described in any of Configurations 1 to 7.

[0084] [Composition 9] The control device is configured to change the shooting conditions when the imaging means captures an image for detecting a hand, according to the type of operation instruction given to the wearer. A control device as described in any of Configurations 1 to 8.

[0085] [Configuration 10] The control device provides the operation instruction to the wearer by displaying an image representing the operation instruction on the display means. A control device as described in any of Configurations 1 to 9.

[0086] [Composition 11] The HMD further comprises output means configured to output sound and / or vibration to the wearer, The control device provides the operation instruction to the wearer by outputting the content of the operation instruction as sound and / or vibration using the output means. A control device as described in any of Configurations 1 to 10.

[0087] [Composition 12] The operations performed on the aforementioned HMD are classified into several categories. The control device is configured to allow operation of the first category of the plurality of categories by a hand that is not authenticated as the wearer's hand, and to allow operation of the second category, which is different from the first category, only by the hand that is authenticated as the wearer's hand. A control device as described in any of Configurations 1 to 11.

[0088] [Composition 13] The operations in the second category above pose a higher security risk than the operations in the first category above. The control device described in configuration 12.

[0089] [Composition 14] The control device is configured to grant a user identifier and control rights to the HMD to the hand authenticated as the wearer's hand, and to keep the control rights to the HMD valid until the hand to which the user identifier has been granted can no longer be detected in the captured image obtained by the imaging means. A control device as described in any of Configurations 1 to 13.

[0090] [Composition 15] The control device is configured to invalidate the control rights of the HMD when the hand to which the user identifier has been assigned can no longer be detected in the captured image obtained by the imaging means. The control device described in configuration 14.

[0091] [Composition 16] The control device is configured such that, after the hand to which the user identifier has been assigned can no longer be detected in the captured image obtained by the imaging means, if a hand similar to the hand to which the user identifier has been assigned is detected in the captured image obtained by the imaging means, the user identifier and control rights of the HMD are reassigned to the detected hand without performing authentication by operation instructions. The control device described in configuration 15.

[0092] [Method 17] A control method for an HMD (Head Mounted Display) comprising an imaging means configured to capture the wearer's field of view, and a display means configured to show the image to the wearer, The steps include: giving instructions to the wearer to perform an action using their hand or finger; If it is determined from the captured image obtained by the imaging means that the hand is performing an action in accordance with the action instruction, the hand is authenticated as the wearer's hand. A method for controlling an HMD, including the control method.

[0093] [Program 18] A program for causing the processor of the HMD to execute each step of the HMD control method described in Method 17. [Explanation of symbols]

[0094] 10 HMD 11. Goggles 12 Control device 101 Imaging Unit 102 Display section 103 Detection unit 104 Instruction section 105 Comparison Section 106 Identification unit 107 Control Unit 108 Image storage unit 109 Hand information storage unit 110 Composite Image Drawing Unit

Claims

1. A control device for an HMD (Head Mounted Display) comprising: an imaging means configured to capture the field of view of the wearer; and a display means configured to show the image to the wearer, The system is configured to give the wearer an action instruction prompting them to perform an action using their hand or fingers, and to authenticate the hand as the wearer's hand when it is determined from the captured image obtained by the imaging means that the hand is performing an action in accordance with the action instruction. Control device.

2. The control device is configured to grant control rights of the HMD to the hand that has been authenticated as the hand of the wearer. The control device according to claim 1.

3. When multiple hands are detected from the captured image obtained by the imaging means, the control device is configured to authenticate the hand performing the action closest to the action instruction as the wearer's hand, and to determine that the other hands are the hands of others. The control device according to claim 1.

4. The control device is configured to display an image on the display means in which the hand authenticated as the wearer's hand is depicted in a manner that makes it distinguishable from the hand of another person. The control device according to claim 1.

5. The aforementioned action instructions include instructions to make a specified shape using the hand or fingers. The control device according to claim 1.

6. The aforementioned action instruction includes an instruction to move the hand or finger to a specified position. The control device according to claim 1.

7. The aforementioned action instructions include instructions to move the hand or finger according to a specified trajectory or specified order. The control device according to claim 1.

8. The aforementioned action instructions include instructions to move the hand or fingers in conjunction with the movement of the eyes. The control device according to claim 1.

9. The control device is configured to change the shooting conditions when the imaging means captures an image for detecting a hand, according to the type of operation instruction given to the wearer. The control device according to claim 1.

10. The control device provides the operation instruction to the wearer by displaying an image representing the operation instruction on the display means. The control device according to claim 1.

11. The HMD further comprises output means configured to output sound and / or vibration to the wearer, The control device provides the operation instruction to the wearer by outputting the content of the operation instruction as sound and / or vibration using the output means. The control device according to claim 1.

12. The operations performed on the aforementioned HMD are classified into several categories. The control device is configured to allow operation of the first category of the plurality of categories by a hand that is not authenticated as the wearer's hand, and to allow operation of the second category, which is different from the first category, only by the hand that is authenticated as the wearer's hand. The control device according to claim 1.

13. The operations in the second category above pose a higher security risk than the operations in the first category above. The control device according to claim 12.

14. The control device is configured to grant a user identifier and control rights to the HMD to the hand authenticated as the wearer's hand, and to keep the control rights to the HMD valid until the hand to which the user identifier has been granted can no longer be detected in the captured image obtained by the imaging means. A control device according to any one of claims 1 to 13.

15. The control device is configured to invalidate the control rights of the HMD when the hand to which the user identifier has been assigned can no longer be detected in the captured image obtained by the imaging means. The control device according to claim 14.

16. The control device is configured such that, after the hand to which the user identifier has been assigned can no longer be detected in the captured image obtained by the imaging means, if a hand similar to the hand to which the user identifier has been assigned is detected in the captured image obtained by the imaging means, the user identifier and control rights of the HMD are reassigned to the detected hand without performing authentication by the operation instruction. The control device according to claim 15.

17. A control method for an HMD (Head Mounted Display) comprising: an imaging means configured to capture the field of view of the wearer; and a display means configured to show the image to the wearer, The steps include: giving instructions to the wearer to perform an action using their hand or finger; If it is determined from the captured image obtained by the imaging means that the hand is performing an action in accordance with the action instruction, the hand is authenticated as the wearer's hand. A method for controlling an HMD, including the control of an HMD.

18. A program for causing the processor of the HMD to execute each step of the HMD control method described in claim 17.

Images