Head-mounted display, method for controlling a head-mounted display, and program

Abstract

We provide a head-mounted display that contributes to enabling comfortable operation for each user. [Solution] The head-mounted display comprises a biometric information acquisition means, an authentication means, a history control means, a history acquisition means, and a display control means. The biometric information acquisition means acquires the user's biometric information. The authentication means authenticates the user using the acquired biometric information. The history control means generates a usage history including information about the authenticated user's operations and stores the generated usage history. The history acquisition means retrieves the stored usage history. The display control means determines the interface on the virtual reality or augmented reality screen based on the acquired usage history.

Description

[[Technical Field]]

[0001] The present invention relates to a head mounted display, a control method for a head mounted display, and a program. [[Background Art]]

[0002] There are conventional technologies that reflect a user's proficiency level in screen display.

[0003] For example, Patent Document 1 describes that user-operable items to be displayed can be switched according to the proficiency level. The information processing apparatus of Patent Document 1 performs pre-settings for realizing mixed reality. The information processing apparatus includes a proficiency acquisition unit that acquires a proficiency level of the pre-settings, and a display switching unit that switches user-operable items to be displayed for the pre-settings according to the proficiency level. [[Prior Art Documents]] [[Patent Documents]]

[0004] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2021-149792 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0005] There exist HMDs (Head Mounted Displays) that are operated via line of sight and finger movements without using a predetermined controller or the like. A user operating such an HMD for the first time is unaccustomed to selecting icons and the like via line of sight, and may not be able to operate the HMD comfortably.

[0006] On the other hand, by continuing to use an HMD, users become accustomed to selecting icons and other elements with their gaze. If a virtual reality experience is provided with an interface adjusted for users unfamiliar with HMD operation, those users may perceive the HMD as sluggish. Thus, existing HMDs have a problem in that they cannot provide virtual reality with an interface that is appropriate for the user's state and circumstances.

[0007] Furthermore, Patent Document 1 merely discloses a technique for switching display items according to the user's proficiency level during calibration settings performed on an HMD. Therefore, applying the technique disclosed in Patent Document 1 will not solve the above problem.

[0008] The primary objective of this invention is to provide a head-mounted display, a method for controlling the head-mounted display, and a program that contribute to enabling comfortable operation for each user. [Means for solving the problem]

[0009] According to a first aspect of the present invention, a head-mounted display is provided, comprising: biometric information acquisition means for acquiring a user's biometric information; authentication means for authenticating the user using the acquired biometric information; history control means for generating a usage history including information about the authenticated user's operations and storing the generated usage history; history acquisition means for acquiring the stored usage history; and display control means for determining an interface on a virtual reality or augmented reality screen based on the acquired usage history.

[0010] A second aspect of the present invention provides a method for controlling a head-mounted display, comprising: a biometric information acquisition step for acquiring a user's biometric information; an authentication step for authenticating the user using the acquired biometric information; a history control step for generating a usage history including information about the authenticated user's operations and storing the generated usage history; a history acquisition step for acquiring the stored usage history; and a display control step for determining an interface on a virtual reality or augmented reality screen based on the acquired usage history.

[0011] A third aspect of the present invention is provided, which causes a computer mounted on a head-mounted display to execute a biometric information acquisition process for acquiring the user's biometric information, an authentication process for authenticating the user using the acquired biometric information, a history control process for generating a usage history including information about the authenticated user's operations and storing the generated usage history, a history acquisition process for acquiring the stored usage history, and a display control process for determining the interface on a virtual reality or augmented reality screen based on the acquired usage history. [Effects of the Invention]

[0012] From each perspective of the present invention, a head-mounted display, a method for controlling the head-mounted display, and a program are provided that contribute to enabling comfortable operation by each user. However, the effects of the present invention are not limited to those described above. The present invention may also produce other effects in lieu of or in conjunction with the effects described above. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a diagram illustrating the outline of one embodiment. [Figure 2] Figure 2 is a flowchart showing the operation of one embodiment. [Figure 3] Figure 3 is a diagram illustrating the use of the HMD according to the embodiment of this disclosure. [Figure 4]Figure 4 shows an example of the screen display of an HMD according to the embodiment of this disclosure. [Figure 5] Figure 5 shows an example of the processing configuration of an HMD according to the present disclosure. [Figure 6] Figure 6 shows an example of a user management database according to the embodiment of this disclosure. [Figure 7] Figure 7 is a flowchart showing an example of the operation of the display control unit according to the present disclosure. [Figure 8] Figure 8 shows an example of the screen display of an HMD according to the present disclosure. [Figure 9] Figure 9 shows an example of the screen display of an HMD according to the present disclosure. [Figure 10] Figure 10 shows an example of a processing configuration for an HMD according to a modified embodiment of the present disclosure. [Figure 11] Figure 11 shows an example of the hardware configuration of the HMD according to this disclosure. [Modes for carrying out the invention]

[0014] First, an overview of one embodiment will be described. The reference numerals in the drawings attached to this overview are provided for convenience as examples to aid understanding, and this overview is not intended to be limiting in any way. Furthermore, unless otherwise specified, the blocks shown in each drawing represent functional units, not hardware units. The connecting lines between blocks in each drawing include both bidirectional and unidirectional lines. Unidirectional arrows schematically indicate the flow of the main signal (data) and do not exclude bidirectional flow. In this specification and in the drawings, elements that can be similarly described are denoted by the same reference numerals, thus omitting redundant explanations.

[0015] A head mounted display (HMD) 100 according to one embodiment comprises biometric information acquiring means 101, authentication means 102, history control means 103, history acquiring means 104, and display control means 105 (see FIG. 1). The biometric information acquiring means 101 acquires biometric information of a user (step S1 in FIG. 2). The authentication means 102 authenticates the user using the acquired biometric information (step S2). The history control means 103 generates a usage history including information related to operations of the authenticated user, and stores the generated usage history (step S3). The history acquiring means 104 acquires the stored usage history (step S4). The display control means 105 determines an interface on a screen of virtual reality or augmented reality based on the acquired usage history (step S5).

[0016] The HMD 100 described above determines an interface suitable for the user based on the user's usage history. For example, when it is determined that the user is unfamiliar with the operation of the HMD 100, the HMD 100 selects an interface for beginners. In contrast, when it is determined that the user is skilled in the operation of the HMD 100, the HMD 100 selects an interface for skilled users. As a result, virtual reality or augmented reality is provided via an interface suitable for each user, and comfortable operation by the user is realized.

[0017] In the present disclosure, the term "interface" refers to "user interface (visual elements and operation methods)".

[0018] Specific embodiments are described in further detail below with reference to the drawings.

[0019] [First Embodiment] The first embodiment is described in further detail with reference to the drawings.

[0020] As shown in FIG. 3, a user wears a head mounted display (HMD) 10.

[0021] The HMD10 is a virtual reality or augmented reality device that seamlessly integrates the real world with virtual digital content. The HMD10 goggles are equipped with a high-definition display that can overlay three-dimensional digital content onto the real world that the user is viewing.

[0022] The HMD10 displays images on separate left and right displays within the goggles. These images enable the user to perceive three-dimensional space. The HMD10 also features a video camera positioned to capture the user's field of view, and employs a video see-through system that overlays digital content onto the real-time video captured by this camera.

[0023] For example, in Figure 4, the plant 31 and the television 32 are real-world objects captured by the video camera. In contrast, icons 41-46 are digital content created by the HMD 10.

[0024] The HMD10 has an operating system (OS) installed, just like a smartphone. Furthermore, the HMD10 has applications installed that run on that OS.

[0025] Icons 41-46 shown in Figure 4 are icons related to various settings of the HMD10 and icons for launching installed applications.

[0026] Furthermore, the screen displaying icons for settings and application launch, as shown in Figure 4, is called the home screen.

[0027] The HMD10 is equipped with a camera capable of capturing the user's eyes (pupil and iris) and is configured to detect the user's gaze. Additionally, the HMD10 is equipped with a camera capable of capturing the user's fingers and is configured to detect gestures made by the user's fingers.

[0028] Users can control the HMD10's OS and applications using eye movements and gestures. They can also control the HMD10's OS and applications using voice commands.

[0029] Next, we will describe the details of the HMD10 according to the first embodiment.

[0030] Figure 5 shows an example of the processing configuration (processing module) of the HMD10 according to the embodiment disclosed herein. Referring to Figure 5, the HMD10 comprises a mounting control unit 201, an authentication control unit 202, a gaze control unit 203, an operation control unit 204, a usage history control unit 205, a display control unit 206, and a storage unit 207.

[0031] The mounting control unit 201 is a control means that detects when a user puts on the HMD 10 and when a user removes the HMD 10. The mounting control unit 201 detects when the user starts using the HMD 10 and when they stop using it.

[0032] The mounting control unit 201 detects when a user has put on the HMD 10 based on image data obtained from a camera installed inside the HMD 10 and output signals obtained from sensors. Similarly, the mounting control unit 201 detects when a user has removed the HMD 10 based on image data obtained from the camera, etc.

[0033] When the mounting control unit 201 detects that the user has put on the HMD10, it notifies other modules accordingly. Similarly, when the mounting control unit 201 detects that the user has removed the HMD10, it notifies other modules accordingly.

[0034] The authentication control unit 202 is a means for authenticating a user who intends to use the HMD 10. The authentication control unit 202 includes a function as a biometric information acquisition means for acquiring the user's biometric information, and a function as an authentication means for authenticating the user based on the acquired biometric information.

[0035] For example, the authentication control unit 202 authenticates the user (the person being authenticated) using biometric information.

[0036] Examples of biometric information include data (feature quantities) calculated from individual physical characteristics such as face, fingerprints, voiceprints, veins, retina, and iris patterns. Alternatively, biometric information may be image data such as facial images or fingerprint images. Biometric information only needs to include information about the user's physical characteristics. The embodiment disclosed here will describe the case in which the iris (iris pattern) is used.

[0037] The authentication control unit 202 acquires and stores the user's iris features (iris pattern) when the user uses the service for the first time, etc. The authentication control unit 202 acquires and stores the iris features of one eye or both eyes.

[0038] More specifically, the authentication control unit 202 assigns a user ID to a new user. The authentication control unit 202 associates the user ID with biometric information (iris features) and stores it in the user management database (see Figure 6). Note that the user management database shown in Figure 6 is an example and is not intended to limit the items to be stored.

[0039] The authentication control unit 202 authenticates the user using the acquired iris features when the user uses the HMD 10 for the second time or later. The authentication control unit 202 performs an authentication process (one-to-many authentication; N is a positive integer, the same applies hereafter) using the user's (authenticated person's) iris features and multiple iris features stored in the user management database.

[0040] The authentication control unit 202 calculates the similarity between the iris feature of the person being authenticated and each of the multiple iris feature values ​​stored in the user management database.

[0041] The authentication control unit 202 determines that authentication is successful if there is an iris feature (an iris feature stored in the user management database) with a similarity score equal to or greater than a predetermined value. Upon successful authentication, the user of the entry with the highest similarity score is identified as the authenticated user.

[0042] The authentication control unit 202 determines that authentication has failed if there are no iris features (iris features stored in the user management database) with a similarity score equal to or greater than a predetermined value.

[0043] If the authentication of the person to be authenticated is successful, the authentication control unit 202 allows the user to use the HMD10. If the authentication of the person to be authenticated fails, the authentication control unit 202 denies the user the use of the HMD10. Alternatively, the authentication control unit 202 registers the user (registers user ID and iris features) upon request from the user who failed to authenticate.

[0044] If the authentication of the person to be authenticated is successful, the authentication control unit 202 notifies other modules of the user ID of the user identified by the authentication process.

[0045] Since existing technologies can be used for calculating iris features and authentication processing using iris features, a detailed explanation of these processes will be omitted.

[0046] The gaze control unit 203 is a means for performing control related to the user's gaze. For example, the gaze control unit 203 tracks the user's gaze and identifies (estimates) the direction of the user's gaze.

[0047] First, the eye-tracking control unit 203 performs calibration for eye tracking when the user uses the HMD 10 for the first time.

[0048] For example, the gaze control unit 203 displays a calibration object on the screen and instructs the user to gaze at the displayed object. The gaze control unit 203 stores the relationship between the position where the object is displayed and the direction of the user's eyes (pupils). The gaze control unit 203 displays objects at various points in the user's field of vision and stores the relationship between the object and the direction of the eyes.

[0049] The gaze control unit 203 determines the direction of the gaze using the direction of the user's eyes obtained through calibration.

[0050] The gaze control unit 203 constantly detects (tracks) the user's gaze direction while the HMD 10 is running and notifies other modules of the detection results.

[0051] The operation control unit 204 is a means for controlling the user's operation of the HMD 10. For example, the operation control unit 204 functions as a hand movement detection means that detects the user's hand movements.

[0052] For example, the operation control unit 204 detects the user's actions based on image data obtained by photographing the user's hands and fingers. For example, the operation control unit 204 detects actions such as pinching an object with the thumb and index finger, and releasing the pinched object.

[0053] For example, the operation control unit 204 detects the user's actions (gestures) by inputting image data into a pre-prepared AI (Artificial Intelligence) model.

[0054] The operation control unit 204 constantly detects the user's actions (e.g., pinching, releasing) while the HMD 10 is running. Based on notifications from the gaze control unit 203 and the detected actions, the operation control unit 204 determines that the user has performed a predetermined operation and notifies other modules accordingly.

[0055] For example, if an icon is displayed in the user's line of sight and the user makes a pinching motion, the operation control unit 204 determines that the user has performed an operation related to that icon. For example, the operation control unit 204 notifies other modules of information related to the operation, such as "The settings menu has been launched" or "An application has been launched" (hereinafter referred to as operation information).

[0056] The usage history control unit 205 is a means for controlling the usage history of the HMD 10 by the user. The usage history control unit 205 generates a usage history that includes information about the operations of the authenticated user and stores the generated usage history.

[0057] Specifically, the usage history control unit 205 generates the user's usage history based on information notified from other modules and stores it in the user management database. The usage history control unit 205 also functions as an update means for updating the usage history stored in the user management database.

[0058] For example, when the usage history control unit 205 obtains a user ID from the authentication control unit 202, it recognizes that the user corresponding to that user ID has started using the HMD 10. Alternatively, when the usage history control unit 205 obtains operation information from the operation control unit 204, it temporarily stores the obtained operation information. When the usage history control unit 205 receives notification from the mounting control unit 201 that the user has removed the HMD 10, it recognizes that the user has finished using the HMD 10.

[0059] When the user finishes using the HMD10, the usage history control unit 205 generates the user's usage history using the temporarily stored operation information, etc., and stores the generated usage history in the user management database.

[0060] Through the operation of the usage history control unit 205, the usage history shown in Figure 6 is stored in the user management database.

[0061] The display control unit 206 is a means for performing control related to the display of the HMD 10. The display control unit 206 includes a function as a history acquisition means for acquiring usage history stored in a user management database, and a function as a display control means for determining the interface on the virtual reality screen based on the acquired usage history.

[0062] The display control unit 206 displays images obtained by capturing the real world using the video see-through method described above on the display inside the goggles, while superimposing objects such as icons and buttons (objects to be operated) onto the images. The display control unit 206 repeats this operation while the HMD 10 is running.

[0063] Figure 7 is a flowchart showing an example of the operation of the display control unit 206 according to the embodiment disclosed herein. The operation of the display control unit 206 will be explained with reference to Figure 7.

[0064] When the user ID is received from the authentication control unit 202 (when the user starts using the HMD 10), the display control unit 206 obtains the usage history of the user corresponding to that user ID (step S101).

[0065] Specifically, the display control unit 206 searches the user management database using the user ID as the key and identifies the corresponding entry. The display control unit 206 then reads the usage history from the identified entry.

[0066] The display control unit 206 determines the virtual reality interface to be provided to the user based on the acquired usage history (determine the interface; step S102).

[0067] For example, the display control unit 206 determines the interface based on the cumulative usage time of the HMD 10 by the user.

[0068] Alternatively, the display control unit 206 may determine the interface based on the user's level of proficiency in operating the HMD 10 (operation in virtual reality).

[0069] For example, the display control unit 206 calculates the user's proficiency level using artificial intelligence (AI). For instance, the display control unit 206 generates a prompt such as, "Please output the user's proficiency level in two stages based on the user's usage history." The display control unit 206 inputs the generated prompt, along with the usage history, into a large-scale language model.

[0070] The Large Language Model (LLM) calculates the user's proficiency level based on the application used, the accuracy of operations, the usage time, etc. The display control unit 206 acquires the proficiency level calculated by the Large Language Model.

[0071] For example, a large-scale language model might calculate a low proficiency score if a user spends a long time using the language but much of that time watching videos. Conversely, a large-scale language model might calculate a high proficiency score if a user spends a short time using the language but much of that time creating documents or performing other similar tasks.

[0072] If the display control unit 206 determines that the user is a "beginner" based on cumulative usage time and proficiency level, it selects "beginner-friendly" as the interface for displaying objects such as icons. In other words, if the cumulative usage time and proficiency level are below a predetermined value, the display control unit 206 recognizes that the user is a beginner unfamiliar with operating the HMD10 and displays objects on the display screen (for example, icons on the home screen) using a beginner-friendly interface.

[0073] The display control unit 206 selects "for experts" as the object display method if it determines that the user is an "expert" based on cumulative usage time and proficiency level. In other words, if the cumulative usage time and proficiency level are greater than a predetermined value, the display control unit 206 recognizes the user as an expert who is proficient in operating the HMD10 and displays the objects on the display screen using the expert interface.

[0074] The display control unit 206 obtains the user's gaze direction from the gaze control unit 203. Based on the obtained gaze direction, the display control unit 206 determines whether the user's gaze is directed towards an object on the screen (for example, a specific icon on the home screen) (gaze determination; step S103).

[0075] If the user's gaze is not directed towards the object (step S104, No branch), the display control unit 206 does not perform any special processing and repeats the gaze determination process in step S103.

[0076] If the user's gaze is directed towards an object (step S104, Yes branch), the display control unit 206 provides visual feedback to the object that the gaze is directed towards using the interface determined above (step S105).

[0077] More specifically, the display control unit 206 provides visual feedback so that the user can recognize that an object has been selected. The display control unit 206 provides appropriate visual feedback to the object selected by the user, according to the user's skill level, etc.

[0078] For example, consider the case where the beginner interface is selected. In this case, when the user is looking at icon 41 shown in Figure 8, the display control unit 206 provides beginner-friendly visual feedback to icon 41.

[0079] For example, the display control unit 206 may surround the icon 41 with a dotted line or the like, as shown in Figure 8. Alternatively, the display control unit 206 may display a star or arrow near the icon 41, or change the color of the icon 41.

[0080] In this way, the display control unit 206 provides exaggerated (over-the-top) visual feedback to the object so that even users unfamiliar with using the HMD 10 can easily recognize the object they have selected.

[0081] Next, let's consider the case where the expert interface is selected. When the user is looking at icon 41 shown in Figure 9, the display control unit 206 provides expert-level visual feedback to icon 41.

[0082] For example, the display control unit 206 displays the icon 41 in three dimensions (as if viewing a triangular pyramid from above), as shown in Figure 9.

[0083] Thus, if the user is an experienced user familiar with the HMD10, the display control unit 206 provides simple visual feedback to the object selected by the user.

[0084] The display control unit 206 repeats the processes from step S103 to step S105 until the user finishes using the HMD 10.

[0085] In this way, the display control unit 206 calculates the user's proficiency level in virtual reality operations based on the acquired usage history, and determines the interface on the virtual reality screen based on the calculated proficiency level. Based on the gaze direction estimated by the gaze control unit 203, the display control unit 206 identifies the object that the user is viewing from among the multiple objects displayed on the virtual reality screen. The display control unit 206 displays the identified object in a display format corresponding to the interface (beginner interface, expert interface) determined based on the usage history. More specifically, the display control unit 206 provides visual feedback to the identified object based on the calculated proficiency level.

[0086] "Display format" refers to the visual representation of an object, including its size, color, shape, and animation effects. For example, in a beginner's interface, the display control unit 206 may enlarge the size of a specified object or enlarge the text contained within a specified object. The display control unit 206 may also display the outer edge of a specified object in red or display a cursor near a specified object.

[0087] The memory unit 207 is a means for storing information necessary for the operation of the HMD 10.

[0088] Next, a modified example of the first embodiment will be described.

[0089] <Example 1> The stability of eye tracking and the accuracy of operation may vary depending on the user's skill level. Therefore, a uniform sensitivity setting may not provide the optimal user experience. Considering these circumstances, HMD10 may be equipped with a function to adjust the eye-tracking sensitivity according to the user's skill level.

[0090] Specifically, the display control unit 206 adjusts the sensitivity to eye gaze according to the user's skill level. More specifically, the display control unit 206 determines that an object has been selected if the user's gaze is directed at the same object for a predetermined period of time. The display control unit 206 determines (adjusts) the predetermined period of time based on the user's skill level.

[0091] For example, for highly skilled users, the display control unit 206 sets a shorter predetermined time. By setting a shorter predetermined time, objects can be quickly selected in response to subtle eye movements.

[0092] Alternatively, for users with low skill levels, the display control unit 206 sets a longer predetermined time. By setting a longer predetermined time, the object is selected as intended by the user.

[0093] The display control unit 206 detects user operations (object selection) based on sensitivity settings corresponding to the user's skill level.

[0094] Thus, the display control unit 206 may adjust the sensitivity when identifying the object being viewed by the user based on the calculated proficiency level. Such adjustment enables optimal eye-tracking according to the user's proficiency level. As a result, the user is provided with a more natural and intuitive operation.

[0095] <Modification 2> The HMD10 may allow users to change settings related to visual feedback. Specifically, the HMD10 may allow users to select the form in which visual feedback is provided to objects according to their preferences.

[0096] For example, the HMD10 may allow the user to set the color of the dotted lines surrounding objects during visual feedback. Alternatively, the HMD10 may allow the user to set the shape (star, arrow, etc.) displayed near objects during visual feedback, as well as the color of that shape.

[0097] Thus, the HMD10 may allow the user to customize the form of visual feedback. The display control unit 206 may provide visual feedback to an object identified based on the setting value selected by the user (the object the user is looking at).

[0098] Furthermore, as the user's skill level increases, HMD10 may increase the number of options (customization options) that the user can select. For example, as the user's skill level increases, HMD10 may increase the types of shapes that can be displayed near objects.

[0099] Alternatively, the HMD10 may allow changes to display settings other than the form of visual feedback. For example, the HMD10 may allow settings to be made on or off (displayed or not displayed) for cursors, arrows, pointers, etc., that follow the user's gaze.

[0100] Alternatively, the HMD10 may decide whether to display or hide a cursor or other elements that follow the user's gaze, depending on the user's skill level. For example, if the user is a beginner, the display control unit 206 will display the cursor or other elements. Conversely, if the user is an expert, the display control unit 206 will hide the cursor or other elements.

[0101] Thus, the display control unit 206 may provide experienced users with simple visual feedback, such as displaying objects in 3D or changing the color of those objects. For beginners, the display control unit 206 may display a cursor or the like so that the user can easily understand where the HMD 10 is looking (the place it is staring at).

[0102] <Variation 3> HMD10 may have a function to synchronize the user's usage history (usage history information) with other HMD10s. In this case, the user management database that stores user IDs and usage history may be configured on a server in the cloud.

[0103] When a user finishes operating the HMD10, the usage history control unit 205 sends the user's user ID and usage history to the server. The server adds the usage history to the entry corresponding to the user ID and updates the user management database.

[0104] The display control unit 206 transmits the user ID to the server and obtains the usage history of the user corresponding to that user ID.

[0105] Thus, the usage history control unit 205 may store the generated usage history on a server in the cloud. The display control unit 206 may obtain usage history, including the usage history of other HMDs 10, from the server in the cloud. By synchronizing usage history on the cloud in this way, usage history is shared among multiple HMDs 10. As a result, the display control unit 206 can calculate a more accurate proficiency level.

[0106] <Modification 4> The HMD10 may provide tutorials explaining how to use new functions and advanced settings, depending on the user's skill level. In this case, the HMD10 may be equipped with a tutorial provision unit 208 (see Figure 10).

[0107] The explanation unit 208 provides users with explanations on how to operate new functions or settings when the HMD10's OS is updated. In this case, the explanation unit 208 may determine the content to be provided according to the user's level of proficiency.

[0108] For example, for experienced users, the explanation section 208 provides a simple explanation. For beginners, the explanation section 208 provides a detailed explanation.

[0109] The explanation provision unit 208 may generate explanations to be provided to the user using a generation AI such as a large-scale language model. For example, the explanation provision unit 208 may generate a prompt such as, "Please create a simple operation procedure for experienced users regarding the new function A," and input it into the large-scale language model. The explanation provision unit 208 then provides the user with the operation procedure obtained from the large-scale language model.

[0110] Thus, the explanation provision unit 208 may provide the user with a tutorial relating to a function or setting, which is tailored to the user's calculated proficiency level.

[0111] The operation of each module, such as the display control unit 206 described above, is mainly realized by the OS installed on the HMD10. However, the operation of the display control unit 206 and other modules may also be realized by various applications installed on the HMD10.

[0112] The following describes examples of how the HMD10 can be used by installing applications on it.

[0113] <Example of use 1> For example, the HMD10 may be used for educational purposes.

[0114] In this case, the HMD10 may further include a learning support unit. The learning support unit analyzes the user's level of understanding from their eye movements, depending on the learning content. Based on the user's level of understanding, the learning support unit may present problems or explanations of appropriate difficulty.

[0115] <Example of use 2> For example, HMD10 may be used for medical purposes.

[0116] In this case, the HMD10 may further include a medical support unit. The medical support unit may adjust the level of detail of the surgical simulation and diagnostic support system according to the skill level of the medical personnel.

[0117] <Example of use 3> For example, the HMD10 may be used for entertainment purposes.

[0118] In this case, the HMD10 may further include a game adjustment unit. The game adjustment unit may automatically adjust the difficulty level and control methods in games and VR (Virtual Reality) experiences according to the user's skill level.

[0119] <Example of use 4> For example, HMD10 may be used in industrial fields.

[0120] In this case, the HMD10 may further include a safety support unit. The safety support unit may provide safety warnings and guidance tailored to the worker's skill level during work in a factory or construction site.

[0121] <Example of use 5> For example, the HMD10 may be used in retail stores, etc.

[0122] In this case, the HMD10 may further include a customer service support unit. The customer service support unit may display support information for customer service and product explanations according to the skill level of the store staff.

[0123] As described above, the HMD 10 according to the first embodiment determines an interface suitable for the user based on the user's usage history. For example, if it is determined that the user is unfamiliar with operating the HMD 100, the HMD 10 selects a beginner's interface. Conversely, if it is determined that the user is proficient in operating the HMD 10, the HMD 10 selects an expert's interface. As a result, virtual reality is provided through an interface suitable for each user, enabling comfortable operation by the user.

[0124] The HMD10 achieves an optimal interface and operability tailored to the user's skill level through the coordinated operation of its various components, including the eye-tracking control unit 203, the operation control unit 204, and the display control unit 206. As a result, the user experience is improved. Furthermore, because an optimal interface is provided according to the user's skill level, the learning curve for beginners is reduced, while the operational efficiency of experienced users is improved. In addition, since the sensitivity of eye-tracking is adjusted according to the user's skill level, the HMD10 achieves a more natural and intuitive operability. Moreover, the HMD10 disclosed in this application can be used in a variety of fields (e.g., education, medicine, entertainment, industry, retail, etc.), and can provide support functions specialized for each field.

[0125] Next, we will describe the hardware of each device that makes up the information processing system. Figure 11 shows an example of the hardware configuration of HMD10.

[0126] The HMD10 can be configured using an information processing device (a so-called computer), and has the configuration illustrated in Figure 11. For example, the HMD10 includes a processor 311, memory 312, input / output interface 313, and communication interface 314, etc. The components of the processor 311, etc., are connected by an internal bus or the like and are configured to communicate with each other.

[0127] However, the configuration shown in Figure 11 is not intended to limit the hardware configuration of the HMD10. The HMD10 may include hardware not shown, and it may not have to have an input / output interface 313 if necessary. Also, the number of processors 311 etc. included in the HMD10 is not intended to be limited to the example in Figure 11; for example, multiple processors 311 may be included in the HMD10.

[0128] The processor 311 is a programmable device such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or DSP (Digital Signal Processor). Alternatively, the processor 311 may be a device such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). The processor 311 executes various programs, including an operating system (OS).

[0129] Memory 312 includes RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. Memory 312 stores the OS program, application programs, and various data.

[0130] The input / output interface 313 is an interface for a display device or input device (not shown). The display device is, for example, a liquid crystal display. The input device is, for example, a device that accepts user input such as a keyboard or mouse.

[0131] The communication interface 314 is a circuit, module, etc., that communicates with other devices. For example, the communication interface 314 includes a NIC (Network Interface Card), etc.

[0132] The functions of the HMD10 are realized by various processing modules. These processing modules are realized, for example, by the processor 311 executing a program stored in memory 312. The program can also be recorded on a computer-readable storage medium. The storage medium can be a non-transitory material such as semiconductor memory, hard disk, magnetic recording medium, or optical recording medium. In other words, the present invention can also be embodied as a computer program product. Furthermore, the program can be downloaded via a network or updated using the storage medium on which the program is stored. Moreover, the processing module may be realized by a semiconductor chip.

[0133] The HMD10, an information processing device, is equipped with a computer, and its functions are realized by having the computer execute a program. Furthermore, the HMD10's control methods are executed by this program.

[0134] [Differentiation] The configuration and operation of the information processing system described in the above embodiment are illustrative examples and are not intended to limit the system configuration.

[0135] In the above embodiment, an HMD10 employing a video see-through method was used as an example. However, the HMD10 may also employ an optical see-through method for screen display.

[0136] In the flowcharts (sequence diagrams) used in the above description, multiple processes (processes) are shown in order, but the execution order of the processes performed in the embodiment is not limited to the order in which they are shown. In the embodiment, the order of the illustrated processes can be changed to the extent that it does not impair the content, for example, by executing each process in parallel.

[0137] The embodiments described above are explained in detail to facilitate understanding of the disclosure, and it is not intended that all the configurations described above are necessary. Furthermore, when multiple embodiments are described, each embodiment may be used individually or in combination. For example, it is possible to replace parts of the configuration of one embodiment with those of another embodiment, or to add configurations from other embodiments to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of one embodiment with those of another.

[0138] As described above, the industrial applicability of the present invention is clear, and it is particularly suitable for use in head-mounted displays and the like that which seamlessly integrate the real world with virtual digital content.

[0139] The present invention provides not merely an abstract idea or software implementation, but a technical means to solve a specific technical problem. In particular, the present invention improves the operability of existing AR (Augmented Reality) / VR (Virtual Reality) devices by combining specific technical features such as authentication using the user's biometric information, eye tracking, and interface adjustment according to skill level. As a result, the user experience is improved. A specific example of its application is in the medical field. For instance, by applying the present invention to a surgical training system, the level of detail and difficulty of surgical simulations can be automatically adjusted according to the skill level of the resident physician. This effectively supports the improvement of the resident physician's skills and brings about a concrete and useful result that contributes to improving the quality of medical care. Thus, the present invention goes beyond an abstract idea, solving real-world problems using specific technical means and producing useful and concrete results, and is therefore patent-eligible.

[0140] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0141] [Note 1] A means for acquiring biometric information of a user, Authentication means for authenticating the user using the acquired biometric information, A history control means that generates a usage history including information about the operations of the authenticated user, and stores the generated usage history, A history acquisition means for acquiring the stored usage history, A display control means that determines the interface on the virtual reality or augmented reality screen based on the acquired usage history, A head-mounted display equipped with [specific features / features].

[0142] [Note 2] The system further includes a gaze control means for estimating the direction of the user's gaze, The display control means is Based on the estimated gaze direction, the system identifies the object that the user is viewing among the multiple objects displayed on the virtual reality or augmented reality screen. A head-mounted display according to Appendix 1, which displays the identified object in a display format corresponding to the determined interface.

[0143] [Note 3] The head-mounted display according to Appendix 2, wherein the display control means calculates the user's proficiency in the virtual reality or augmented reality based on the acquired usage history, and determines the interface based on the calculated proficiency.

[0144] [Note 4] The display control means provides visual feedback to the identified object based on the calculated proficiency level, as described in Appendix 3, for the head-mounted display.

[0145] [Note 5] The head-mounted display described in Appendix 4, wherein the display control means adjusts the sensitivity for identifying the object being viewed by the user based on the calculated proficiency level.

[0146] [Note 6] The display control means provides the visual feedback to the identified object based on the setting value selected by the user, as described in Appendix 5 of the head-mounted display.

[0147] [Note 7] The history control means stores the generated usage history on a server in the cloud. The head-mounted display described in any one of the appendices 1 to 6, wherein the history acquisition means acquires usage history, including usage history of other head-mounted displays, from the server on the cloud.

[0148] [Note 8] A head-mounted display according to any one of the appendices 3 to 6, further comprising means for providing a tutorial on a function or setting to the user, the tutorial being tailored to the calculated level of proficiency.

[0149] [Note 9] The process involves acquiring the user's biometric information, and An authentication step is performed to authenticate the user using the acquired biometric information, A history control step includes generating a usage history that includes information about the operations of the authenticated user, and storing the generated usage history. A history acquisition step to acquire the stored usage history, A display control step, which determines the interface on the virtual reality or augmented reality screen based on the acquired usage history, A method for controlling a head-mounted display, comprising the following features.

[0150] [Note 10] The system further includes a gaze control step for estimating the user's gaze direction, The display control step is as follows: Based on the estimated gaze direction, the system identifies the object that the user is viewing among the multiple objects displayed on the virtual reality or augmented reality screen. A method for controlling a head-mounted display according to Appendix 9, which displays the identified object in a display format corresponding to the determined interface.

[0151] [Note 11] The control method for a head-mounted display according to Appendix 10, wherein the display control step calculates the user's proficiency in the virtual reality or augmented reality based on the acquired usage history, and determines the interface based on the calculated proficiency.

[0152] [Note 12] The display control step is a method for controlling a head-mounted display according to Appendix 11, wherein the display control step provides visual feedback to the identified object based on the calculated proficiency level.

[0153] [Note 13] The display control step is a method for controlling a head-mounted display as described in Appendix 12, wherein the sensitivity for identifying an object being viewed by the user is adjusted based on the calculated proficiency level.

[0154] [Note 14] The display control step is a method for controlling a head-mounted display according to Appendix 13, wherein the display control step provides the visual feedback to the identified object based on the setting value selected by the user.

[0155] [Note 15] The history control step involves storing the generated usage history on a server in the cloud. The head-mounted display control method according to any one of the appendices 9 to 14, wherein the history acquisition step acquires usage history, including usage history of other head-mounted displays, from the server on the cloud.

[0156] [Note 16] A method for controlling a head-mounted display according to any one of the appendices 11 to 14, further comprising a provision step of providing a tutorial on a function or setting to the user, the tutorial being tailored to the calculated level of proficiency.

[0157] [Note 17] The computer mounted in the head-mounted display, The process involves obtaining the user's biometric information, and Authentication process for authenticating the user using the acquired biometric information, A history control process that generates a usage history including information about the operations of the authenticated user, and stores the generated usage history, A history acquisition process to acquire the stored usage history, Based on the acquired usage history, a display control process determines the interface on the virtual reality or augmented reality screen. A program to execute.

[0158] [Note 18] Further, a gaze control process is performed to estimate the user's gaze direction. The aforementioned display control process is: Based on the estimated gaze direction, the system identifies the object that the user is viewing among the multiple objects displayed on the virtual reality or augmented reality screen. The program described in Appendix 17, which displays the identified object in a display format corresponding to the determined interface.

[0159] [Note 19] The display control process is a program as described in Appendix 18, which calculates the user's proficiency in the virtual reality or augmented reality based on the acquired usage history, and determines the interface based on the calculated proficiency.

[0160] [Note 20] The display control process is the program described in Appendix 19, which provides visual feedback to the identified object based on the calculated proficiency level.

[0161] [Note 21] The display control process is the program described in Appendix 20, which adjusts the sensitivity for identifying the object being viewed by the user based on the calculated proficiency level.

[0162] [Note 22] The display control process is the program described in Appendix 21, which provides the visual feedback to the identified object based on the setting value selected by the user.

[0163] [Note 23] The aforementioned history control process stores the generated usage history on a server in the cloud. The aforementioned history acquisition process is a program described in any one of the items in Appendix 17 to 22, which acquires usage history, including the usage history of other head-mounted displays, from the server on the cloud.

[0164] [Note 24] A program according to any one of the items in Appendix 19 to 22, which provides the user with a tutorial relating to a function or setting, and further causes the user to perform a provisioning process.

[0165] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 9 and 17 in the same way as those described in Appendices 2 to 8. Moreover, not limited to Appendices 1, 9 and 17, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0166] Furthermore, each disclosure of the above-mentioned prior art documents cited herein is incorporated herein by reference. Although embodiments of the present invention have been described above, the present invention is not limited to these embodiments. It will be understood by those skilled in the art that these embodiments are merely illustrative and that various modifications are possible without departing from the scope and spirit of the present invention. That is, the present invention naturally includes the entire disclosure, including the claims, and various modifications and alterations that can be made by those skilled in the art in accordance with the technical idea. [Explanation of Symbols]

[0167] 10 HMD 31 Plants 32 TV 41 Icons 42 icons 43 icons 44 icons 45 icons 46 icons 100 HMD 101 Means for acquiring biometric information 102 Authentication methods 103 History control means 104 Means of obtaining history 105 Display control means 201 Mounting Control Unit 202 Authentication Control Unit 203 Eye-line control unit 204 Operation Control Unit 205 Usage History Control Unit 206 Display Control Unit 207 Memory section 208 Commentary Department 311 Processors 312 memory 313 Input / Output Interfaces 314 Communication Interface

Claims

1. A means for acquiring biometric information of a user, Authentication means for authenticating the user using the acquired biometric information, A history control means that generates a usage history including information about the operations of the authenticated user, and stores the generated usage history, A history acquisition means for acquiring the stored usage history, A display control means that determines the interface on the virtual reality or augmented reality screen based on the acquired usage history, A head-mounted display equipped with [specific features / features].

2. The system further includes a gaze control means for estimating the direction of the user's gaze, The display control means is Based on the estimated gaze direction, the system identifies the object that the user is viewing among the multiple objects displayed on the virtual reality or augmented reality screen. The head-mounted display according to claim 1, which displays the identified object in a display mode corresponding to the determined interface.

3. The head-mounted display according to claim 2, wherein the display control means calculates the user's proficiency in the virtual reality or augmented reality based on the acquired usage history, and determines the interface based on the calculated proficiency.

4. The head-mounted display according to claim 3, wherein the display control means provides visual feedback to the identified object based on the calculated proficiency level.

5. The head-mounted display according to claim 4, wherein the display control means adjusts the sensitivity for identifying the object being viewed by the user based on the calculated proficiency level.

6. The head-mounted display according to claim 5, wherein the display control means provides the visual feedback to the identified object based on a setting value selected by the user.

7. The history control means stores the generated usage history on a server in the cloud. The head-mounted display according to any one of claims 1 to 6, wherein the history acquisition means acquires usage history, including usage history of other head-mounted displays, from the server on the cloud.

8. The head-mounted display according to any one of claims 3 to 6, further comprising means for providing a tutorial on a function or setting to the user, the tutorial being tailored to the calculated level of proficiency.

9. The process involves acquiring the user's biometric information, and An authentication step is performed to authenticate the user using the acquired biometric information, A history control step includes generating a usage history that includes information about the operations of the authenticated user, and storing the generated usage history. A history acquisition step to acquire the stored usage history, A display control step, which determines the interface on the virtual reality or augmented reality screen based on the acquired usage history, A method for controlling a head-mounted display, comprising the following features.

10. The computer mounted in the head-mounted display, The process involves obtaining the user's biometric information, and Authentication process for authenticating the user using the acquired biometric information, A history control process that generates a usage history including information about the operations of the authenticated user, and stores the generated usage history, A history acquisition process to acquire the stored usage history, Based on the acquired usage history, a display control process determines the interface on the virtual reality or augmented reality screen. A program to execute.

Images