Information processing device, method, and program

The information processing device detects human interventions and presents them within the VR space, addressing the risk of missing external interactions for users wearing HMDs, allowing for immediate responses and enhanced safety during VR experiences.

WO2024228329A1PCT designated stage expired Publication Date: 2024-11-07SONY GROUP CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/014972
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-01
Filing Date
2024-04-15
Publication Date
2024-11-07

AI Technical Summary

Technical Problem

Users wearing Head Mounted Displays (HMDs) for Virtual Reality (VR) experiences are at risk due to blocked external sight and sound, making them unaware of surrounding obstacles and human interventions, which existing technologies do not adequately address, especially regarding human intervention detection and prompt response support.

Method used

An information processing device and method that includes a detection unit to identify intervening persons and a presentation control unit to provide notifications or presentations to the user, allowing for immediate awareness of human interventions while maintaining the VR experience, using a system with a camera and microphone to detect and recognize interventions and present information within the VR space.

Benefits of technology

Enables users to promptly respond to human interventions without leaving the VR environment, reducing the risk of missing important interactions or collisions by providing real-time awareness of external interventions within the VR space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024014972_07112024_PF_FP_ABST
    Figure JP2024014972_07112024_PF_FP_ABST
Patent Text Reader

Abstract

The present technology relates to an information processing device, method, and program that make it possible to support a quick response of a user and reduce risk. The information processing device detects an intervening person from the outside with respect to a user wearing a device which blocks the external field of view and sound, and performs control to present information on the detected intervening person to the user. The present technology can be applied to an ambient computing system.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, method, and program

[0001] The present technology relates to an information processing device, a method, and a program, and more particularly to an information processing device, a method, and a program that can support a user's quick response and reduce risks.

[0002] With the advancement of high-performance HMDs (head-mounted displays) and advances in VR (virtual reality) content generation technology, experiencing (viewing) VR spaces and viewing VR content has become common. In these cases, users generally wear an HMD, but because wearing an HMD blocks out surrounding vision and sound, they often fail to notice surrounding obstacles or human intervention (such as calls).

[0003] Regarding surrounding obstacles, for example, Patent Document 1 proposes a technology that detects real obstacles around a user, replaces the obstacles with virtual objects, and displays them on the screen of an HMD. However, Patent Document 1 does not describe the detection of human intervention.

[0004] Japanese Patent Application Laid-Open No. 2013-257716

[0005] The risk of not noticing human intervention while wearing an HMD while viewing VR content is expected to continue to increase. Even if a user does notice human intervention, they will need to take off the HMD or leave the VR space to respond to communication, etc.

[0006] The present technology was developed in light of these circumstances, and is intended to support users in taking prompt action and reduce risks.

[0007] An information processing device according to one aspect of the present technology includes a detection unit that detects an external intervener who is wearing a user device that blocks external vision and sound, and a presentation control unit that controls the presentation of information about the detected intervener to the user.

[0008] In one aspect of the present technology, an external intervener is detected for a user wearing a user device that blocks external vision and sound, and control is performed to present information about the detected intervener to the user.

[0009] 17 is a block diagram showing an example of the configuration of an ambient computing system according to an embodiment of the present technology. FIG. 18 is a diagram showing an overview of an HMD system according to a first embodiment of the present technology. FIG. 19 is a block diagram showing an example of the configuration of the HMD system of FIG. 2. FIG. 20 is a block diagram showing an example of the configuration of the detection unit of FIG. 3. FIG. 21 is a flowchart explaining processing of the HMD system of FIG. 2. FIG. 22 is a flowchart explaining detection processing of step S11 of FIG. 5. FIG. 23 is a diagram showing a modified example of the HMD system of FIG. 2. FIG. 24 is a flowchart explaining other processing of the HMD system of FIG. 2. FIG. 8 is a flowchart explaining detection processing of step S51 of FIG. 8. FIG. 8 is a flowchart explaining another example of the detection processing of step S51 of FIG. 8. FIG. 19 is a diagram showing an overview of an HMD system according to a second embodiment of the present technology. FIG. 11 is a block diagram showing an example of the configuration of the HMD system of FIG. 11. FIG. 12 is a flowchart explaining processing of the HMD system of FIG. 11. FIG. 12 is a block diagram showing an example of the configuration of an HMD system according to a third embodiment of the present technology. FIG. 19 is a diagram showing an overview of an HMD system according to a fourth embodiment of the present technology. FIG. 15 is a block diagram showing an example of the configuration of the HMD system of FIG. 15. FIG. 16 is a block diagram showing an example of the configuration of an HMD system according to a fifth embodiment of the present technology. FIG. 17 is a flowchart explaining processing of the HMD system of FIG. 17. 26 is a flowchart illustrating another process of the HMD system of FIG. 17. FIG. 27 is a block diagram illustrating a configuration example of an HMD system according to a sixth embodiment of the present technology. FIG. 28 is a flowchart illustrating the process of the HMD system of FIG. 22. FIG. 29 is a diagram illustrating a method of presenting a message to a person speaking. FIG. 30 is a diagram illustrating an overview of an HMD system according to a seventh embodiment of the present technology. FIG. 31 is a diagram illustrating an overview of an HMD system according to an eighth embodiment of the present technology. FIG. 32 is a flowchart illustrating the process of the HMD system of FIG. 26. FIG. 33 is a flowchart illustrating a first control process according to an intervention level by an HMD system according to a ninth embodiment of the present technology. FIG. 34 is a diagram illustrating a control method according to whether the user and the interventionist are an adult or a child. FIG. 35 is a flowchart illustrating a second control process according to an intervention level by an HMD system according to the ninth embodiment of the present technology.It is a figure which shows an example of a control method determination table. It is a figure which shows an example of a VR space which a user views or experiences. It is a block diagram which shows an example of the configuration of a computer.

[0010] Hereinafter, embodiments for implementing the present technology will be described. The description will be made in the following order: 1. Overall system 2. First embodiment of the present technology (notification of intervention detection) 3. Second embodiment of the present technology (superimposed presentation in VR space) 4. Third embodiment of the present technology (notification after person recognition) 5. Fourth embodiment of the present technology (superimposed presentation of avatar in VR space) 6. Fifth embodiment of the present technology (determination of response according to user selection) 7. Sixth embodiment of the present technology (determination of response by system) 8. Seventh embodiment of the present technology (intervention and obstacle detection) 9. Eighth embodiment of the present technology (sharing of VR space) 10. Ninth embodiment of the present technology (control according to intervention level) 11. Others

[0011] <1. Overall System> <System Configuration> FIG. 1 is a diagram showing an example of the configuration of an ambient computing system according to an embodiment of the present technology.

[0012] The ambient computing system 1 in FIG. 1 is a system that detects information about the user's situation and surrounding environment, controls each device based on the detected information and prior information, and provides the services desired by the user.

[0013] In providing such services, the ambient computing system 1 provides, as a function related to the present technology, a service that detects human intervention while a user is viewing VR content (hereinafter referred to as "viewing VR"), controls an HMD in response to the detected human intervention, and presents information.

[0014] The ambient computing system 1 is configured to include a user interface 21, an environmental prior information acquisition unit 22, a user prior information acquisition unit 23, a collaborative sensing unit 24, a surrounding environment detection unit 25, a user state detection unit 26, a user response detection unit 27, a context management unit 28, an equipment control unit 29, an environmental context model memory unit 30, a user context model memory unit 31, and a service list memory unit 32.

[0015] In this specification, the term "context" refers to information including user information, environmental information, and the relationship between these pieces of information. The term "context model" refers to a model used by the context management unit 28 when learning, and is also a model (i.e., an algorithm or machine learning model) used by the context management unit 28 to build personalized services for users.

[0016] The user interface 21 performs interface functions for the user, such as sending notifications to the user and receiving instructions from the user.

[0017] The ambient computing system 1 has two paths for acquiring information.

[0018] The first path is a path for acquiring prior information in advance based on user input, and is realized by the prior environment information acquisition unit 22 and the prior user information acquisition unit 23.

[0019] The environmental prior information acquisition unit 22 acquires prior information about the environment based on a user input, and outputs the acquired environmental prior information to the context management unit 28 as environmental information.

[0020] As a function related to the present technology, in particular, the environmental prior information acquisition unit 22 outputs, for example, information from a recognition database for recognizing the face and voice of a person intervening (hereinafter also referred to as the intervener) to the context management unit 28 as environmental information.

[0021] The user prior information acquisition unit 23 acquires prior information about the user based on the user input, and outputs the acquired user prior information to the context management unit 28 as user information.

[0022] As a function related to the present technology, in particular, the user advance information acquisition unit 23 outputs, for example, information indicating the type of viewing content being viewed by the user to the context management unit 28 as user information.

[0023] The second path is a path that detects the environment and the user's current state information (in this technology, the state that the user is watching VR) through a recognizer that performs sensing by coordinating multiple sensing devices such as IoT (Internet of Things) and handles the sensing information, and is also capable of multimodal processing. Multimodal processing refers to processing multiple types of information at once. The second path is realized by a cooperative sensing unit 24 as a recognizer, a surrounding environment detection unit 25, a user state detection unit 26, and a user response detection unit 27.

[0024] The collaborative sensing unit 24 acquires sensing information from multiple sensing devices and performs predetermined signal processing on the acquired information to make it easier to handle in subsequent processing, such as removing noise components and performing data cleansing to synchronize data. The collaborative sensing unit 24 outputs the signal-processed information to the detection unit corresponding to the information, among the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27.

[0025] The surrounding environment detection unit 25 detects surrounding information such as images, temperature, and sound from the information supplied by the collaborative sensing unit 24, and outputs information regarding the environment obtained as a result of the detection (hereinafter referred to as environmental information) to the context management unit 28.

[0026] As a function related to the present technology, the surrounding environment detection unit 25 detects human intervention, such as a person looking, approaching, or speaking, and outputs the resulting environmental information to the context management unit 28.

[0027] The user state detection unit 26 detects information about the user from the information supplied from the collaborative sensing unit 24. For example, the user state detection unit 26 performs individual user identification by facial recognition using RGB images, detects the user's behavior from skeletal structure detection and environmental map information, and detects the user's emotions, such as comfort or discomfort, by multimodally processing facial expressions acquired from an RGB camera and vital information, such as body temperature, acquired from a wearable device.

[0028] The user state detection unit 26 outputs information about the user obtained as a result of the detection (hereinafter referred to as user information) to the context management unit 28 .

[0029] As a function related to the present technology, in particular, the user state detection unit 26 detects, for example, that the user is watching VR, and outputs the user information obtained as a result to the context management unit 28.

[0030] The user response detection unit 27 detects the user's response to the service provided by the context management unit 28 from the information supplied from the collaborative sensing unit 24, and outputs the user's response information and feedback information obtained as a result of the detection to the context management unit 28.

[0031] As a function related to the present technology, in particular, the user response detection unit 27 detects, for example, user permission or selection for services such as notifications provided by the context management unit 28, and outputs the resulting user response information and feedback information to the context management unit 28.

[0032] The context management unit 28 learns an environmental context model, such as an environment that is likely to be created in a certain time period and in a certain area, by accumulating information supplied from the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27. The context management unit 28 stores the learned environmental context model in the environmental context model storage unit 30.

[0033] The context management unit 28 learns a model relating to the user's habitual behavioral patterns, such as what activity the user is performing in what situation, as a user context model by accumulating information supplied from the surrounding environment detection unit 25, the user state detection unit 26, and the user response detection unit 27. The context management unit 28 stores the learned user context model in the user context model storage unit 31.

[0034] The context management unit 28 learns each context model, and when information indicating a similar situation is obtained, it controls the device control unit 29 to provide personalized services that use the user context model to anticipate actions for the user from devices that are linked to the ambient computing system 1.

[0035] The context management unit 28 receives user responses when a personalized service is provided via the user response detection unit 27, thereby obtaining feedback information on whether the service was actually appropriate, and re-learning each context model to improve the accuracy of each context model.

[0036] Furthermore, by repeatedly learning context models of habitual behavioral patterns, the context management unit 28 can also model behavioral norms that indicate which options a user is likely to choose in certain situations, making it possible to provide a personalized service to the user in advance, even if the user does not habitually do something.

[0037] As a function related to this technology, in particular, the context management unit 28 controls the device control unit 29 to provide personalized services such as notifications, presentations, and inquiries to the user from the HMD, which is one of the devices that the ambient computing system 1 is linked to.

[0038] Under the control of the context management unit 28, the device control unit 29 causes a device (in the case of the present technology, an HMD) linked to the ambient computing system 1 to provide personalized services.

[0039] The environmental context model storage unit 30 stores the environmental context model. The environmental context model storage unit 30 also stores, as environmental context, information used when learning the environmental context model and information including the relationships between the information.

[0040] The user context model storage unit 31 stores a user context model, and also stores information used when learning the user context model and information including the relationship between the information and the user context as user context.

[0041] The service list storage unit 32 stores a list of devices that perform services that can be provided to users.

[0042] Hereinafter, the functions of the present technology in the ambient computing system 1 of FIG. 1 will be described in detail.

[0043] 2. First Embodiment of the Present Technology (Notification of Intervention Detection) Overview of HMD System FIG. 2 is a diagram illustrating an overview of an HMD system according to a first embodiment of the present technology.

[0044] FIG. 2 shows the state of a user using an HMD system 51 having the functions of the present technology, which is included in the ambient computing system 1 of FIG.

[0045] For example, a different person (an intervener, a child in FIG. 2) speaks to a user wearing the HMD 61 while watching VR, saying, "Hey, Dad."

[0046] The HMD system 51 detects that a child is speaking to the user using a camera 62 and a microphone 71 (see FIG. 3, which will be described later) provided in the HMD 61, and notifies the user of the detection result by displaying it on the HMD 61.

[0047] This allows the user to immediately notice that their child is speaking to them, even if they are immersed in the VR space.

[0048] <Configuration of HMD System> FIG. 3 is a block diagram showing an example of the configuration of the HMD system 51 in FIG.

[0049] The HMD system 51 is configured to include the camera 62, microphone 71, detection unit 72, notification information generation unit 73, and information presentation control unit 74 shown in FIG.

[0050] The camera 62, microphone 71, and detection unit 72 correspond to the collaborative sensing unit 24, surrounding environment detection unit 25, user state detection unit 26, and user response detection unit 27 in Fig. 1. The notification information generation unit 73 corresponds to the context management unit 28 in Fig. 1. The information presentation control unit 74 corresponds to the device control unit 29 in Fig. 1.

[0051] The camera 62 is an imaging unit that captures images of the surroundings, and is provided in the HMD 61 worn by the user. The camera 62 outputs video data generated by capturing images of the surroundings of the user to the detection unit 72.

[0052] The microphone 71 is a sound collection unit and is provided in the HMD 61 worn by the user. The microphone 71 collects sounds around the user and generates sound data, which is output to the detection unit 72.

[0053] The detection unit 72 detects external human intervention (presence of an intervening person) based on the video data supplied from the camera 62 and the audio data supplied from the microphone 71, and supplies the detection result to the notification information generation unit 73. Human intervention can be, for example, an audio intervention indicating that someone is speaking to you, an eye intervention indicating that someone is looking at you, or an action intervention indicating that someone is approaching you.

[0054] The notification information generating unit 73 generates notification information for notifying the detection result supplied from the detecting unit 72 , and outputs the generated notification information to the information presentation control unit 74 .

[0055] The information presentation control unit 74 causes a presentation unit (not shown) provided in the HMD 61 to present the notification information supplied from the notification information generation unit 73 .

[0056] In this case, the notification may be made by "displaying on the video the user is watching" or "notifying by audio", but either method is acceptable. Therefore, the notification information generated by the notification information generating unit 73 is video data when notifying by video, and audio data when notifying by audio.

[0057] <Configuration of Detector> FIG. 4 is a block diagram showing an example of the configuration of the detector 72 in FIG.

[0058] 4, the detection unit 72 is configured to include a person detection unit 81 , a movement determination unit 82 , a voice activity detection unit 83 , a speaker recognition unit 84 , a speaker data DB (Database) 85 , and an intervention detection unit 86 .

[0059] The person detection unit 81 detects people from the video data supplied from the camera 62. The person detection unit 81 outputs information about the detected people to the movement determination unit .

[0060] The movement determination unit 82 determines the face direction from the person information supplied from the person detection unit 81, and outputs the determination result of the face direction to the intervention detection unit 86. Furthermore, the movement determination unit 82 determines whether the size of the person in the video is increasing or not from the person information supplied from the person detection unit 81, and outputs the determination result of whether the size of the person is increasing or not to the intervention detection unit 86.

[0061] The voice interval detection unit 83 detects a voice interval from the audio data supplied from the microphone 71. The voice interval detection unit 83 outputs information indicating the detected voice interval to the speaker recognition unit .

[0062] The speaker recognition unit 84 determines whether the person speaking (hereinafter also referred to as the speaker) is a person other than the user, based on the voice data of the speech section indicated by the information supplied from the speech section detection unit 83 and the speaker data registered in the speaker data DB 85. The speaker recognition unit 84 outputs information on the determined speaker to the intervention detection unit 86.

[0063] The intervention detection unit 86 detects human intervention, such as someone speaking to the user, someone looking at the user, or someone approaching the user, based on the face direction determination result supplied from the movement determination unit 82 and speaker information supplied from the speaker recognition unit 84. The intervention detection unit 86 outputs the human intervention detection result to the notification information generation unit 73.

[0064] <HMD System Processing> Fig. 5 is a flowchart illustrating the processing of the HMD system 51 in Fig. 2. The processing in Fig. 5 is performed for each frame or for each frame at a predetermined interval. Note that other processing of the HMD system described below is also performed for each frame or for each frame at a predetermined interval.

[0065] For example, a child may speak to a user wearing the HMD 61 and viewing VR, saying, "Hey, Dad."

[0066] In step S11, the camera 62, the microphone 71, and the detection unit 72 detect that a person has spoken to them.

[0067] That is, the camera 62 and microphone 71 provided in the HMD 61 sense the situation around the user and output video data and audio data, respectively, to the detection unit 72. The detection unit 72 detects that a person is speaking to the user based on the video data supplied from the camera 62 and the audio data supplied from the microphone 71, and supplies the detection result to the notification information generation unit 73. Details of the detection process in step S11 will be described later with reference to FIG.

[0068] In step S12, the notification information generation unit 73 generates notification information to notify the user that a person has spoken to them, which is the detection result supplied from the detection unit 72, and outputs the generated notification information to the information presentation control unit 74.

[0069] In step S13, the information presentation control unit 74 notifies the user that a person is speaking to them by displaying the notification information supplied from the notification information generation unit 73 on a presentation unit provided in the HMD 61. Then, the process ends.

[0070] <Detection Process> FIG. 6 is a flowchart illustrating the details of the detection process in step S11 of FIG.

[0071] In step S31, the camera 52 and the microphone 71 sense the surrounding conditions. The camera 62 captures an image of the user's surroundings and generates video data, which is output to the detection unit 72. The microphone 71 collects sounds around the user and generates audio data, which is output to the detection unit 72.

[0072] In step S32, the person detection unit 81 determines whether or not a person has been detected in the image of the video data supplied from the camera 62. If it is determined in step S32 that a person has not been detected in the image, the process returns to step S31, and the subsequent processes are repeated.

[0073] If it is determined in step S32 that a person has been detected in the video, the process proceeds to step S33.

[0074] In step S33, the movement determination unit 82 determines whether the face of the person in the video is facing the user, based on the person information supplied from the person detection unit 81. If it is determined in step S33 that the face of the person in the video is not facing the user, the process returns to step S31, and the subsequent processes are repeated.

[0075] If it is determined in step S33 that the face of the person in the video is facing the user, the process proceeds to step S34.

[0076] The vocalization section detector 83 detects a vocalization section from the collected sound based on the audio data supplied from the microphone 71 .

[0077] In step S34, the vocalization section detection unit 83 determines whether or not a human voice has been detected from the collected sound, based on the audio data supplied from the microphone 71. If it is determined in step S34 that a human voice has not been detected from the collected sound, the process returns to step S31, and the subsequent processes are repeated.

[0078] If it is determined in step S34 that a human voice has been detected from the collected sound, the process proceeds to step S35.

[0079] In step S35, the speaker recognition unit 84 determines whether the speaker is a person other than the user based on the voice data of the speech interval indicated by the information supplied from the speech interval detection unit 83 and the speaker data registered in the speaker data DB 85. If it is determined that the speaker is the user, the process returns to step S31, and the subsequent processes are repeated.

[0080] If it is determined in step S35 that the speaker is a person other than the user, the process proceeds to step S36.

[0081] In step S36, the intervention detection unit 86 determines that "a person has spoken to the user" and outputs a detection result indicating that a person has spoken to the user. After that, the process returns to step S11 in FIG. 5.

[0082] In the above explanation, an example of sensing using the camera 62 and microphone 71 provided on the HMD 61 was described, but if VR viewing is performed in a specific location, sensing may also be performed using an external camera and microphone.

[0083] <Modification of HMD System> FIG. 7 is a diagram showing a modification of the HMD system of FIG.

[0084] FIG. 7 shows an example in which a user using the HMD system 51 is watching VR in a specific place such as a room, and is approached by a child.

[0085] The HMD system 51 in FIG. 7 differs from the HMD system 51 in FIG. 2 in that the camera 62 and microphone 71 are replaced with a camera 91 and microphone 92.

[0086] The camera 91 and microphone 92 are not provided in the HMD 61 but are provided outside the room, for example.

[0087] That is, the camera 91 is configured similarly to the camera 62 except for its installation location. The microphone 92 is configured similarly to the microphone 71 except for its installation location.

[0088] As described above, when the user watches VR in a fixed location, sensing may be performed using an external camera 91 and microphone 92.

[0089] Although the example of detecting when a person is speaking to the device has been described with reference to FIG. 5, it is also possible to detect when a person is looking at the device or when a person is approaching the device.

[0090] <Other Processing of HMD System> FIG. 8 is a flowchart illustrating other processing of the HMD system 51 of FIG.

[0091] For example, a person approaches a user wearing the HMD 61 while watching VR, or a person looks at the user.

[0092] In step S51, the camera 62, microphone 71, and detection unit 72 detect that a person is approaching or that a person is looking at the user.

[0093] That is, the camera 62 and microphone 71 provided in the HMD 61 sense the situation around the user and output video data and audio data, respectively, to the detection unit 72. The detection unit 72 detects that a person is approaching or that a person is looking at the user based on the video data supplied from the camera 62 and the audio data supplied from the microphone 71, and supplies the detection result to the notification information generation unit 73. Details of the detection process in step S51 will be described later with reference to FIGS. 9 and 10.

[0094] In step S52, the notification information generation unit 73 generates notification information that notifies the user that a person is approaching or that a person is looking at the user, which is the detection result supplied from the detection unit 72, and outputs the generated notification information to the information presentation control unit 74.

[0095] In step S53, the information presentation control unit 74 notifies the user that a person is approaching or that a person is looking at the user by displaying the notification information supplied from the notification information generation unit 73 on a presentation unit provided in the HMD 61. Then, the processing ends.

[0096] <Detection Process> FIG. 9 is a flowchart illustrating the detection process in step S51 of FIG.

[0097] FIG. 9 shows the process of detecting an approaching person.

[0098] In step S71, the camera 62 and the microphone 71 sense the surrounding situation. The camera 62 captures an image of the user's surroundings and generates video data, which is output to the detection unit 72. The microphone 71 collects sounds around the user and generates audio data, which is output to the detection unit 72.

[0099] In step S72, the person detection unit 81 determines whether or not a person has been detected in the image of the video data supplied from the camera 62. If it is determined in step S72 that a person has not been detected in the image, the process returns to step S71, and the subsequent processes are repeated.

[0100] If it is determined in step S72 that a person has been detected in the video, the process proceeds to step S73.

[0101] In step S73, the movement determination unit 82 determines whether or not the face is facing the user based on the person information supplied from the person detection unit 81. If it is determined in step S73 based on the person information that the face is not facing the user, the process returns to step S71, and the subsequent processes are repeated.

[0102] If it is determined in step S73 that the face is facing the user, the process proceeds to step S74.

[0103] In step S74, the movement determination unit 82 determines whether or not the size of the person in the video has increased based on the person information supplied from the person detection unit 81. If it is determined in step S74 that the size of the person in the video has not increased, the process returns to step S71, and the subsequent processes are repeated.

[0104] If it is determined in step S74 that the size of the person in the video has increased, the process proceeds to step S75.

[0105] In step S75, the intervention detection unit 86 determines that "a person is approaching" and outputs a detection result indicating that it has detected that a person is approaching. Thereafter, the process returns to step S51 in FIG. 8.

[0106] <Detection Process> FIG. 10 is a flowchart illustrating another example of the detection process of step S51 in FIG.

[0107] In FIG. 10, the process of detecting when a person is looking at the user is shown.

[0108] In step S91, the camera 52 and the microphone 71 sense the surrounding conditions. The camera 62 captures an image of the user's surroundings and outputs the generated video data to the detection unit 72. The microphone 71 collects sounds around the user and outputs the generated audio data to the detection unit 72.

[0109] In step S92, the person detection unit 81 determines whether or not a person has been detected in the image of the video data supplied from the camera 62. If it is determined in step S92 that a person has not been detected in the image, the process returns to step S91, and the subsequent processes are repeated.

[0110] If it is determined in step S92 that a person has been detected in the video, the process proceeds to step S93.

[0111] In step S93, the movement determination unit 82 determines whether the person's face is facing the user based on the person information supplied from the person detection unit 81. If it is determined in step S93 based on the person information that the person's face is not facing the user, the process returns to step S91, and the subsequent processes are repeated.

[0112] If it is determined in step S93 that the face is facing the user, the process proceeds to step S94.

[0113] In step S94, the intervention detection unit 86 determines that "a person is looking in the direction" and outputs a detection result indicating that it has detected that a person is looking in the direction. Thereafter, the process returns to step S51 in FIG. 8.

[0114] 9 and 10, since audio data is not used, the HMD system 51 may be configured without the microphone 71.

[0115] 3. Second Embodiment of the Present Technology (Superimposed Presentation in VR Space) Overview of HMD System FIG. 11 is a diagram illustrating an overview of an HMD system according to a second embodiment of the present technology.

[0116] FIG. 11 shows the state of a user using an HMD system 101 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.

[0117] For example, while a user is wearing the HMD 61 and watching VR, another person (an intervener, a child in FIG. 11) speaks to the user from outside, saying, "Hey, Dad."

[0118] The HMD system 101 detects when a child is speaking to the user using a camera 62 and a microphone 71 (Figure 12) provided in the HMD 61, and not only notifies the user of the detection result, but also projects the acquired video data and audio data into a VR space 111 displayed on the presentation unit of the HMD 61 and presents it within the VR space 111.

[0119] As described above, in the HMD system 101 of Fig. 11, the video and audio of the person speaking to the user are superimposed on the data of the VR space 111 being viewed by the user, thereby enabling the user to continue to converse with the person speaking to the user without leaving the VR space 111.

[0120] <Configuration of HMD System> FIG. 12 is a block diagram showing an example of the configuration of the HMD system 101 in FIG.

[0121] The HMD system 101 differs from the HMD system 51 in Fig. 3 in that the notification information generation unit 73 is replaced with a notification information superimposition unit 131. The notification information superimposition unit 131 corresponds to the context management unit 28 in Fig. 1 .

[0122] The detection unit 72 supplies the video data supplied from the camera 62, the audio data supplied from the microphone 71, and the detection result of the intervention to the notification information superimposing unit 131.

[0123] Based on the video data, audio data, and detection results supplied from the detection unit 72, the notification information superimposing unit 131 superimposes the video data of the intervener on the video data of the VR space 111, and superimposes (combines) the audio data on the audio data of the VR space 111, and outputs the superimposed data (referred to as superimposed data) to the information presentation control unit 74. Note that the notification information superimposing unit 131 may generate notification information notifying the detection results of the intervention supplied from the detection unit 72, and may also superimpose the generated notification information on the data of the VR space 111.

[0124] The information presentation control unit 74 displays the superimposition data supplied from the notification information superimposition unit 131 in the VR space 111 of the presentation unit provided in the HMD 61 .

[0125] <Processing of HMD System> FIG. 13 is a flowchart illustrating the processing of the HMD system 101 in FIG.

[0126] For example, a child may speak to a user wearing the HMD 61 and viewing VR, saying, "Hey, Dad."

[0127] In step S111, the camera 62, microphone 71, and detection unit 72 detect that a person has spoken to them, as described above with reference to FIG.

[0128] That is, the camera 62 and microphone 71 provided in the HMD 61 sense the situation around the user and output video data and audio data, respectively, to the detection unit 72. The detection unit 72 detects that a person is speaking to the user based on the video data supplied from the camera 62 and the audio data supplied from the microphone 71, and supplies the video data, audio data, and detection result to the notification information superimposition unit 131.

[0129] In step S112, the notification information superimposition unit 131 superimposes the video data and audio data of the intervener on the data of the VR space 111 based on the detection results supplied from the detection unit 72, and outputs the superimposed data to the information presentation control unit 74.

[0130] In step S113, the information presentation control unit 74 presents the superimposed data supplied from the notification information superimposing unit 131 in the VR space 111 of the presentation unit provided in the HMD 61, thereby notifying the user that a person is speaking to them. Then, the processing ends.

[0131] 4. Third Embodiment of the Present Technology (Notification by Person Recognition) Configuration of HMD System FIG. 14 is a block diagram showing an example configuration of an HMD system according to a third embodiment of the present technology.

[0132] FIG. 14 shows an example of the configuration of an HMD system 151 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.

[0133] The HMD system 151 in Fig. 14 differs from the HMD system 51 in Fig. 3 in that a recognition unit 161 and a recognition database 162 are added. The recognition unit 161 and the recognition database 162 correspond to the context management unit 28 in Fig. 1.

[0134] The detection unit 72 supplies the video data supplied from the camera 62, the audio data supplied from the microphone 71, and the detection result of the intervention to the recognition unit 161.

[0135] The recognition unit 161 uses information registered in the recognition database 162, refers to the intervention detection result supplied from the detection unit 72, and recognizes from the video data who is speaking to the user.

[0136] The method for recognizing a person may be face recognition using video data or voice recognition using voice. In either case, the recognition unit 161 performs recognition by, for example, checking against a recognition database 162 in which user information for recognizing people's faces and voices is registered in advance, and if the person is included in the database, returning the ID of the person, or if not, returning an unknown result.

[0137] The recognition unit 161 outputs the video data and audio data supplied from the detection unit 72 and the person recognition result to the notification information generation unit 73 .

[0138] The recognition database 162 stores user information for recognizing a person's face and voice in advance.

[0139] The notification information generating unit 73 generates notification information for notifying the recognition result supplied from the recognizing unit 161 , and outputs the generated notification information to the information presentation control unit 74 .

[0140] 5. Fourth Embodiment of the Present Technology (Superimposed Presentation of Avatar in VR Space) Overview of HMD System FIG. 15 is a diagram illustrating an overview of an HMD system according to a fourth embodiment of the present technology.

[0141] FIG. 15 shows the state of a user using an HMD system 181 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.

[0142] For example, while a user is wearing the HMD 61 and watching VR, another person (a child in the case of the intervener in FIG. 15) speaks to the user from outside, saying, "Hey, Dad."

[0143] The HMD system 181 detects when a child is speaking to the user using a camera 62 and a microphone 71 (Figure 16) provided in the HMD 61, and not only notifies the user of the detection result, but also generates an avatar 190 from the acquired video data and presents the generated avatar 190 in the VR space 111.

[0144] In contrast to the HMD system 101 of Figure 11 in which the image and audio of the person speaking to the user are superimposed on the data of the VR space 111 being viewed by the user, the HMD system 181 of Figure 15 presents an avatar 190 generated from the image of the person speaking to the user as superimposed on the data of the VR space 111 being viewed by the user.

[0145] <Configuration of HMD System> FIG. 16 is a block diagram showing an example of the configuration of the HMD system 181 in FIG.

[0146] 3 in that a skeleton data extraction unit 191 is added and the notification information generation unit 73 is replaced with an avatar generation & 3D information reconstruction unit 192. The skeleton data extraction unit 191 and the avatar generation & 3D information reconstruction unit 192 correspond to the context management unit 28 in FIG.

[0147] The detection unit 72 supplies the video data supplied from the camera 62, the audio data supplied from the microphone 71, and the detection result of the intervention to the skeleton data extraction unit 191.

[0148] The skeleton data extraction unit 191 extracts human skeleton data from the video data supplied from the detection unit 72 , and outputs the video data, audio data, and skeleton data to the avatar generation and three-dimensional information reconstruction unit 192 .

[0149] The avatar generation and 3D information reconstruction unit 192 generates an avatar 190 based on the skeletal data from the skeletal data extraction unit 191, reconstructs it into 3D information, and then superimposes it on the data of the VR space 111, and outputs the superimposed data to the information presentation control unit 74.

[0150] The information presentation control unit 74 displays the superimposition data supplied from the avatar generation and three-dimensional information reconstruction unit 192 on a presentation unit provided in the HMD 61 .

[0151] 6. Fifth embodiment of the present technology (determination of response according to user selection) Configuration of HMD system FIG. 17 is a block diagram showing an example configuration of an HMD system according to a fifth embodiment of the present technology.

[0152] FIG. 17 shows an example of the configuration of an HMD system 201 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.

[0153] The HMD system 201 in Fig. 17 differs from the HMD system 151 in Fig. 14 in that a detection result presentation and inquiry unit 211 is added and the notification information generation unit 73 is replaced with an extracted information superimposition unit 212. The detection result presentation and inquiry unit 211 and the extracted information superimposition unit 212 correspond to the context management unit 28 in Fig. 1.

[0154] That is, the HMD system 201 in FIG. 17 notifies the user of the detection of an interventionist, asks the user how to respond to the interventionist, and determines whether to respond to the interventionist according to the user's selection.

[0155] The recognition unit 161 outputs the video data and audio data supplied from the detection unit 72 and the person recognition results to the detection result presentation and inquiry unit 211. When recognition is not performed, the recognition unit 161 outputs the video data, audio data, and intervention detection results supplied from the detection unit 72 to the detection result presentation and inquiry unit 211.

[0156] The detection result presentation and inquiry unit 211 generates notification information that notifies the person recognition result supplied from the recognition unit 161, and causes the presentation unit provided in the HMD 61 to present the generated notification information and an inquiry as to whether or not to display the person who has spoken in the VR space.

[0157] The detection result presentation and inquiry unit 211 receives the user's selection (approval or denial) via an operation unit (not shown), etc. If the user approves, the detection result presentation and inquiry unit 211 outputs the video data and audio data supplied from the detection unit 72 and the person recognition result (or intervention detection result) to the extracted information superimposition unit 212.

[0158] The extracted information superimposing unit 212 extracts the video and audio data of the interventionist from the video data and audio data supplied from the detection result presentation and inquiry unit 211. The extracted information superimposing unit 212 superimposes the extracted video and audio data of the interventionist on the data of the VR space 111, and outputs the superimposed data to the information presentation control unit 74.

[0159] <Processing of HMD System> FIG. 18 is a flowchart illustrating the processing of the HMD system 201 in FIG.

[0160] For example, a child may speak to a user wearing the HMD 61 and viewing VR, saying, "Hey, Dad."

[0161] In step S201, the camera 62, microphone 71, and detector 72 detect that a person has spoken to them, as described above with reference to FIG.

[0162] In step S202, the recognition unit 161 uses information registered in the recognition database 162, refers to the intervention detection result supplied from the detection unit 72, and recognizes who has spoken from the video data.

[0163] In step S203, the detection result presentation and inquiry unit 211 generates notification information notifying the person recognition result supplied from the recognition unit 161, and causes the generated notification information and an inquiry as to whether or not to display the person who spoke in the VR space to be presented to the presentation unit provided in the HMD 61.

[0164] For example, a question is presented such as "Mr. Q has spoken to you. Would you like to display him in the VR space?" and the user selects yes or no.

[0165] In step S204, the detection result presentation and inquiry unit 211 determines whether or not to display the interventionist in the VR space, depending on the user's selection. If it is determined in step S204 that the interventionist is not to be displayed in the VR space, the processing of the HMD system 201 in FIG. 17 ends.

[0166] If it is determined in step S204 that the interventionist should be displayed in the VR space, the process proceeds to step S205. At this time, the detection result presentation and inquiry unit 211 outputs the video data, audio data, and person recognition results supplied from the recognition unit 161 to the extracted information superimposition unit 212.

[0167] In step S205, the extracted information superimposition unit 212 extracts the video and audio data of the interventionist from the video and audio data supplied from the detection result presentation and inquiry unit 211, superimposes the extracted video and audio data of the interventionist on the data of the VR space 111, and outputs the superimposed data to the information presentation control unit 74.

[0168] In step S206, the information presentation control unit 74 presents the superimposed data supplied from the extracted information superimposing unit 212 in the VR space 111 of the presentation unit provided in the HMD 61, thereby presenting the video and audio of the interventionist to the user. Then, the processing ends.

[0169] This allows the user to continue the conversation with the person speaking to them without leaving the VR space.

[0170] <Example of Communication> FIG. 19 is a diagram showing communication between a user and an intervener.

[0171] FIG. 19A shows "real" communication as seen from the perspective of the intervener who is speaking.

[0172] The intervener who speaks to the user carries out a conversation in real space with the user who is still wearing the HMD 61.

[0173] FIG. 19B shows "virtual" communication from the perspective of the user who is being spoken to.

[0174] The user who is spoken to has a conversation with the projected intermediary in the VR space 111.

[0175] <Example of Presentation Method> FIG. 20 is a diagram showing a display method for presenting an image of an interventionist to a user.

[0176] In FIG. 20, two display methods are shown.

[0177] As shown in A of Figure 20, the HMD system 201 reconstructs a virtual image (avatar) 232 of the intervener captured by an omnidirectional camera 231 installed in the HMD 61 as three-dimensional information and superimposes it on the VR space 111 in which the user is located.

[0178] In this case, a virtual image 232 of the interventionist can be placed in the VR space 111 regardless of the position of the interventionist.

[0179] 20B, the HMD system 201 displays, in a see-through manner, an image of the participant captured by the front camera 241 provided in the HMD 61. In the see-through display, for example, the VR space 111 is displayed with a transmittance of 50%, and the image of the participant is displayed with a transmittance of 50%.

[0180] In this case, while in the VR space 111, the user can see the intervener in the real space at their actual position in the real space.

[0181] <Other Processing of HMD System> FIG. 21 is a flowchart illustrating other processing of the HMD system 201 of FIG.

[0182] In step S221, the camera 52 and the microphone 71 sense the surrounding situation.

[0183] In step S222, the detection unit 72 determines whether or not intervention by a nearby person (talking to, looking at, or approaching) has been detected based on the video data supplied from the camera 62. If it is determined in step S222 that intervention by a nearby person has not been detected, the process returns to step S221, and the subsequent processes are repeated.

[0184] If it is determined in step S222 that intervention by a nearby person has been detected, the process proceeds to step S223.

[0185] In step S223, the detection unit 72 determines whether or not a person has spoken to the user. If it is determined in step S223 that a person has spoken to the user, the process proceeds to step S224.

[0186] In step S224, the detection result presentation and inquiry unit 211 generates notification information notifying the user of the detection result (that a person has spoken to them) supplied from the detection unit 72 via the recognition unit 161, and an inquiry as to whether the user will respond to the intervener (the person who has spoken to them), and presents this information on the presentation unit of the HMD 61.

[0187] In step S225, the detection result presentation and inquiry unit 211 determines whether or not the result corresponds to an intervention person, depending on the user's selection in response to the presentation. If it is determined in step S225 that the result does not correspond to an intervention person, the process returns to step S221, and the subsequent processes are repeated.

[0188] If it is determined in step S225 that the call corresponds to an intervener, the process proceeds to step S226.

[0189] In step S226, the extracted information superimposing unit 212 extracts the voice of the interventionist from the voice data, and the information presenting unit 74 presents it, for example, on the presenting unit (in the VR space). Note that, if necessary, a video of the interventionist may also be presented. Thereafter, the processing ends.

[0190] On the other hand, if it is determined in step S223 that no person is speaking to the user, that is, that a person is looking at the user or approaching the user, the process proceeds to step S227.

[0191] In step S227, the detection result presentation and inquiry unit 211 generates notification information notifying the detection result (that a person is looking or approaching) supplied from the detection unit 72 via the recognition unit 161, and an inquiry as to whether to respond to the intervening person (the person looking or approaching), and presents this information on the presentation unit of the HMD 61.

[0192] In step S228, the detection result presentation and inquiry unit 211 determines whether or not the result corresponds to an intervention person, depending on the user's selection in response to the presentation. If it is determined in step S228 that the result does not correspond to an intervention person, the process returns to step S221, and the subsequent processes are repeated.

[0193] If it is determined in step S228 that the call corresponds to an intervener, the process proceeds to step S229.

[0194] In step S229, the extracted information superimposing unit 212 extracts the image of the interventionist from the video data, and the information presenting unit 74 presents it, for example, on the presenting unit (in the VR space). Note that, if necessary, the voice of the interventionist may also be presented. Then, the processing ends.

[0195] The timing of notifying the user may be changed depending on the state of the intervener (talking to the user, looking at the user, approaching the user).

[0196] For example, if someone speaks to the user, the user will be notified at the timing when the person speaks to them.

[0197] In this case, for example, the voice picked up by the microphone 71 is reproduced as is within the HMD 61. In this way, the user can continue the conversation with the person speaking to them without removing the HMD 61. In some cases, an image of the person speaking to them may also be presented.

[0198] On the other hand, if a person approaches or looks at the user, the user is notified at the timing when the person approaches or looks at the user.

[0199] In this case, for example, the image captured by the camera 62 is directly played back in the HMD 61. In this way, the user can recognize an intervener (a person approaching or watching) without removing the HMD 61. Furthermore, when the intervener speaks, the voice of the intervener may be played back in the HMD 61.

[0200] 7. Sixth Embodiment of the Present Technology (Response Determination by System) Configuration of HMD System FIG. 22 is a block diagram showing an example configuration of an HMD system according to a sixth embodiment of the present technology.

[0201] FIG. 22 shows an example of the configuration of an HMD system 251 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.

[0202] The HMD system 251 in Fig. 22 differs from the HMD system 201 in Fig. 17 in that the detection result presentation and inquiry unit 211 is replaced with a registered data matching unit 261 and in that a registered data storage unit 262 is added. The registered data matching unit 261 and the registered data storage unit 262 correspond to the context management unit 28 in Fig. 1.

[0203] That is, the HMD system 251 of FIG. 22 differs from the HMD system 201 of FIG. 17 in that the HMD system 251 determines whether or not a person corresponds to an interventionist, which was determined based on the user's selection.

[0204] The recognition unit 161 outputs the video data and audio data supplied from the detection unit 72 and the recognition result to the registered data matching unit 261 .

[0205] The registered data comparison unit 261 determines whether or not a person who speaks to the user while the user is watching VR is a person who the user is willing to respond to, based on the recognition result supplied from the recognition unit 161 and the data registered in the registered data storage unit 262.

[0206] If the person who speaks to the user while watching VR is someone the user is willing to respond to, the registered data matching unit 261 outputs the video data and audio data and recognition results supplied from the recognition unit 161 to the extracted information superimposition unit 212, deciding to present them in the VR space.

[0207] If the person who speaks to the user while watching VR is not a person who the user can accept, the registration data comparison unit 261 will refuse to present the person in the VR space. In this case, a message may be displayed to the person informing them that they cannot respond, such as "I'm busy and can't use my hands," and the reason for this, or informing them of the user's situation.

[0208] The registered data storage unit 262 stores in advance data on people who are allowed to respond and people who are not allowed to respond when the user speaks to them while watching VR, for example.

[0209] <Processing of HMD System> FIG. 23 is a flowchart for explaining the processing of the HMD system 251 in FIG.

[0210] For example, a child may speak to a user wearing the HMD 61 and viewing VR, saying, "Hey, Dad."

[0211] In step S241, the camera 62, microphone 71, and detector 72 detect that a person has spoken to them, as described above with reference to FIG.

[0212] In step S242, the recognition unit 161 uses information registered in the recognition database 162, refers to the intervention detection result supplied from the detection unit 72, and recognizes who is speaking from the video data.

[0213] In step S243, the registered data comparison unit 261 determines whether or not to display the intervener in the VR space based on the person recognition result supplied from the recognition unit 161 and the data registered in the registered data storage unit 262.

[0214] If the interventionist is a person who is allowed to respond, it is determined in step S243 that the interventionist is to be displayed in the VR space, and the process proceeds to step S244. At this time, the registered data matching unit 261 outputs the video data, audio data, and person recognition results supplied from the recognition unit 161 to the extracted information superimposing unit 212.

[0215] In step S244, the extracted information superimposition unit 212 extracts the video and audio data of the interventionist from the video data and audio data supplied from the registered data matching unit 261, superimposes the extracted video and audio data of the interventionist on the data of the VR space 111, and outputs the superimposed data to the information presentation control unit 74.

[0216] In step S245, the information presentation control unit 74 presents the superimposed data supplied from the extracted information superimposing unit 212 in the VR space 111 of the presentation unit provided in the HMD 61, thereby presenting the video and audio of the interventionist to the user. Then, the processing ends.

[0217] On the other hand, if the interventionist is a person who is not permitted to respond, it is determined in step S243 that the interventionist will not be displayed in the VR space, and the process proceeds to step S246.

[0218] In step S246, the registered data comparison unit 261 notifies the intervener by voice, text, or video that the data is being captured (message).

[0219] As described above, the HMD system 251 in FIG. 22 can eliminate the need for the user to select whether or not to respond to the interventionist, as compared to the HMD system 201 in FIG.

[0220] <Message Example> FIG. 24 is a diagram showing a method of presenting a message to the interventionist.

[0221] In FIG. 24, two presentation methods are shown.

[0222] In the case of A in FIG. 24, the HMD system 251 outputs a voice message such as "I'm busy right now" from the speaker (not shown) of the HMD 61 to the outside.

[0223] By hearing this message, the intervener can understand that the user is busy and unable to respond.

[0224] In this case, the user does not need to hear the voice message.

[0225] In the case of B in FIG. 24, the HMD system 251 includes an outer display unit 271 on the outside of the HMD 61, and displays a message such as "Importing" on the outer display unit 271.

[0226] By seeing this message, the intervener can understand that the user is currently busy and cannot respond.

[0227] 8. Seventh Embodiment of the Present Technology (Intervention and Obstacle Detection) Overview of HMD System FIG. 25 is a diagram illustrating an overview of an HMD system according to a seventh embodiment of the present technology.

[0228] FIG. 25 shows the state of a user using an HMD system 301 having the functions of the present technology, which is part of the ambient computing system 1 in FIG.

[0229] The HMD system 301 is a system that combines, for example, the HMD system 181 of the fourth embodiment described above with reference to FIG. 15 with object recognition and reproduction in VR space as described in Patent Document 1.

[0230] For example, a person from outside (a child in FIG. 25) speaks to a user wearing the HMD 61 while watching VR, saying, "Hey, Dad."

[0231] In the HMD system 301, the detection unit 72 detects an object Ob that may be an obstacle, in addition to detecting that a child is speaking to the user, using the camera 62 and microphone 71 provided in the HMD 61.

[0232] The HMD system 301 then presents the avatar 190 generated from the video of the child and the virtual object Obr of the detected object Ob superimposed on the data of the VR space 111 being viewed by the user.

[0233] As shown in Figure 25, by detecting objects Ob such as obstacles as well as people and reproducing the virtual objects Obr together with an avatar 190 in the VR space 111, it becomes possible to perform actions on the real space, such as "remove an object that is in the way," and it is also possible to reduce the risk of bumping into an object Ob without realizing it.

[0234] Although an example of combining the HMD system 181 of the fourth embodiment with Patent Document 1 has been described, it is not limited to the fourth embodiment, and may be combined with other embodiments as long as they provide superimposed presentation in a VR space.

[0235] 9. Eighth Embodiment of the Present Technology (Sharing of VR Space) Overview of HMD System FIG. 26 is a diagram showing an overview of an HMD system according to an eighth embodiment of the present technology.

[0236] FIG. 26 shows the state of a user using an HMD system 351 having the functions of the present technology, which is part of the ambient computing system 1 of FIG.

[0237] In the HMD system 201 of the fifth embodiment described above with reference to Figure 17, communication is maintained between a user wearing HMD 61 and viewing VR and an intervener in real space, but in the HMD system 351 of Figure 26, communication is maintained between a user wearing HMD 61-1 and viewing VR and an intervener wearing HMD 61-2 and viewing VR.

[0238] For example, an intermediary person wearing HMD 61-2 and viewing VR may ask a user wearing HMD 61-1 and viewing VR, "Can I join you?"

[0239] The HMD system 351 notifies the user of the detection of an intervener, asks the user how to respond to the intervener, and if the user responds to the intervener according to their choice, invites the intervener's avatar 361 into the VR space 111.

[0240] As described above, in the HMD system 351 of FIG. 26, communication is maintained between a user wearing the HMD 61-1 and viewing VR and an intervener wearing the HMD 61-2 and viewing VR.

[0241] <Processing of HMD System> FIG. 27 is a flowchart illustrating the processing of the HMD system 351 in FIG.

[0242] The configuration of the HMD system 351 is basically the same as the configuration of the HMD system 201 in Fig. 17. Therefore, in Fig. 27, the configuration of the HMD system 351 will be described using the configuration of the HMD system 201 in Fig. 17.

[0243] For example, an intermediary person wearing HMD 61-2 and viewing VR may ask a user wearing HMD 61-1 and viewing VR, "Can I join you?"

[0244] In step S261, the camera 62, microphone 71, and detector 72 detect that a person has spoken to them, as described above with reference to FIG.

[0245] In step S262, the recognition unit 161 uses information registered in the recognition database 162, refers to the detection result supplied from the detection unit 72, and recognizes who is speaking from the video data.

[0246] In step S263, the detection result presentation and inquiry unit 211 generates notification information notifying the user of the recognition result supplied from the recognition unit 161, and causes the presentation unit provided in the HMD 61-1 to present the generated notification information and an inquiry as to whether or not to invite the intervener into the same VR space.

[0247] For example, a question is presented such as "Mr. Q has spoken to you. Would you like to invite him into the same VR space?" and the user selects yes or no.

[0248] In step S264, the detection result presentation and inquiry unit 211 determines whether to invite the intervener into the VR space in accordance with the user's selection. If it is determined in step S264 that the intervener is not to be invited into the same VR space, the processing of the HMD system 351 in FIG. 27 ends.

[0249] If it is determined in step S264 that the interventionist should be invited into the same VR space, the process proceeds to step S265. At this time, the detection result presentation and inquiry unit 211 invites the interventionist into the same VR space 111, for example, by communicating with the interventionist's HMD 61-2.

[0250] In step S265, the extracted information superimposing unit 212 and the information presentation control unit 74 share the VR space 111 with the interventionist's HMD 61-2 and present each other's avatars, etc. on each other's HMDs 61. That is, the extracted information superimposing unit 212 shares the data of the VR space 111 and presents the data of the shared VR space 111 to the information presentation control unit 74, whereby the interventionist is presented in the VR space 111 of the user's HMD 61-1 as an avatar 361. The VR space 111 is also presented on the presentation unit of the interventionist's HMD 61-2 together with the user's avatar. Thereafter, the processing ends.

[0251] As described above, by inviting an intervener into the VR space 111, the user can have a continuous dialogue with the intervener.

[0252] 10. Ninth embodiment of the present technology (control according to intervention level) First control process according to intervention level of HMD system FIG. 28 is a flowchart illustrating a first control process according to an intervention level by an HMD system according to a ninth embodiment of the present technology.

[0253] The control process of Figure 28 is a first control process that changes control depending on conditions when there is external human intervention in the HMD system 201 of the fifth embodiment described above with reference to Figure 17, for example.

[0254] In FIG. 28, the intervention level is defined (set) and controlled based on three conditions (types) of human intervention: looking at you, approaching you, and speaking to you.

[0255] In step S271, the camera 62 and microphone 71 sense the surrounding situation.

[0256] In step S272, the detection unit 72 determines the state of the surrounding people and the state as a result of detecting the people and the state.

[0257] That is, if it is determined in step S272 that people in the vicinity are looking at the user, the process proceeds to step S273.

[0258] In step S273, the detection result presentation and inquiry unit 211 determines that the intervention level is low and notifies the user that a person is looking. That is, the detection result presentation and inquiry unit 211 generates notification information notifying the user of the detection result (that a person is looking) supplied from the detection unit 72 via the recognition unit 161, and causes the presentation unit provided in the HMD 61 to present the notification information. Thereafter, the first control process in FIG. 28 ends.

[0259] If it is determined in step S272 that a nearby person has spoken to the user, the process proceeds to step S274.

[0260] In step S274, the detection result presentation and inquiry unit 211 determines that the intervention level is high and notifies the user that a person has spoken to them. That is, the detection result presentation and inquiry unit 211 generates notification information notifying the user of the detection result (that a person has spoken to them) supplied from the detection unit 72 via the recognition unit 161, and causes the presentation unit provided in the HMD 61 to present the notification information. Thereafter, the processing proceeds to step S277.

[0261] If it is determined in step S272 that a nearby person is approaching, the process proceeds to step S275.

[0262] In step S275, the detection result presentation and inquiry unit 211 determines that the intervention level is medium, notifies the user that a person is approaching, and presents an inquiry as to whether to respond to the approaching person.

[0263] That is, the detection result presentation and inquiry unit 211 generates notification information notifying the user of the detection result (that a person is approaching) supplied from the detection unit 72 via the recognition unit 161 and an inquiry as to whether to respond to the approaching person, and has the notification information and inquiry information presented on the presentation unit provided in the HMD 61.

[0264] In step S276, the detection result presentation and inquiry unit 211 determines whether or not to respond to the user's selection in response to the presentation. If it is determined in step S276 that the detection result presentation and inquiry unit 211 does not respond, the first control process in FIG. 28 ends.

[0265] If it is determined in step S276 that they correspond, the process proceeds to step S277.

[0266] In step S277, the extracted information superimposing unit 212 extracts the image and voice of the intervention person from the video data and voice data, and the information presenting unit 74 presents the image and voice of the intervention person on the presenting unit (in the VR space) to start the dialogue. After that, the first control process in FIG. 28 ends.

[0267] In the above explanation, an example was given in which control is performed by defining an intervention level based on the intervention conditions, but the intervention level may also be defined as follows based on the type of intervention and its conditions (degree of intervention):

[0268] For example, the intervention level may be defined as small < medium < large for the degree of distance when approaching, far < medium < near. The intervention level may be defined as small < medium < large for the degree of speed when approaching, slow < medium < fast. The intervention level may be defined as small < medium < large for the degree of volume of the speaking voice, small < medium < large.

[0269] As described above, there are three types of control methods: notifying the user of the interventionist's status; notifying the user of the interventionist's status and inquiring about a response method; and notifying the user of the interventionist's status and automatically determining and initiating a response method. The control method is selected according to the intervention level.

[0270] Finally, there are three response methods: not responding to the intervener (doing nothing or informing the intervener that they cannot respond), presenting the intervener's status to the user through video and audio, and starting an interactive mode between the intervener and the user. The response method is selected depending on the intervention level.

[0271] FIG. 29 is a diagram showing a control method depending on whether the user and the interventionist are an adult or a child.

[0272] There are five types of control methods depending on whether the user and the intervener are adults or children: (1) Start an interactive mode in which only audio is presented in the VR space without stopping the viewing content; (2) Start an interactive mode in which images and audio are presented in the VR space without stopping the viewing content; (3) Start an interactive mode in which the viewing content is stopped and only audio is presented in the VR space; (4) Start an interactive mode in which the viewing content is stopped and images and audio are presented in the VR space; (5) Do not respond to the intervener.

[0273] For example, as shown in Figure 29, when the user is an adult and the intervener is also an adult, the control method (1) among the above-mentioned control methods (1) to (5) is preset to an interactive mode in which only audio is presented in the HMD without stopping the viewing content.

[0274] When the user is an adult and the intervener is a child, among the above-mentioned control methods (1) to (5), control method (2) is pre-set to start an interactive mode in which images and audio are presented in the HMD without stopping the viewing content.

[0275] If the user is a child and the intervener is an adult, among the above-mentioned control methods (1) to (5), the control method (3) is pre-set to stop the viewing content and start an interactive mode in which only audio is presented within the HMD.

[0276] If the user is a child and the intervener is also a child, among the above-mentioned control methods (1) to (5), the viewing content of control method (4) is stopped and an interactive mode is started in which images and audio are presented within the HMD.

[0277] It should be noted that the optimum pattern may vary depending on the use case, so it is preferable to make the allocation of control methods to each condition changeable.

[0278] Furthermore, the determination of whether the intervener is a child or an adult may be made from the video, and the determination of whether the user is a child or an adult may be set in advance.

[0279] Additionally, the optimal control method for adult or child conditions may vary depending on the use case.

[0280] For example, in a case where an adult talks to a child who is watching VR, the settings may be changed so that the adult can forcibly stop the child from watching VR.

[0281] For example, the setting may be changed to automatically select control method (4) so ​​that when an adult speaks to the child, the adult can communicate without disturbing the child's VR viewing.

[0282] For example, the settings may be changed so that control method (1) is automatically selected when an adult speaks to the device.

[0283] The above describes an example in which an adult speaks to a child, but when a child speaks to an adult, the settings are not as described above.

[0284] In the above description, the attributes of the user and the intervener are different for adults and children, but they may be different for the elderly and young people, or may be different for gender.

[0285] Here, the above-mentioned control methods (1) to (5) can be automatically changed depending on the type of content being viewed.

[0286] For example, the control method corresponding to the type of viewing content may be as follows.

[0287] If the viewing content is a game, the control method (1) is selected because the user does not want to stop the game.

[0288] If the viewing content is dynamic content such as a movie, it is OK to stop it, and if it is not stopped, there is a possibility that the surrounding images and sounds may not be presented, so control method (4) is selected.

[0289] If the viewing content is static content such as photo viewing, research, or work, control method (2) is selected because it is possible to grasp the external situation without stopping the content.

[0290] If the viewing content is a conference (online meeting, etc.), control method (5) is selected because it is desired to limit external intervention.

[0291] <Second Control Processing According to Intervention Level of HMD System> FIG. 30 is a flowchart illustrating second control processing according to the intervention level by the HMD system according to the ninth embodiment of the present technology.

[0292] The control process of Figure 30 is a second control process that changes control depending on conditions when there is external human intervention in the HMD system 201 of the fifth embodiment described above with reference to Figure 17, for example.

[0293] In the second control process, [Auto or Manual], [Type of control method], [Calculation of intervention level], and [Control method corresponding to intervention level] are set in advance as follows:

[0294] [Auto or Manual] It is preset whether the system will control automatically (Auto) or the user will select the control method (Manual).

[0295] [Types of control methods] The following control methods 1 to 5 are preset. Control method 1: Do nothing. Control method 2: Notify the user of the presence of an intervener and the situation. Control method 3: After implementing control method 2, start an interaction mode in which only audio is presented in the HMD without stopping the visual content. Control method 4: After implementing control method 2, start an interaction mode in which images and audio are presented in the HMD without stopping the visual content. Control method 5: After implementing control method 2, start an interaction mode in which the visual content is stopped and images and audio are presented in the HMD.

[0296] The intervention levels are as follows: control method 1<control method 2<control method 3<control method 4<control method 5.

[0297] [Calculation of Intervention Level] An intervention level weight (each a value between 0.0 and 1.0) is preset for each of the following elements.

[0298] Person status (looking, approaching, talking): Ws Type of viewing content (static content, dynamic content, game, meeting): Wc User attributes (child, adult, etc.): Wu Intervener attributes (child, adult, or individual registered in advance): Wo

[0299] The intervention level (Li) is calculated from the above weights using the following formula (1): Li = Ws * Wc * Wu * Wo (1)

[0300] [Control Methods Corresponding to Intervention Levels] In order to determine the control methods corresponding to the intervention levels, a control method determination table showing the control methods corresponding to the above intervention levels is set in advance.

[0301] <Example of Control Method Determination Table> FIG. 31 is a diagram showing an example of a control method determination table.

[0302] When the intervention level is between 0 and 0.2, control method 1 is performed.

[0303] When the intervention level is between 0.2 and 0.4, control method 2 is performed.

[0304] When the intervention level is 0.4 to 0.6, control method 2 is performed and then control method 3 is performed.

[0305] When the intervention level is 0.6 to 0.8, control method 2 is performed and then control method 4 is performed.

[0306] When the intervention level is between 0.8 and 1.0, control method 2 is performed and then control method 5 is performed.

[0307] Next, returning to FIG. 30, the second control process will be described.

[0308] In step S291, the camera 62 and microphone 71 sense the surrounding situation.

[0309] In step S292, the detection unit 72 detects human intervention (looking, approaching, speaking).

[0310] In step S293, the detection result presentation and inquiry unit 211 determines whether the setting is Auto or Manual. If it is determined in step S273 that the setting is automatic, that is, Auto, the process proceeds to step S294.

[0311] In step S294, the detection result presentation and inquiry unit 211 calculates an intervention level from the detection information, content information, and user information.

[0312] In step S295, the detection result presentation and inquiry unit 211 determines which control method the calculated intervention level corresponds to. If it is determined in step S295 that the calculated intervention level corresponds to control method 1, the second control process in FIG. 30 ends.

[0313] If it is determined in step S295 that the control method 2 is supported, the process proceeds to step S296. In step S296, the detection result presentation and inquiry unit 211 notifies the user of the detection result. Thereafter, the second control process of FIG. 30 ends.

[0314] If it is determined in step S295 that the control methods 3 to 5 are supported, the process proceeds to step S299.

[0315] On the other hand, if it is determined in step S293 that the control method is confirmed and set by the user, that is, that the control method is Manual, the process proceeds to step S297.

[0316] In step S297, the detection result presentation and inquiry unit 211 notifies the user of the detection result and inquires about a control method.

[0317] In step S298, the detection result presentation and inquiry unit 211 determines which control method the user selected. Note that in the example of Fig. 30, notification is required, so control method 1 is not included in the options. If it is determined in step S298 that the user selected control method 2, the second control process in Fig. 30 ends.

[0318] If it is determined in step S298 that the user has selected one of control methods 3 to 5, the process proceeds to step S299.

[0319] In step S299, the extracted information superimposing unit 212 extracts the video and audio of the interventionist from the video data and audio data, and the information presenting unit 74 presents the video and audio of the interventionist on the presenting unit or in the VR space, and starts the dialogue mode according to the selected content (content of the control method). After that, the second control process in FIG. 30 ends.

[0320] <11. Others> <Application Examples of the Present Technology> FIG. 32 is a diagram showing an example of a VR space that a user views or experiences.

[0321] FIG. 32 shows a VR space 381 that a user can view or experience.

[0322] The VR space 381 that the user wearing the HMD 61-1 views or experiences may be a locally deployed space.

[0323] The VR space 381 may be a space like a metaverse that is deployed on a network and in which multiple users participate, such as a user wearing an HMD 61-1 and a user wearing an HMD 61-2.

[0324] This technology also targets a user wearing an HMD that operates in a dedicated application, game, or the like, and intervention from another person in the real space where the user is present.

[0325] Here, the device for viewing the VR space is not limited to an HMD, but may be any user device, such as a VR headset, that blocks outside vision and sound when viewing the VR space.

[0326] A VR headset is a headset used in, for example, VR that operates with a dedicated application or VR that operates with a dedicated game console.

[0327] <Effects of the Present Technology> In the present technology, an external intervener is detected for a user wearing a user device that blocks external vision and sound, and control is performed to present information about the detected intervener to the user.

[0328] This helps users to respond quickly and reduces risks.

[0329] <Example of Computer Configuration> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, or into a general-purpose personal computer, etc.

[0330] FIG. 33 is a block diagram showing an example of the hardware configuration of a computer 900 that executes the above-described series of processes according to a program.

[0331] A CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 are interconnected by a bus 904 .

[0332] An input / output interface 910 is also connected to the bus 904. An input unit 911 including a keyboard, a mouse, etc., and an output unit 912 including a display, a speaker, etc. are connected to the input / output interface 910. Also connected to the input / output interface 910 are a storage unit 913 including a hard disk, a nonvolatile memory, etc., a communication unit 914 including a network interface, etc., and a drive 915 that drives a removable recording medium 921.

[0333] In the computer 900 configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing it.

[0334] The program executed by the CPU 901 is recorded on a removable recording medium 921, or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and installed in the storage unit 913.

[0335] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0336] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0337] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0338] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0339] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0340] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0341] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0342] <Examples of Combinations of Configurations> The present technology may also be configured as follows. (1) An information processing device comprising: a detection unit that detects an external intervener for a user wearing a user device that blocks external visibility and sound; and a presentation control unit that controls the presentation of information about the detected intervener to the user. (2) The information processing device described in (1), wherein the presentation control unit controls the presentation of notification information notifying the user of the presence of the intervener, as information about the intervener. (3) The information processing device described in (1), wherein the presentation control unit controls the presentation of at least one of a video of the intervener, an avatar of the intervener, and a voice of the intervener, as information about the intervener. (4) The information processing device described in any of (1) to (3), wherein the presentation control unit controls the presentation of information about the intervener in the VR space being viewed on the device. (5) The information processing device described in any of (1) to (4), further comprising: a response determination unit that determines whether to respond to the intervener; and the presentation control unit controls the presentation of information about the intervener when it is determined that a response should be made to the intervener. (6) The information processing device according to (5), wherein, when it is determined that no action will be taken against the intervener, the presentation control unit controls to present to the intervener information informing the user that the user is unable to take action. (7) The information processing device according to (5), wherein the action determination unit determines whether or not to take action against the intervener according to the user's selection. (8) The information processing device according to (5), further comprising a response storage unit that pre-stores information indicating whether or not an action can be taken for each intervener, and the response determination unit determines whether or not to take action against the intervener according to the information indicating whether or not an action can be taken for each intervener. (9) The information processing device according to (5), wherein the response determination unit determines whether or not to take action against the intervener according to an intervention level. (10) The information processing device according to (9), wherein the intervention level is set according to the type of intervention or the degree of intervention by the intervener.(11) The information processing device according to (10), wherein the type of intervention includes at least one of an action of talking to the user, an action of approaching the user, or an action of looking at the user. (12) The information processing device according to (10), wherein the degree of intervention by the intervener includes at least one of a distance when approaching the user, a speed at which the intervener approaches the user, or a volume of a voice when speaking to the user. (13) The information processing device according to (9), wherein the intervention level is set according to at least one of an attribute of the user and an attribute of the intervener. (14) The information processing device according to (13), wherein the attribute is an adult or a child. (15) The information processing device according to (9), wherein the intervention level is set according to a type of content being viewed by the user. (16) The information processing device according to any of (1) to (15), wherein the detection unit detects the intervener using at least one of video data captured and generated by an imaging unit and audio data collected and generated by a sound collection unit. (17) The information processing device according to (16), wherein the imaging unit and the sound collection unit are provided in the user device. (18) The information processing device according to any of (1) to (17), wherein the user device is a head-mounted display. (19) An information processing method, in which an information processing device detects an external intervener for a user wearing a device that blocks external view and sound, and controls presentation of information about the detected intervener to the user. (20) A program that causes a computer to function as a detection unit that detects an external intervener for a user wearing a device that blocks external view and sound, and a presentation control unit that controls presentation of information about the detected intervener to the user.

[0343] 1 Ambient computing system, 21 User interface, 22 Environmental prior information acquisition unit, 23 User prior information acquisition unit, 24 Collaborative sensing unit, 25 Surrounding environment detection unit, 26 User state detection unit, 27 User response detection unit, 28 Context management unit, 29 Device control unit, 30 Environmental context model storage unit, 31 User context model storage unit, 32 Service list storage unit, 51 HMD system, 61, 61-1, 61-2 HMD, 62 Camera, 71 Microphone, 72 Detection unit, 73 Notification information generation unit, 74 Information presentation control unit, 81 Person detection unit, 82 Action determination unit, 83 Speech interval detection unit, 84 Speaker recognition unit, 85 Speaker data DB, 86 Intervention detection unit, 91 Camera, 92 Microphone, DESCRIPTION OF SYMBOLS 101 HMD system, 111 VR space, 131 notification information superimposition unit, 151 HMD system, 161 recognition unit, 162 recognition database, 181 HMD system, 190 avatar, 191 skeleton data extraction unit, 192 3D information reconstruction unit, 201 HMD system, 211 detection result presentation & inquiry unit, 212 extracted information superimposition unit, 231 omnidirectional camera, 232 virtual image, 241 front camera, 251 HMD system, 261 registration data matching unit, 262 registration data storage unit, 271 external presentation unit, 301 HMD system, 351 HMD system, 361 avatar, 381 VR space

Claims

1. An information processing device comprising: a detection unit that detects an external intervener who is wearing a user device that blocks outside vision and sound; and a presentation control unit that controls the presentation of information about the detected intervener to the user.

2. The information processing device according to claim 1, wherein the presentation control unit controls the presentation of notification information notifying the user of the presence of the intervention person.

3. The information processing device according to claim 1, wherein the presentation control unit controls the presentation of at least one of an image of the interventionist, an avatar of the interventionist, and a voice of the interventionist.

4. The information processing device according to claim 1, wherein the presentation control unit controls the presentation of information about the intervener in the VR space being viewed on the device.

5. An information processing device as described in claim 1, further comprising a response determination unit that determines whether or not to respond to the intervener, and the presentation control unit controls the presentation of information regarding the intervener when it is determined that a response should be made to the intervener.

6. The information processing device according to claim 5, wherein the presentation control unit, when it is determined that no action should be taken against the intervention person, controls the presentation of information to the intervention person informing the intervention person that the user is unable to take action.

7. The information processing device according to claim 5, wherein the response determination unit determines whether or not to respond to the intervener in accordance with a selection made by the user.

8. An information processing device as described in claim 5, further comprising a response memory unit that pre-stores information indicating whether or not a response can be made for each intervener, and the response determination unit determines whether or not to take action against the intervener depending on the information indicating whether or not a response can be made for each intervener.

9. The information processing device according to claim 5, wherein the response determination unit determines whether or not to respond to the intervention person depending on an intervention level.

10. The information processing device according to claim 9, wherein the intervention level is set according to the type of intervention or the degree of intervention by the interventionist.

11. The information processing device according to claim 10, wherein the type of intervention includes at least one of an action of talking to the user, an action of approaching the user, and an action of looking at the user.

12. The information processing device according to claim 10, wherein the degree of intervention by the interventionist includes at least one of the distance at which the interventionist approaches the user, the speed at which the interventionist approaches the user, or the volume of the voice at which the interventionist speaks to the user.

13. The information processing device according to claim 9, wherein the intervention level is set according to at least one of an attribute of the user and an attribute of the intervention person.

14. The information processing device according to claim 13, wherein the attribute is an adult or a child.

15. The information processing device according to claim 9, wherein the intervention level is set according to the type of content being viewed by the user.

16. The information processing device according to claim 1, wherein the detection unit detects the intervener using at least one of video data captured by an imaging unit and audio data collected by an audio collection unit.

17. The information processing device according to claim 16, wherein the imaging unit and the sound collection unit are provided in the user device.

18. The information processing device according to claim 1, wherein the user device is a head-mounted display.

19. An information processing method, in which an information processing device detects an external intervener who is presenting information about the detected intervener to a user who is wearing a device that blocks outside vision and sound, and controls the information processing device to present information about the detected intervener to the user.

20. A program that causes a computer to function as a detection unit that detects an external intruder in relation to a user wearing a device that blocks outside vision and sound, and a presentation control unit that controls the presentation of information about the detected intruder to the user.

Citation Information

Patent Citations

  • Communication device and control method thereof

    JP2018067156A

  • Information processing unit, information processing method, and program

    JP2018113616A

  • Dynamic Entering and Leaving of Virtual-Reality Environments Navigated by Different HMD Users

    US20180005429A1

  • Methods for Providing Interactive Content in a Virtual Reality Scene to Guide an HMD User to Safety Within a Real World Space

    US20180093186A1

  • Systems and methods for reducing an effect of occlusion of a tracker by people

    US20180097975A1