Head-mounted display device, control method thereof, and program

Abstract

To provide a head-mounted display, in which a user participating in a virtual office constructed in a VR space more comfortably works in a metaverse space, while more appropriately grasping the state of his surroundings in a real space.SOLUTION: In a user environment monitoring system, a monitoring unit of a client terminal which is a head-mounted display device (HMD): notifies a user of approach of a person through a field of view of the user, when detecting the approach of the person to the user, while projecting a virtual space to the field of view of the user (S402); and further controls the HMD to transfer to a see-through display mode (S406).SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a head mounted display device, a control method thereof, and a program. [Background technology]

[0002] XR (cross reality) is a general term for technologies that combine the real and virtual worlds, such as VR (virtual reality) and AR (augmented reality), to create a space that provides a simulated experience, and is attracting attention, with various standardization efforts underway.

[0003] In recent years, virtual spaces and services that use such technology are called metaverses, and are being used not only for entertainment purposes such as games, but also in business situations such as virtual offices and VR conference rooms. In virtual spaces, each user wears an HMD (Head Mounted Display, hereafter referred to as "HMD") and can communicate with other users' avatars in the virtual space through the HMD just as they would in the real world, and can also bring in a PC (Personal Computer) to work in the virtual space.

[0004] In addition, users who wear non-transparent HMDs in a virtual space lose their view of the real world and become completely immersed in the virtual space displayed on the display. This makes it difficult for them to grasp the surroundings in the real world.

[0005] Patent document 1 proposes a technology that detects the surrounding environment in real space using surrounding images acquired from a camera, associates information about the surrounding environment with virtual space information, and outputs it to an HMD, thereby enabling a user wearing an HMD and immersed in a three-dimensional virtual space to detect changes in their surrounding environment while still wearing the HMD. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2016-198180 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, when a user is wearing an HMD and working in a virtual office or the like, various situations may occur in the real space, such as people approaching, talking to the user, conversations being overheard, the screen of the PC the user is using being viewed, etc. It is difficult for a user wearing an HMD and working in a virtual office or the like to appropriately grasp and respond to such situations, and conventional technologies such as Patent Document 1 have been insufficient.

[0008] The present invention has been made to solve the above problems, and aims to provide a mechanism that enables a user who is participating in a virtual space by wearing a head mounted display to more appropriately and easily grasp the state of his / her surroundings in the real space and to more comfortably perform work in the virtual space. [Means for solving the problem]

[0009] The present invention is a head-mounted display device that projects a virtual space into a user's field of vision, and is characterized by having a notification means that, when it is detected that a person is approaching the user while the virtual space is being projected into the user's field of vision, notifies the user of the approach of a person in the user's field of vision, and a control means that, when it is detected that the person is approaching, controls the device to transition to a see-through display mode in at least a portion of the user's field of vision, in which the real space can be viewed through the virtual space. Effect of the Invention

[0010] According to the present invention, a user who is wearing a head-mounted display and participating in a virtual space can more appropriately and easily grasp the state of their surroundings in real space, and can work more comfortably in the virtual space. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing the overall configuration of a user environment monitoring system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of the devices that make up the user environment monitoring system. [Diagram 3] FIG. 2 is a software configuration diagram of the user environment monitoring system according to the embodiment. [Figure 4] 5 is a flowchart illustrating a monitoring process according to the first embodiment. [Figure 5A] FIG. 13 is a diagram showing an example of a screen display of a client terminal. [Figure 5B] FIG. 13 is a diagram showing an example of a screen display of a client terminal. [Figure 5C] FIG. 13 is a diagram showing an example of a screen display of a client terminal. [Figure 6] 10 is a flowchart illustrating a monitoring process according to a second embodiment. [Figure 7] 11 is a flowchart illustrating a conference data control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing an example of the overall configuration of a user environment monitoring system according to an embodiment of the present invention. The user environment monitoring system of this embodiment includes a virtual space management server 111 and client terminals 121 to 132, which are connected to each other via networks 100 to 102.

[0013] The networks 100 to 102 are so-called communication networks realized by, for example, the Internet, LAN, WAN, telephone lines, dedicated digital lines, ATM or frame relay lines, cable television lines, wireless lines for data broadcasting, etc. The networks 100 to 102 may be capable of transmitting and receiving data. In this embodiment, the network 100 is the Internet, and the networks 101 and 102 are the Internet, networks in ordinary homes, companies, and schools, wireless LANs set up in town, etc.

[0014] The client terminals 121 and 131 are HMDs (head mounted display devices) and include a camera for capturing images of the surroundings, a microphone for inputting voice, and a display for displaying virtual objects.

[0015] The client terminals 122 and 132 are PCs (personal computers) or the like, and are information processing devices having a keyboard, a display, etc. The client terminals 122 and 132 have a function of transmitting content displayed on the PC screen to the virtual space management server 111 in order to bring the PC screen of the real space into the virtual space.

[0016] The virtual space management server 111 is a server for providing each client terminal with virtual objects in the virtual space, avatars that mimic each user, UIs (User Interfaces) and AI (Artificial Intelligence) avatars of services and systems, and their location information and sounds. The virtual space management server 111 also manages users who log in to the virtual space using the client terminals 121 to 132. That is, the virtual space management server 111 receives login and logout requests from the client terminals 121 to 132 and performs login and logout processing. The virtual space management server 111 also manages location information of users who are logged in to the virtual space. Furthermore, the virtual space management server 111 has a function of receiving and providing contents transmitted from the client terminals 121 to 132 and the like, and managing the sharing state.

[0017] The virtual space management server 111 is constructed using a server computer. Alternatively, the virtual space management server 111 can be constructed by adopting cloud computing technology or the like. The functions of the virtual space management server 111 described in this embodiment may be realized by a single server or a single virtual server, or may be realized by multiple servers or multiple virtual servers. Alternatively, multiple virtual servers may be executed on a single server.

[0018] FIG. 2 is a diagram showing an example of the hardware configuration of the virtual space management server 111 and the client terminals 121 to 132. As shown in FIG. 2, a CPU (Central Processing Unit) 202 controls the entire device. The CPU 202 executes application programs, an OS, and the like stored in a HDD 205. At this time, the CPU 202 controls temporary storage of information, files, and the like required for executing the programs in a RAM 203. A GPU (Graphics Processing Unit) 210 performs the calculations required to render virtual objects and avatars in real time.

[0019] A ROM (Read Only Memory) 204 is a storage unit, and stores various data such as a basic I / O program therein. A RAM (Random Access Memory) 203 is a temporary storage unit, and functions as a main memory, a work area, etc. for the CPU 202 and the GPU 210 . The HDD (Hard Disc Drive) 205 is one of the external storage units, functions as a large-capacity memory, and stores application programs such as a Web browser, programs for a group of service servers, an OS, related programs, etc. Note that other storage devices such as an SSD (Solid State Drive) may be provided instead of or in combination with the HDD.

[0020] The display 206 is a display unit that displays virtual objects, information necessary for operations, etc. The display 206 may also be a device that has a function of receiving operation instructions from a user, such as a touch panel.

[0021] The camera 207 may be an out-camera in the client terminals 121 to 132 that captures images of the surroundings, or an in-camera that mainly captures images of the user himself / herself. By analyzing the images captured by the camera 207, particularly the out-camera, with a program stored in the HDD 205, it is possible to synchronize the movements of the fingers in the real space with those of the fingers of the user avatar in the virtual space. Furthermore, it becomes possible to virtually contact with virtual objects or other avatars in the virtual space displayed on the display 206 by the synchronized fingers. The virtual space management server 111 and the client terminals 122 and 132 do not necessarily require the camera 207.

[0022] The interface 208 is an external device I / F that connects peripheral devices such as various external sensors. The camera 207 mentioned above recognizes the user's fingers in real space, allowing the user to operate virtual objects in virtual space and touch other avatars, but equivalent functions can also be achieved by operating a dedicated controller connected to the interface 208. Voice input can also be performed by speaking into a voice input device such as a microphone connected to the interface 208.

[0023] The speaker 209 is a device that converts electrical signals in the client terminals 121 to 132 into physical sound, and the user can hear the sound provided by the virtual space management server 111 and the client terminals 121 to 132 through the speaker 209. The user may hear the sound through the speaker 209, or through an external device such as headphones connected to the interface 208.

[0024] The system bus 201 controls the flow of data within the device. A NIC (Network Interface Card) 211 exchanges data with external devices via the networks 100-102.

[0025] The above computer configuration is an example, and is not limited to the configuration example in Fig. 2. For example, the storage destination of data and programs can be changed to RAM 203, ROM 204, HDD 205, etc., depending on the characteristics of the data and programs. In addition, the CPU 202 and GPU 210 execute processing based on programs stored in HDD 205, etc., thereby realizing processing in the software configuration shown in Fig. 3.

[0026] Next, the software configuration of the user environment monitoring system according to this embodiment will be described with reference to FIG. FIG. 3 is a diagram for explaining the software configuration of the virtual space management server 111 and the client terminals 121, 122, 131, and 132, extracting functions related to this embodiment.

[0027] <Virtual Space Management Server 111> Each unit of the virtual space management server 111 shown in FIG. 3 is realized by the CPU 202 of the virtual space management server 111 reading and executing a program stored in the ROM 204, the HDD 205, or the like. The virtual space management server 111 includes a user management unit 301 and a login processing unit 302 as user management functions.

[0028] The user management unit 301 manages user information and login information. The login processing unit 302 receives login requests from the client terminals 121 to 132 , collates the requests with information from the user management unit 301 , and returns the results of the login processing to the client terminals 121 , 131 , and 132 .

[0029] Table 1 shows an example of data managed by the user management unit 301. [Table 1]

[0030] The user management unit 301 manages data using a user management table such as Table 1. In a user management table such as Table 1, the user ID is an ID that uniquely identifies a user. The password is a basic authentication password used when logging in with the user ID. The login processing unit 302 checks the combination of the user ID and password included in the login request from the client terminals 121 to 132 against Table 1, and if they match, returns the login result to each client terminal as a success. On the other hand, if they do not match, returns the login result to each client terminal as a failure.

[0031] In the user management table, the login status indicates the login status of the user, with "on" indicating a logged-in status and "off" indicating a logged-out status. The login expiration date indicates the expiration date of the authentication status of the logged-in user. The monitoring setting indicates the state of the monitoring setting for monitoring the real space of the user participating in the virtual office configured in the VR space. If the monitoring setting is set by the user, it is set to "on", and if the monitoring setting is not set, it is set to "off". The location information is the three-dimensional location of each user in real space, and indicates the current location information of each user obtained using, for example, a GPS sensor connected via the interface 208 of the client terminals 121 and 131.

[0032] The virtual space management server 111 also includes a virtual object management unit 303, a virtual object providing unit 304, a virtual space position information management unit 305, a virtual space position information acquisition unit 306, and a virtual space position information providing unit 307 as basic functions for providing a virtual space.

[0033] The virtual object management unit 303 manages virtual objects placed in the virtual space, 3D data of each user avatar, and the like. The virtual object providing unit 304 provides the 3D data managed by the virtual object management unit 303 to the client terminals 121 and 131 . The virtual space position information management unit 305 manages the position information of the virtual objects and avatars managed by the virtual object management unit 303 .

[0034] Table 2 shows an example of the position information of the avatar of each user managed by the virtual space position information management unit 305. [Table 2]

[0035] The virtual space position information management unit 305 manages the user avatar position information using a user avatar position information management table such as Table 2. In a user avatar position information management table such as Table 2, the user ID is an ID that uniquely identifies a user. The space ID is an ID that uniquely identifies a virtual space. The main coordinates are information that indicates the current position (center of gravity) of the avatar in the virtual space. The left hand coordinates and right hand coordinates are information that indicates the positions of the avatar's left and right hands in the virtual space.

[0036] Table 3 shows an example of the position information of a virtual object. [Table 3]

[0037] The virtual space position information management unit 305 manages the virtual object position information using a virtual object position information management table such as Table 3. In a virtual object position information management table such as Table 3, a virtual object ID is an ID that uniquely identifies a virtual object in the virtual space. The space ID and main coordinates are the same as those in Table 2, so their explanation will be omitted.

[0038] The virtual space position information acquisition unit 306 receives position information from the client terminals 121 and 131 periodically or when there is a change in the position of a virtual object or avatar in the virtual space due to an operation from the client terminals 121 and 131 . The virtual space position information providing unit 307 provides the client terminals 121 and 131 with position information of virtual objects and avatars in the virtual space managed by the virtual space position information management unit 305 .

[0039] The virtual space management server 111 also includes an audio data acquisition unit 308, an audio data provision unit 309, a sharing state management unit 310, and a screen data provision unit 311 as functions for managing and controlling the sharing state of audio and screen content.

[0040] The voice data acquisition unit 308 acquires voice input by a voice input device such as a microphone connected to the interface 208 of the client terminals 121 and 131 from the voice data provision unit 327 . The voice data providing unit 309 controls whether or not to provide the user's voice in the virtual space based on the voice sharing state in a sharing state management table such as Table 4 described later. The sharing status management unit 310 manages data for determining whether screen display content and audio can be shared.

[0041] Table 4 shows an example of data managed by the shared state management unit 310. [Table 4]

[0042] In a sharing status management table such as Table 4, the user ID is an ID that uniquely identifies a user. The screen sharing status indicates whether or not the content displayed on the PC screen is being shared within the virtual space. The screen sharing destination indicates the destination of the screen content of one's own PC within the virtual space. For example, if the screen sharing status is "Brought-in PC", it indicates that the screen display content of one's own PC is being shared with the screen of one's own PC that has been brought into the virtual space. Also, if the screen sharing status is "Slide A", it indicates that the screen display content of one's own PC is being shared with a specific slide ("Slide A" in this example) within the virtual space.

[0043] The screen mode indicates the display mode state of one's own PC screen that is shared in the virtual space. For example, when the screen mode is "Duplicate", it indicates that the display content of one's own PC screen is duplicated and displayed in the virtual space. When the screen mode is "VR only", it indicates that the display content of one's own PC screen is only displayed in the virtual space (i.e., it is not displayed on the real PC display 206). When the screen mode is "False", it indicates that the display content of one's own PC screen is not displayed in the virtual space.

[0044] The audio sharing state indicates whether or not each user's audio (audio input via an audio input device such as a microphone connected to the interface 208 of the real client terminal) is being shared in the virtual space. The audio sharing destination indicates the destination of each user's audio. For example, when the audio sharing destination is "virtual space", it indicates that the user's audio is provided to all users in the virtual space. Also, when the audio sharing destination is "conference participant", it indicates that the user's audio is provided only to the conference participant.

[0045] The screen data providing unit 311 controls whether or not to provide the content displayed on each user's PC (client terminal 122, 132) screen into the virtual space based on the screen sharing status in the sharing status management table as shown in Table 4. Furthermore, if the screen sharing status is "true", the screen data providing unit 311 controls the screen sharing destination and screen mode based on the screen sharing destination and screen mode information in the sharing status management table.

[0046] For example, when a user instructs the screen sharing status, screen sharing destination, and screen mode of his / her PC (e.g., a client terminal (PC) logged in to the virtual space management server 111 with his / her user ID), and also instructs the audio sharing status and audio sharing destination from the client terminal (HMD) worn by the user, these instructions are transmitted to the virtual space management server 111. In response to this, the sharing status management unit 310 of the virtual space management server 111 manages the sharing status. Note that, although the user's own PC is a client terminal (PC) logged in to the virtual space management server 111 with his / her user ID, this is not limiting. For example, even if the user ID logged in to the virtual space management server 111 is a client terminal different from the user's own user ID, it may be considered as the user's own PC by linking it to the user's own user ID.

[0047] The virtual space management server 111 also includes a conference information management unit 312 as a function for managing conference information in the virtual space. The conference information management unit 312 manages conference information in the virtual space set by a user based on a conference information management table such as Table 5, and further controls audio data during the conference of each user.

[0048] Table 5 shows an example of the conference information management table managed by the conference information management unit 312. [Table 5]

[0049] In a conference information management table such as Table 5, the conference ID is an ID that uniquely identifies a conference in a virtual space set by a user. The date and time of the meeting is information indicating the date and time of the meeting associated with the meeting ID. The conference participants are information indicating the participants of the conference linked to the conference ID, and are managed by user IDs that uniquely identify users managed in a user management table such as Table 1. The confidentiality level is information indicating the confidentiality level of the conference associated with the conference ID. When the confidentiality level is set to "high", the voice data providing unit 327 controls the voice data not to be provided to anyone other than the conference participants according to the flowchart of FIG. 7 described later.

[0050] <Client terminals 121, 131> Next, the software configuration of the client terminals 121 and 131 will be described. Each unit of the client terminals 121 and 131 shown in FIG. 3 is realized by the CPU 202 of the client terminals 121 and 131 reading and executing a program stored in the ROM 204, the HDD 205, or the like.

[0051] The client terminals 121 and 131 are HMDs (head mounted displays). The client terminals 121 and 131 include a display unit 320, a login unit 321, a local virtual object management unit 322, a virtual object acquisition unit 323, a local virtual space position information management unit 324, a virtual space position information providing unit 325, and a virtual space position information acquiring unit 326. The client terminals 121 and 131 also include an audio data providing unit 327, an audio data acquiring unit 328, an audio data output unit 329, a screen data acquiring unit 330, a screen data output unit 331, a real space shooting unit 332, and a monitoring unit 333. The HMD according to this embodiment is a non-transparent HMD, and a user wearing the HMD is completely immersed in the virtual space. Each part will be described in detail below.

[0052] The display unit 320 projects a virtual space into the field of view of the user wearing the HMD via the display 206, and displays virtual objects and avatars within the virtual space. The login unit 321 transmits a finger image captured by the camera 207 or a user name and password input by an input device connected to the interface 208 to the login processing unit 302 of the virtual space management server 111. At this time, the login unit 321 may transmit to the virtual space management server 111 that it is a device that projects a virtual space into the user's field of vision, such as an HMD. The authentication method for the virtual space management server 111 may be face authentication using a face image captured by the camera 207, iris authentication using an iris, fingerprint authentication using a fingerprint sensor connected to the interface 208, or the like.

[0053] The local virtual object management unit 322 manages information such as virtual objects and avatar 3D data acquired from the virtual space management server 111 on the client terminals 121 and 131 . The virtual object acquisition unit 323 acquires information such as 3D data of virtual objects and avatars from the virtual object provision unit 304 of the virtual space management server 111 , and stores the information in the local virtual object management unit 322 .

[0054] The local virtual space position information management unit 324 manages information of the user avatar position information management table (example shown in Table 2) and the virtual object position information management table (example shown in Table 3) acquired from the virtual space management server 111, i.e., the position information of virtual objects and avatars in the virtual space. The local virtual space position information management unit 324 also has a function of detecting the positions of virtual objects and avatars on its own terminal that have changed due to the operation of the client terminals 121 and 131, and storing the position information in the local virtual space position information management unit 324 itself.

[0055] The virtual space position information providing unit 325 transmits the position information of the virtual objects and avatars in the virtual space to the virtual space position information acquisition unit 306 of the virtual space management server 111 when there is a change in the position information of the virtual objects and avatars in the virtual space due to operations on the client terminals 121 and 131, and also periodically. The virtual space position information acquisition unit 326 periodically acquires position information of virtual objects and avatars from the virtual space position information provision unit 307 of the virtual space management server 111 , and stores it in the local virtual space position information management unit 324 .

[0056] The voice data providing unit 327 transmits voice input by a voice input device such as a microphone connected to the interface 208 to the voice data acquiring unit 308 of the virtual space management server 111 . The audio data acquisition unit 328 acquires the audio of other users in the virtual space, system sounds, and BGM (Back Ground Music) from the audio data provision unit 309 of the virtual space management server 111 . The audio data output unit 329 outputs the audio data acquired by the audio data acquisition unit 328 through an external device such as the speaker 209 or headphones connected to the interface 208 .

[0057] The screen data acquisition unit 330 acquires the content displayed on the screen of its own PC (client terminal 122, 132) via the screen data provision unit 311 of the virtual space management server 111. The screen data output unit 331 outputs the content data acquired by the screen data acquisition unit 330 to the screen of the user's own PC brought into the virtual space, a specific slide, or the like.

[0058] The real space photographing unit 332 stores images or videos of the real space photographed by the camera 207 of the client terminal 121, 131 in the RAM 203, HDD 205, etc. of the client terminal 121, 131, or displays them on the display 206 of the client terminal 121, 131.

[0059] The monitoring unit 333 detects the approach of a person in the user's real space by using an image acquired by the real space shooting unit 332. In this embodiment, the image captured by the camera 207 of the HMD is used to detect a person by face detection or the like, but the detection may be performed by motion detection (moving object detection), human detection by AI, or a sensor connected to the interface 208. The monitoring unit 333 may also detect the approach of a person by acquiring an image captured by the camera 207 of its own PC (client terminal 122, 132, etc.) from the virtual space management server 111. When the monitoring unit 333 detects that a person is approaching while projecting the virtual space into the user's field of view, it notifies the user wearing the HMD of the approach of the person. Furthermore, in this case, the monitoring unit 333 instructs the display unit 320 to control the display unit 320 to transition to a see-through mode (also called a "see-through display mode") in which the real space can be confirmed through the virtual space in at least a part of the user's field of view. Details are shown in FIG. 4, which will be described later.

[0060] By the above-mentioned functions 320 to 332, the behavior, state, and sound of virtual objects and other avatars in the virtual space provided by the virtual space management server 111 can be provided to the user in real time.

[0061] <Client terminals 122, 132> Next, the software configuration of the client terminals 122 and 132 will be described. Each unit of the client terminal 122, 132 shown in FIG. 3 is realized by the CPU 202 of the client terminal 122, 132 reading and executing a program stored in the ROM 204, the HDD 205, or the like. The client terminals 122 and 132 are information processing devices such as a PC (herein described as a PC), and include a display unit 340, a login unit 341, and a screen data transmission unit 342.

[0062] The display unit 340 displays the content on the display 206 of the PC of the client terminal 122, 132 in real space. The login unit 341, like the login unit 321 of the client terminals 121 and 131 which are HMDs, transmits a user name and password inputted by an input device connected to the interface 208 to the login processing unit 302 of the virtual space management server 111. At this time, the login unit 341 may transmit to the virtual space management server 111 that it is an information processing device such as a PC. The screen data transmission unit 342 performs control as to whether or not to transmit the content displayed on the user's PC screen (client terminal 122, 132) into the virtual space in response to a screen sharing instruction from the user.

[0063] <Monitoring process> 4 is a flowchart for explaining the process of monitoring the real space of a user participating in a virtual office or the like configured in a VR space by the monitoring unit 333 of the client terminals 121 and 131 of the user environment monitoring system according to the first embodiment. The process of this flowchart is realized by the CPU 202 of the client terminals 121 and 131 reading and executing a program stored in the ROM 204 or HDD 205 or the like. This process is performed only for users whose monitoring settings are set to "on" in the user management table such as that shown in Table 1.

[0064] First, in S401, when the monitoring unit 333 of the client terminal 121, 131, which is an HMD worn by a user, detects the approach of a person in the real space of the wearing user, the process proceeds to S402. Note that the detection in S401 is not limited to the simple approach of a person, but may also include, for example, detection of a person approaching the user continuously for a certain period of time or more, or detection of the line of sight of the approaching person being directed at the user for a certain period of time, and it may be determined that the detection in S401 has occurred when at least one of these has been detected.

[0065] In S402, the display unit 320 of the client terminal 121, 131 notifies the wearing user participating in the virtual space of the approach of a person in the real space via the display 206 of the client terminal 121, 131 (i.e., via the field of vision of the wearing user).

[0066] FIG. 5A is a diagram showing an example of a message or the like displayed on the display 206 of the client terminals 121 and 131 in S402 of FIG. 4 to notify users participating in the virtual space of an approaching person. Reference numeral 500 denotes a screen projected into the field of view of a user participating in the virtual space. Reference numeral 501 denotes the screen of the user's own PC that is brought into the virtual space. An image (503) of the user's surroundings, including approaching people, captured by the camera 207, is displayed next to a screen 501 of the user's PC along with a message 502. In this way, when a virtual space is being projected into the user's field of vision and it is detected that a person is approaching the user, a notification of the approaching person is displayed in the user's field of vision, for example as shown in 502 and 503.

[0067] Then, in S403, the monitoring unit 333 judges whether or not the screen of the PC of the user is being shared in the virtual space. Specifically, the monitoring unit 333 requests the virtual space management server 111 to acquire the screen sharing status. Then, the virtual space management server 111, which has received the request to acquire the screen sharing status from the client terminals 121 and 131, returns information on the screen sharing status of the user based on the screen sharing status in the sharing status management table (illustrated in Table 4). The information on the screen sharing status is information on whether or not the content displayed on the user's own PC screen (client terminals 122 and 132) is being provided in the virtual space, and information on the screen display mode. Furthermore, the monitoring unit 333 judges whether or not the screen is being shared based on the information acquired from the virtual space management server 111.

[0068] If the monitoring unit 333 determines that the screen is not in a shared state (No in S403), the process proceeds to S406. The process from S406 onwards will be described later. On the other hand, if it is determined that the screen is being shared (Yes in S403), the monitoring unit 333 advances the process to S404.

[0069] In S404, it is determined whether the content on the PC screen is also being displayed in the real space. Whether the content is being displayed on the PC screen in the real space is determined by whether the screen mode of the user acquired from the virtual space management server 111 in S403 above is "copy" (i.e., the mode in which the content of the PC in the real space is copied and displayed in the virtual space).

[0070] Here, if the monitoring unit 333 determines that the content is not being displayed on the PC screen in the real space (that is, the PC is not in the "copy" mode) (No in S404), the process proceeds to S406. On the other hand, if it is determined that the content is being displayed on the PC screen in the real space (that is, the mode is "copy") (Yes in S404), the monitoring unit 333 advances the process to S405.

[0071] In S405, the monitoring unit 333 changes the screen mode of the user wearing the device from "replication" to "VR only", thereby changing the PC screen display on the real space side to OFF. Here, the monitoring unit 333 notifies the virtual space management server 111 of the change in the screen mode of the user (change from "replication" to "VR only"). As a result, the screen mode of the user in the sharing state management table (illustrated in Table 4) managed by the sharing state management unit 310 of the virtual space management server 111 is also changed to "VR only". Furthermore, the change in the screen mode (change from "replication" to "VR only") is provided to the display unit 340 of the user's PC (client terminal 122, 132) via the virtual space management server 111. For example, the user's PC (client terminal 122, 132) periodically acquires the above-mentioned screen mode information from the virtual space management server 111. In response to this, the display unit 340 of the user's PC turns off the PC screen display (for example, masks it). When the user turns off the screen display of the PC he or she is using, a notification to that effect may be displayed in the user's field of vision. After the process of S405, the monitoring unit 333 advances the process to S406.

[0072] In S406, the monitoring unit 333 instructs the display unit 320 to change the display of the client terminal 121, 131, which is the HMD worn by the user, to a see-through mode (transparent display). As a result, the display unit 320 of the HMD (client terminal 121, 131) worn by the user controls the display so that the real world can be confirmed through the virtual space in at least a part of the user's field of vision. In this way, by switching to the see-through mode, the user can confirm the real world through the virtual space, making it possible to more easily and reliably grasp the situation around him / her in the real space. As a result, more appropriate measures can be taken.

[0073] Furthermore, in S407, the monitoring unit 333 turns off the microphone output of the user on the virtual space side via the voice data providing unit 327. Specifically, the voice data providing unit 327 temporarily stops the process of transmitting the voice input by the voice input device of the microphone connected to the interface 208 to the voice data acquiring unit 308 of the virtual space management server 111, thereby turning off the voice output to the virtual space of the user.

[0074] FIG. 5B is a diagram showing an example in which the display mode of the client terminals 121 and 131 is changed to a see-through mode (transparent display) in S406 of FIG. In the see-through mode, the real world can be confirmed through the virtual space in at least a part of the user's field of vision. For example, as shown in FIG. 5B, the user can confirm a person 505 approaching the user in the real space. This allows the user to properly grasp changes in the surrounding environment (such as the approach of a person) even while wearing the HMD, and to make appropriate decisions and responses in response to the changes in the surrounding environment.

[0075] In the see-through mode, a button 504 for canceling the see-through mode is displayed on the display 206. For example, when the user finishes the conversation in the real space, the user can cancel the see-through mode by tapping the button 504 displayed on the display 206 of the client terminal 121, 131. Furthermore, by tapping the button 504, the provision of microphone audio to the virtual space by the audio data providing unit 327 is also resumed. After the process of S407, the monitoring unit 333 ends the process of this flowchart.

[0076] The operation of tapping the button 504 corresponds to, for example, an operation of tapping the button 504 displayed on the display with the fingers of a user avatar in the virtual space that moves in conjunction with the movement of the fingers in the real space. Also, the see-through mode may be cancelled by the user operating (for example, by physical operation) an operation unit in the real space of the client terminals 121 and 131 that are HMDs or an operation unit connected to the interface 208. Also, a predetermined gesture by the user may be detected based on an image from the camera 207 to cancel the see-through mode.

[0077] When audio output to the virtual space is turned OFF or turned ON again, a notification to that effect may be displayed in the user's field of vision.

[0078] By the above process, when a person approaches a user who is working on his / her own PC in the virtual space, the user can grasp the approach of the person through the display of the HMD. Furthermore, by temporarily switching the display of the HMD to a see-through mode, the user can check the real space through the virtual space while wearing the HMD, and for example, it is possible to not only talk to the person approaching, but also to prevent the contents of the conversation from being provided to the virtual space. Furthermore, by automatically switching off the display of the PC, it is possible to prevent the content of the PC screen in the real space from being seen by the person approaching. In this way, the monitoring system of this embodiment monitors the surroundings of the real space of a user who is participating in a virtual office configured in a VR space by wearing an HMD or the like, and can more easily notice the presence of a person approaching the user in the real space or being spoken to by the user. In addition, the risk of conversations being eavesdropped on or PC screens being peeked at can be reduced.

[0079] Second Embodiment In the monitoring process by the monitoring unit 333 of the client terminals 121 and 131 shown in Fig. 4 in the first embodiment, when detecting the approach of a person in the real space of the user wearing the client terminal, it is not considered whether the approaching person is participating in the same virtual space as the user. In the first embodiment, as in the example shown in Fig. 5C, even if a person (515, 516 in the figure) who is having a conversation in the same virtual space as the user approaches, the person is notified as an approaching person. In the second embodiment, the monitoring process flow by the monitoring unit 333 is expanded so as to consider whether an approaching person is a user participating in the same virtual space as the user. This will be described in detail below.

[0080] <Monitoring process> 6 is a flowchart for explaining the process of monitoring the real space of a user participating in a virtual office configured in a VR space by the monitoring unit 333 of the client terminals 121 and 131 of the user environment monitoring system according to the second embodiment. The process of this flowchart is realized by the CPU 202 of the client terminals 121 and 131 reading and executing a program stored in the ROM 204 or HDD 205, etc.

[0081] First, in S601, similarly to S401 in FIG. 4, when the monitoring unit 333 of the client terminal 121, 131, which is an HMD worn by a user, detects the approach of a person in the real space of the user wearing the HMD, the process proceeds to S602.

[0082] In S602, the monitoring unit 333 judges whether the approaching person is participating in the same virtual space as the user. Specifically, the monitoring unit 333 inquires of the virtual space management server 111 whether the approaching person is logged in to the same virtual space as the user, together with the estimated position coordinate information of the approaching person. The virtual space management server 111, which has received an inquiry from the client terminal 121, 131, checks whether there is a user whose position information is close based on the current position information of each user in the user management table of Table 1. If there is a user whose position information is close, the monitoring unit 333 returns the information of the user ID to the client terminal 121, 131. If the information acquired from the virtual space management server 111 includes a user ID, the monitoring unit 333 judges that the approaching person is participating in the same virtual space as the user. On the other hand, if the information acquired from the virtual space management server 111 does not include a user ID, the monitoring unit 333 judges that the approaching person is not participating in the same virtual space as the user.

[0083] If the monitoring unit 333 determines that the approaching person is participating in the same virtual space as the user (Yes in S602), the process of this flowchart ends. On the other hand, if the approaching person is participating in the same virtual space as the user (No in S602), the monitoring unit 333 advances the process to S603. Note that the processes of S603 to S608 are the same as those of S402 to S407 in FIG. 4, and therefore description thereof will be omitted.

[0084] 6, a configuration has been described in which an approaching person is notified as an approaching person when the person is not participating in the same virtual space as the user, and is not notified when the person is participating in the same virtual space. However, a configuration may be used in which an approaching person is notified as an approaching person when the person is not participating in the same conference as the user, and is not notified when the person is participating in the same conference.

[0085] By the above process, if the approaching person is a user participating in the same virtual space or the same conference as the user to be monitored, the person can be excluded from the notification target. Note that even if the answer is Yes in S602, the mode may be switched to the see-through mode so that the approaching person can be confirmed in the real space.

[0086] Third Embodiment In a conference in a virtual space, depending on the contents of the conference (confidentiality level), even if users are participating in the same virtual space, they may not want people other than the conference participants to hear the contents of the conversation. In the third embodiment, a configuration will be described in which the virtual space management server 111 limits the scope of provision of the voice data of the conference according to the security level (confidentiality level) set for the conference held in the virtual space.

[0087] <Conference data control processing> 7 is a flowchart explaining the conference data control process by the virtual space management server 111 of the user environment monitoring system. The process of this flowchart is realized by the CPU 202 of the virtual space management server 111 reading and executing a program stored in the ROM 204, the HDD 205, or the like.

[0088] First, in S701, when the conference information management unit 312 of the virtual space management server 111 receives a conference participation request from one of the client terminals 121 to 132, the process proceeds to S702. In S702, the conference information management unit 312 checks whether the combination of the user ID and the conference ID included in the conference participation request exists in the conference information management table (Table 5) managed by the conference information management unit 312.

[0089] If the combination of the user ID and the conference ID included in the conference participation request does not exist in the conference information management table (No in S702), the conference information management unit 312 advances the process to S704. In S704, the conference information management unit 312 returns the authentication result as a failure to the client terminals 121 to 132, and then ends the process of this flowchart.

[0090] On the other hand, if the combination of the user ID and the conference ID included in the conference participation request exists in the conference information management table (Yes in S702), the conference information management unit 312 advances the process to S703. In S703, the conference information management unit 312 returns the authentication result to the client terminals 121 to 132 as a success, and advances the process to S705.

[0091] In S705, the conference information management unit 312 determines whether the confidentiality level of the conference ID acquired from the conference participation request received from the client terminals 121-132 is set to "high" in the conference information management table (Table 5). If it is determined that the confidentiality level is not "high" (No in S705), the conference information management unit 312 ends the process of this flowchart. In this case, the voice data providing unit 309 provides voice to users other than the conference participants, that is, all users logged in to the virtual space management server 111.

[0092] On the other hand, if it is determined that the confidentiality level is "high" (Yes in S705), the conference information management unit 312 advances the process to S706. In S706, the conference information management unit 312 controls so that audio sharing is not performed with anyone other than the conference participants of the user corresponding to the user ID included in the conference participation request. Specifically, the audio sharing destination of the user is changed to "conference participants" in the sharing state management table (Table 4) via the sharing state management unit 310. This causes the audio data providing unit 309 to enter a state in which audio is not provided to anyone other than the conference participants. Note that the audio sharing destination of the user changed here is changed from "conference participants" to "virtual space" when the conference ends.

[0093] Next, in S707, the conference information management unit 312 determines whether the screen mode in the sharing status management table (Table 4) for the user ID obtained from the conference participation request received from the client terminals 121 to 132 is “duplicate” (i.e., whether the screen content displayed on the user's PC is displayed in both the real space and the virtual space).

[0094] Here, if the conference information management unit 312 determines that the screen mode is not "copy" (No in S707), it ends the process of this flowchart. On the other hand, if it is determined that the screen mode is "copy" (Yes in S707), the conference information management unit 312 advances the process to S708.

[0095] In S708, the conference information management unit 312 controls so that the PC screen of the user is not displayed on the real space side. Specifically, the screen mode of the user in the sharing state management table of Table 4 is changed to "VR only" via the sharing state management unit 310 of the virtual space management server 111. This screen mode information is provided to the display unit 340 of the PC (client terminal 122, 132) of the user. For example, the user's PC periodically acquires the above screen mode information from the virtual space management server 111. In response to this, the display unit 340 of the PC of the user turns off the PC screen display. As a result, the display content of the PC screen of the user in the screen data providing unit 311 is in a state where it is displayed only on the virtual space side. After the process of S708, the conference information management unit 312 ends the process of this flowchart.

[0096] If the security (confidentiality) level preset for the conference is "high," the display contents on the screen may also be controlled so as not to be provided to anyone other than the conference participants. Through the above processing, depending on the security (confidentiality) level preset for the conference, it is possible to control so that the conference's audio data and screen display contents are not provided to anyone other than the conference participants, even if the user is participating in the same virtual space as the user being monitored.

[0097] According to each embodiment described above, while a user is wearing an HMD and working in a virtual office or the like, the user can appropriately grasp and respond to various situations that may occur in real space, such as people approaching, talking to them, conversations being overheard, and the screen of the PC being used by the user being viewed, while still wearing the HMD. As a result, a user who is participating in a virtual office wearing an HMD can work more comfortably in the metaverse space while maintaining security (reducing the risk of information leakage) and appropriately grasping the situation around him / her in real space.

[0098] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and the data may have various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied, for example, as a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system composed of multiple devices, or may be applied to an apparatus composed of a single device. Furthermore, any combination of the above-described embodiments is also included in the present invention.

[0099] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. Furthermore, the present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device. The present invention is not limited to the above-mentioned embodiment, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. In other words, the present invention includes all configurations that combine the above-mentioned embodiments and their modifications.

[0100] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A head-mounted display device that projects a virtual space into a user's field of vision, a notification means for notifying the user of the approach of a person in the user's field of vision when detecting that the user is approaching the virtual space while the virtual space is being projected in the user's field of vision; a control means for controlling, when the approach of the person is detected, to transition to a see-through display mode in which a real space can be confirmed through a virtual space in at least a part of the field of vision of the user; A head mounted display device comprising: (Configuration 2) The head mounted display device according to configuration 1, characterized in that in the control, the control means further stops displaying a screen in the real space of an information processing device whose screen the user is sharing with the virtual space. (Configuration 3) The head mounted display device according to configuration 1 or 2, characterized in that in the control, the control means further stops outputting to the virtual space of audio input via an audio input device for inputting the user's voice. (Configuration 4) The head mounted display device according to configuration 3, wherein the control means resumes output of the audio input via the audio input device to the virtual space in response to the cancellation of the see-through display mode. (Configuration 5) The head mounted display device according to any one of configurations 1 to 4, characterized in that the control means does not perform the control when the approaching person is a user participating in the same virtual space as the user. (Configuration 6) A head mounted display device as described in any one of configurations 1 to 5, characterized in that when it is detected that a person is approaching the user, the detection includes a case where the person continues to approach the user for a certain period of time or more, and a case where the gaze of the person approaching the user is directed toward the user for a certain period of time. (Configuration 7) The head mounted display device according to any one of configurations 1 to 5, wherein the control means cancels the see-through display mode in response to a gesture by the user or a physical operation on the head mounted display device. (Method 1) A method for controlling a head mounted display device that projects a virtual space into a user's field of vision, comprising: a notification step of notifying the user of the approach of a person in the user's field of view when detecting that the user is approaching the virtual space while the virtual space is being projected in the user's field of view; a control step of controlling, when the approach of the person is detected, to transition to a see-through display mode in which a real space can be confirmed through a virtual space in at least a part of the field of view of the user; A method for controlling a head mounted display device comprising: (Program 1) A program for causing a computer to execute the control method described in Method 1.

Claims

1. A head-mounted display device, A first control means that projects a virtual space into the field of view of the user wearing the head-mounted display device, A second control means that performs control to automatically display a person detected based on an image of the real world captured by the camera of the head-mounted display device in a part of the virtual space projected into the user's field of view, The head-mounted display device includes a receiving means for receiving user input operations to prevent the second control means from automatically displaying a person, A head-mounted display device characterized by having the following features.

2. The head-mounted display device according to Claim 1, characterized in that the second control means performs control to automatically display a person who has been detected to be approaching based on the image.

3. The head-mounted display device according to claim 1, further characterized in that the second control means stops outputting the voice input via the voice input device for inputting the user's voice to the virtual space side.

4. The head-mounted display device according to claim 3, characterized in that the second control means resumes outputting the audio input via the audio input device to the virtual space side in response to the acceptance of the input operation by the receiving means.

5. The head-mounted display device according to claim 1, characterized in that the input operation in the receiving means is received by an operation on an operation unit virtually displayed on the head-mounted display device that is linked to the user's finger movements in the real space.

6. The head-mounted display device according to claim 1, characterized in that the input operation in the receiving means is received by a physical operation on the head-mounted display device.

7. A control method for a head-mounted display device, A first control step involves projecting a virtual space into the field of view of a user wearing the head-mounted display device, A second control step involves executing a control that automatically displays a person detected based on an image of the real world captured by the camera of the head-mounted display device in a part of the virtual space projected into the user's field of view. The head-mounted display device includes a reception step for receiving user input operations to prevent the control in the second control step from automatically displaying a person, A control method characterized by having the following features.

8. The terminal is A first control means that projects a virtual space using a display into the field of view of the user wearing the terminal, A second control means that performs control to automatically display a person detected based on an image of the real world captured by the camera of the terminal in a part of the virtual space projected into the user's field of view, The terminal includes a receiving means for receiving user input operations to prevent the second control means from automatically displaying a person, A program designed to function as such.

9. The program according to claim 8, characterized in that the second control means performs control to automatically display a person who has been detected approaching based on the image.

10. The program according to claim 8, further characterized in that the second control means stops outputting to the virtual space side the voice input via the voice input device for inputting the user's voice.

11. The program according to claim 10, characterized in that the second control means resumes outputting the voice input via the voice input device to the virtual space side in response to the acceptance of the input operation by the receiving means.

12. The program according to claim 8, characterized in that the input operation in the receiving means is received by an operation on a virtually displayed operation unit that is linked to the user's finger movements in the real space.

Images