Head-mounted display device, control method therefor, and program

Abstract

To solve the following problem in which a user needs to put on or take off an HMD during image capturing of a physical object with a camera, and the operation for integrating a virtual object into a virtual space and sharing the virtual object is complicated when the user with the HMD working in the virtual space captures an image of the physical object in the real space to be integrated into the virtual space as a virtual object to share the virtual object with another user working within the same virtual space, which leads to deterioration in user experience.SOLUTION: A client terminal 121 serving as an HMD is configured to: project a virtual space in a field of view of a user wearing the HMD (S616, S618); enable see-through mode and execute a control so that a real space can be confirmed through the projected virtual space (S620); display a virtual object generated from imaging data related to a physical object in the real space, in the virtual space projected in the field of view of the user (S634); and receive sharing setting (S638). The virtual object set to be shared is shared with another member participating in the virtual space to allow the other member to view the virtual object.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a head mounted display device, a method thereof, and a program. [Background technology]

[0002] XR (cross reality) is a general term for technologies that create spaces that provide simulated experiences by fusing the real and virtual worlds, such as VR (virtual reality) and AR (augmented reality), and various standardization efforts are underway. In recent years, virtual spaces and services that use such technologies are called metaverses, and they are being used not only for entertainment purposes such as games, but also in business situations, such as virtual offices and VR conference rooms.

[0003] HMDs (Head Mounted Displays), devices for experiencing VR and AR, include fully immersive HMDs that block the wearer's view from real space, and see-through HMDs that provide a view that combines real space with virtual space and virtual objects. There are also HMDs that have both fully immersive and see-through HMD functions, and such HMDs allow you to switch the see-through mode on and off depending on the service or scene you are using.

[0004] A fully immersive HMD blocks the wearer's view from the real world, allowing them to be immersed in a virtual space. However, because they cannot see objects that exist in the real world (called physical objects), it is difficult to perform real-world actions such as picking up a nearby physical object while wearing a fully immersive HMD.

[0005] Patent document 1 proposes a technology that uses a camera installed in an HMD to capture images of physical objects near the wearer and projects images of the physical objects into a virtual space, enabling the wearer to act in the real world without losing the sense of immersion in the virtual space. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-26431 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology of Patent Document 1 simply projects an image of a physical object captured by a camera onto a virtual space displayed by an HMD worn by the user. The technology of Patent Document 1 does not allow a user to import a physical object in hand into the virtual space as a virtual object and to share the imported virtual object so that other users in the virtual space can also use it.

[0008] The present invention has been made to solve the above problems, and has an object to provide a mechanism that enables a physical object in the real world to be seamlessly incorporated into a virtual space with simple operations, without the need to attach or detach a head-mounted display, and to be shared with other users. [Means for solving the problem]

[0009] The present invention is a head-mounted display device comprising a display control means for projecting a virtual space into the field of view of a user wearing the head-mounted display device, and an execution means for enabling a see-through mode and executing control to allow a real space to be confirmed through the projected virtual space, wherein the display control means displays virtual objects generated from shooting data relating to physical objects in the real space within the virtual space projected into the field of view of the user, and the virtual objects are shared so that they can be viewed by other members participating in the virtual space. Effect of the Invention

[0010] According to the present invention, it is possible to seamlessly incorporate physical objects in the real world into a virtual space with simple operations, without having to put on or take off a head-mounted display, and to share the same with other users. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing the overall configuration of a virtual space management system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a hardware configuration diagram of a device constituting the virtual space management system of the embodiment. [Diagram 3] FIG. 2 is a software configuration diagram of the virtual space management system of the present embodiment. [Figure 4A] FIG. 2 is a diagram showing a display example of an HMD in the first embodiment. [Figure 4B] FIG. 2 is a diagram showing a display example of an HMD in the first embodiment. [Figure 4C] FIG. 2 is a diagram showing a display example of an HMD in the first embodiment. [Diagram 5] 3A to 3C are diagrams showing a method of operating the camera in the first embodiment. [Figure 6] FIG. 4 is a diagram showing an overall processing sequence in the first embodiment. [Figure 7A] FIG. 11 is a diagram showing a display example of an HMD in the second embodiment. [Figure 7B] FIG. 11 is a diagram showing a display example of an HMD in the second embodiment. [Figure 7C] FIG. 11 is a diagram showing a display example of an HMD in the second embodiment. [Figure 8] FIG. 11 is a diagram showing an overall processing sequence in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [First embodiment] FIG. 1 is a diagram showing an example of the overall configuration of a virtual space management system according to an embodiment of the present invention.

[0013] As shown in FIG. 1, in the virtual space management system of this embodiment, a virtual space management server 111, a client terminal 121, a client terminal 122, and a client terminal 131 are connected via networks 101 to 103.

[0014] The networks 101 to 103 are so-called communication networks realized by, for example, the Internet, LAN, WAN, telephone lines, dedicated digital lines, ATM or frame relay lines, cable television lines, wireless lines for data broadcasting, etc. The networks 101 to 103 may be any network capable of transmitting and receiving data. In this embodiment, the network 101 is the Internet, and the networks 102 and 103 are the Internet or wired or wireless LANs set in general household or company networks, etc. The wireless communication 104 is a communication means realized by a short-range wireless communication standard such as Bluetooth (registered trademark) or NFC (Near Field Communication), and performs data communication with the client terminals 121 and 122.

[0015] The client terminals 121 and 131 are, for example, dedicated hardware that supports the projection of virtual objects used in XR (cross reality), such as an HMD (head-mounted display) or smart glasses, or a mobile terminal with a built-in program execution environment, such as a smartphone.

[0016] Moreover, the client terminals 121 and 131 are equipped with a camera for capturing images of the surroundings and a display for projecting virtual objects. The client terminals 121 and 131 recognize the user's fingers through the camera and provide a simulated experience that fuses the real and virtual worlds by superimposing actions in the real space and actions in the virtual space. Furthermore, the client terminals 121 and 131 have a function for projecting not only the virtual space but also the whole or part of the real space on the display through the camera (hereinafter referred to as "see-through mode").

[0017] When the client terminals 121 and 131 are information processing devices that are not dedicated hardware such as HMDs (Head Mounted Displays), virtual objects are projected using APIs provided by a web browser, an OS (Operating System), etc. In this embodiment, the client terminals 121 and 131 will be described as HMDs.

[0018] The client terminal 122 is an information processing device having a photographing function (camera) such as a smartphone or tablet, and is hardware required for capturing physical objects into a virtual space. The client terminal 122 may be a dedicated photographing device such as a single-lens reflex camera or a 3D scanner capable of generating 3D data (three-dimensional data), but in this embodiment, the description will be given assuming that a mobile terminal such as a smartphone is used.

[0019] The virtual space management server 111 is a system for providing the client terminals 121, 131 with virtual objects in the virtual space, virtual objects that resemble each user (hereinafter referred to as "avatars"), and their position information. The virtual space management server 111 also has a function for managing the virtual objects and sharing status transmitted from the client terminals 121, 131, etc. The virtual space management server 111 also manages the users who use the client terminals 121, 131. That is, the virtual space management server 111 receives login / logout requests from the client terminals 121, 131 and performs login / logout processing.

[0020] The virtual space management server 111 is constructed using a server computer. Alternatively, the virtual space management server 111 can be constructed by adopting cloud computing technology or the like. The functions of the virtual space management server 111 described in this embodiment may be realized by a single server or a single virtual server, or may be realized by multiple servers or multiple virtual servers. Alternatively, multiple virtual servers may be executed on a single server.

[0021] FIG. 2 is a diagram showing an example of the hardware configuration of the virtual space management server 111 and the client terminals 121, 122, and 131. As shown in FIG. 2, a CPU (Central Processing Unit) 202 controls the entire device. The CPU 202 executes various programs such as application programs and an OS (Operating System) stored in an HDD / SSD 205, and at this time, controls the temporary storage of information, files, etc. required for the execution of the programs in a RAM 203. A GPU (Graphics Processing Unit) 210 performs the calculations required to project virtual objects and avatars in real time.

[0022] A RAM (Random Access Memory) 203 is a temporary storage unit, and functions as a main memory, a work area, etc. for the CPU 202 and the GPU 210 . A ROM (Read Only Memory) 204 is a storage unit, and stores various data such as a basic I / O program therein. The HDD / SSD 205 is one of the external storage units. Note that HDD is an abbreviation for "Hard Disk Drive" and SSD is an abbreviation for "Solid State Drive." The HDD / SSD 205 functions as a large-capacity memory and stores application programs such as a web browser, programs for service servers, an OS, related programs, and the like.

[0023] The display 206 is a display unit that displays virtual objects, information necessary for operations, etc. The display 206 may also be a device that has a function of receiving operation instructions from a user, such as a touch panel. The camera 207 is an out-camera in the client terminals 121, 122, and 131 that captures images of the surroundings, or an in-camera that mainly captures images of oneself. By analyzing images captured by the camera 207, particularly the out-camera, with an application program stored in the HDD / SSD 205, it is possible to synchronize the movements of one's own fingers in the real space with the fingers of one's own avatar in the virtual space. In addition, the application program stored in the HDD / SSD 205 synthesizes and processes the photos and images (multiple images captured from multiple angles as described later) captured by the camera 207 of the client terminal 122. This makes it possible to generate virtual objects that can be handled by the client terminals 121 and 131 and the virtual space management server 111. Note that the virtual space management server 111 does not necessarily require the camera 207.

[0024] The interface 208 is an external device I / F that connects peripheral devices such as various external sensors. The camera 207 mentioned above recognizes the user's fingers in the real space, making it possible to operate virtual objects in the virtual space and touch other avatars, but the same functions can also be achieved by operating a dedicated controller connected to the interface 208.

[0025] The system bus 201 controls the flow of data within the device. A NIC (Network Interface Card) 209 exchanges data with external devices via the networks 101-102. The proximity communication I / F 211 is a network I / F for proximity communication such as NFC or BlueTooth. By using the proximity communication I / F 211, data can be transmitted and received between each client terminal.

[0026] The above computer configuration is an example, and is not limited to the configuration example in Fig. 2. For example, the storage destination of data and programs can be changed to RAM 203, ROM 204, HDD / SSD 205, etc. depending on the characteristics of the data and programs. In addition, the CPU 202 and GPU 210 execute processing based on the programs stored in the HDD / SSD 205, thereby realizing processing in the software configuration as shown in Fig. 3.

[0027] FIG. 3 is a diagram for explaining the software configuration of the virtual space management server 111 and the client terminals 121, 122, and 131, extracting functions related to this embodiment. <Client terminal 122> The client terminal 122 includes a display unit 332, a connection request unit 334, an image capture control unit 336, an image capture data generation unit 338, and an image capture data transmission unit 340, and generates 3D data by capturing an image of a physical object in real space to be captured as a virtual object. Each unit of the client terminal 122 shown in Fig. 3 is realized by the CPU 202 of the client terminal 122 reading and executing a program stored in the ROM 204 or HDD / SSD 205. Each unit will be described in detail below.

[0028] The display unit 332 is a software module for displaying on the display 206 the real space captured via the camera 207 and a UI (user interface) for selecting a shooting mode, which will be described later. The connection request unit 334 issues a connection request to the connection processing unit 322 of the client terminal 121 via the near field communication IF 211 in order to exchange data with the client terminal 121, which will be described later.

[0029] The imaging control unit 336 is a software module that instructs the camera 207 to start up the camera and take pictures, and also has a function of switching between imaging modes, which will be described later, for example. The shooting data generation unit 338 is a software module for generating 3D data by synthesizing images of real space captured by the camera 207 (multiple images captured from multiple angles as described below). The shooting data generation unit 338 also generates 3D data and acquires feature amounts (the physical object itself and the background, objects on the table, patterns, etc.) of the surrounding space including the physical object being captured from the captured image. The shooting data generation unit 338 associates the generated 3D data with the feature amounts and converts them into data. This allows the virtual object generation unit 326 of the client terminal 121 to generate a virtual object from this 3D data, not only by simply projecting the generated virtual object onto the display 206, but also by projecting it in accordance with the shooting position.

[0030] This technology is called a spatial anchor, and when a target virtual object is projected on a different terminal, the virtual object can be placed at a position that matches the feature amount in space. In this embodiment, this technology is used to determine the projection position when projecting 3D data of a physical object photographed and generated by the client terminal 122 onto the display 206 of the client terminal 121. Hereinafter, the 3D data generated by the photographing data generation unit 338 and the feature amount associated with it are collectively referred to as "photographing data". The photographing data transmission unit 340 transmits the photographing data generated by the photographing data generation unit 338 to the photographing data reception unit 324 of the client terminal 121.

[0031] <Client terminal 121> The client terminal 121 includes a display unit 312, a login unit 314, a virtual object information transmission / reception unit 316, a local virtual object management unit 318, a see-through mode switching unit 320, and a connection processing unit 322 as basic functions for realizing a virtual space. The client terminal 121 further includes a shooting data receiving unit 324, a virtual object generation unit 326, and a shooting data sharing unit 328. Each unit of the client terminal 121 shown in Fig. 3 is realized by the CPU 202 of the client terminal 121 reading and executing a program stored in the ROM 204 or the HDD / SSD 205. Each unit will be described in detail below.

[0032] The display unit 312 is a software module for controlling the display of an operation UI, virtual objects and avatars in a virtual space, and images of the real space captured by the camera 207 of the client terminal 121, which is an HMD, on the display 206 of the client terminal 121.

[0033] The login unit 314 transmits a finger image captured by the camera 207 of the client terminal 121, or a username and password input by an input device connected to the interface 208, to the login processing unit 354 of the virtual space management server 111. Note that the authentication method for the virtual space management server 111 may be face authentication using a face image captured by the camera 207 of the client terminal 121, iris authentication using the iris, fingerprint authentication using a fingerprint sensor connected to the interface 208, or the like.

[0034] The virtual object information transmitting / receiving unit 316 acquires 3D data and position information of virtual objects and avatars from the virtual object information transmitting / receiving unit 358 of the virtual space management server 111, and stores the data in the local virtual object management unit 318. The virtual object information transmitting / receiving unit 316 also has a function of transmitting position information of virtual objects and avatars that have changed as a result of an operation by the user on the client terminal 121 to the virtual object information transmitting / receiving unit 358 of the virtual space management server 111.

[0035] The local virtual object management unit 318 manages the 3D data and position information of virtual objects and avatars acquired from the virtual object information transmission / reception unit 358 of the virtual space management server 111. Furthermore, the local virtual object management unit 318 also has a function of detecting the positions of virtual objects and avatars on the client terminal 121 that have changed due to an operation of the client terminal 121, and storing the position information in the local virtual object management unit 318 itself. The substance of the data managed by the local virtual object management unit 318 is stored in, for example, the RAM 203 or HDD / SSD 205 of the client terminal 121.

[0036] Using the functions of the virtual object information transmission / reception unit 316 and the local virtual object management unit 318, the behavior and state of virtual objects and avatars in the virtual space provided by the virtual space management server 111 can be reflected in real time on the display 206 of the client terminal 121.

[0037] The see-through mode switching unit 320 is a software module that instructs the display unit 312 whether or not to project an image of real space captured via the camera 207 of the client terminal 121 onto the display 206 of the client terminal 121. In this embodiment, it is assumed that the switching of the see-through mode is initiated by a user operation, but the condition that is the starting point is not particularly limited. For example, the switching can be initiated by a user turning a hardware switch provided in the client terminal 121 on / off, or by gesture or voice assistance, or by automatically detecting a physical object.

[0038] The connection processing unit 322 processes a connection request from a connection request unit 334 of the client terminal 122 , and after permitting the connection, exchanges data with the client terminal 122 . The image capturing data receiving unit 324 receives the image capturing data transmitted from the image capturing data transmitting unit 340 of the client terminal 122 .

[0039] The virtual object generation unit 326 generates a virtual object that can be projected in a virtual space projected on the client terminal 121 based on the shooting data received by the shooting data receiving unit 324, and determines the projection position. The virtual object generation unit 326 uses the mechanism of the spatial anchor described above to detect a position where the feature amount included in the shooting data shot via the camera 207 of the client terminal 121 is similar to the feature amount of the real space. At this time, the see-through mode of the HMD of the client terminal 121 is ON (enabled), and an image of the real space is visible via the camera 207 of the client terminal 121. When the virtual object generation unit 326 detects a position where the feature amount included in the shooting data is similar, it instructs the display unit 312 to project a virtual object generated based on the 3D data included in the shooting data to the detected position.

[0040] By performing such processing, the virtual object generation unit 326 can not only treat the photographed physical object as a virtual object in the virtual space, but also determine the projection position and capture it. Furthermore, since it is possible to project the virtual object so that it is superimposed on the physical object as seen from the user wearing the HMD, the user can experience an effect in which the photographed physical object appears as if it were floating up as a virtual object. Note that a specific sequence of this series and a display example on the display 206 of the client terminal 121 will be described later.

[0041] The photographed data sharing unit 328 is a software module for sharing the photographed data received by the photographed data receiving unit 324 with other users (members) participating in the virtual space, and transmits the photographed data and sharing settings to the shared photographed data receiving unit 360 of the virtual space management server 111. Hereinafter, the photographed data and sharing settings will be collectively referred to as "shared photographed data."

[0042] Although the software configuration of the client terminals 121 and 122 has been described above, the client terminal 122 may be embedded in the client terminal 121. In other words, by providing the client terminal 121 with the functions of the client terminal 122, the above-mentioned functions can be realized by the hardware of the client terminal 121 alone.

[0043] <Virtual Space Management Server 111> The virtual space management server 111 includes a user management unit 352, a login processing unit 354, a virtual object management unit 356, and a virtual object information transmission / reception unit 358 as basic functions for providing a virtual space. Furthermore, the virtual space management server 111 includes a shared captured data receiving unit 360, a shared captured data management unit 362, a captured data providing unit 364, and a collaboration app management unit 366 described in a second embodiment. Each unit of the virtual space management server 111 shown in Fig. 3 is realized by the CPU 202 of the virtual space management server 111 reading and executing a program stored in the ROM 204 or HDD / SSD 205. Each unit will be described in detail below.

[0044] The user management unit 352 manages user information and login information. The login processing unit 354 accepts a login request from the client terminal 121, 131, performs login processing by checking against data managed by the user management unit 352, and returns the login processing result to the client terminal 121, 131. Table 1 below shows an example of data managed by the user management unit 352.

[0045] [Table 1]

[0046] The user management unit 352 manages data using a user management table such as Table 1. In the user management table shown in Table 1, the user ID column is an ID that uniquely identifies a user. The password column is a basic authentication password used when logging in with the user ID. The login processing unit 354 checks the combination of user ID and password included in the login request from the client terminals 121 and 131 against Table 1, and if they match, returns the login result to each client terminal as a success. On the other hand, if they do not match, returns the login result to each client terminal as a failure. The login status column indicates the login status of the user, with "on" indicating a logged-in status and "off" indicating a logged-out status. The login expiration date column indicates the expiration date of the authentication status of the logged-in user.

[0047] The virtual object management unit 356 manages virtual objects placed in the virtual space, 3D data of each user's avatar, and their position information in the virtual space. The virtual object information transmitting / receiving unit 358 has a function of providing the virtual objects and avatars managed by the virtual object management unit 356, their position information, and the like to the client terminals 121 and 131. Furthermore, the virtual object information transmitting / receiving unit 358 also has a function of receiving position information from the client terminals 121 and 131 periodically or when the positions of the virtual objects and avatars in the virtual space are changed by an operation from the client terminals 121 and 131.

[0048] Table 2 shows an example of the position information of the avatar of each user managed by the virtual object management unit 356. [Table 2]

[0049] The virtual object management unit 356 manages the avatar position information using an avatar position information management table such as that shown in Table 2. In the avatar position information management table of Table 2, the user ID column is an ID that uniquely identifies a user. The space ID column is an ID that uniquely identifies a virtual space. The main coordinate column is information indicating the current position (center of gravity) of the avatar. The left hand coordinate column and the right hand coordinate column are information indicating the positions of the avatar's left and right hands, and can also be used as information for determining the contact state with a virtual object or an avatar.

[0050] Table 3 shows an example of location information of a virtual object. [Table 3]

[0051] The virtual object management unit 356 manages the virtual object position information using a virtual object position information management table such as Table 3. In the virtual object position information management table of Table 3, the virtual object ID column is an ID that uniquely identifies a virtual object in the virtual space. The space ID column and main coordinate column are the same as those explained in Table 2. In this way, the client terminals 121, 131 and the virtual space management server 111 periodically exchange and manage position information, so that the positions and actions of virtual objects and avatars in the virtual space can be reflected in real time.

[0052] The shared photographed data receiving unit 360 is a software module that receives the shared photographed data from the photographed data sharing unit 328 of the client terminal 121 . The shared photographed data management unit 362 manages the shared photographed data received by the shared photographed data receiving unit 360 .

[0053] Table 4 shows an example of a shared photographic data management table managed by the shared photographic data management unit 362. [Table 4]

[0054] The shared photographing data management unit 362 manages the shared photographing data using a shared photographing data management table such as Table 4. In the shared photographing data management table of Table 4, the share ID column is an ID that uniquely identifies the sharing setting. The sharing source user column is an ID that uniquely identifies the sharing source user who is the owner of the photographing data. The sharing destination user column is an ID that uniquely identifies the user to whom the photographing data is shared. The photographing data ID column is an ID that uniquely identifies the photographing data received by the shared photographing data receiving unit 360, and the shared photographing data management unit 362 manages the photographing data body by linking it to this photographing data ID.

[0055] The photographed data providing unit 364 checks the photographed data against the sharing settings and provides it to the client terminal 131. For example, assume that "userA" is logged in to the client terminal 121 and "userB" is logged in to the client terminal 131, and "userA" has set photographed data ID "capturedDataA" to be shared with "userB". At this time, a virtual object with photographed data ID "capturedDataA" is projected onto the display 206 of the client terminal 131 to which "userB" is logged in. Note that, unless otherwise specified, in this embodiment, the client terminal 121 side will be described as a terminal operated by a user who shares photographed data of a physical object (a sharing source user), and the client terminal 131 side will be described as a terminal operated by a user who will receive the photographed data (a sharing destination user).

[0056] The collaboration app management unit 366 is software that provides a service for users to communicate with each other (for example, a collaboration service having a message posting function), and has a function of providing a means of communication such as chat. The collaboration app management unit 366 has a function of managing chat messages and attachments from external client terminals and viewing them for each user. The collaboration app management unit 366 also has a function of returning content corresponding to the user who has logged in to the client terminal 131 in response to a chat content acquisition request from the collaboration app control unit 384. The collaboration app management unit 366 will be used in the explanation of the second embodiment.

[0057] The entities of the data managed by the above-mentioned user management unit 352, virtual object management unit 356, shared shooting data management unit 362, and collaboration app management unit 266 are stored, for example, in the RAM 203 or HDD / SSD 205 of the virtual space management server 111.

[0058] <Client terminal 131> The client terminal 131 includes a display unit 372, a login unit 374, a virtual object information transmission / reception unit 376, and a local virtual object management unit 378 as basic functions for realizing a virtual space. The client terminal 131 further includes a shooting data acquisition unit 380, a virtual object generation unit 382, ​​and a collaboration application control unit 384, which are features of the present invention. Each unit of the client terminal 131 shown in Fig. 3 is realized by the CPU 202 of the client terminal 131 reading and executing a program stored in the ROM 204 or HDD / SSD 205. Each unit will be described in detail below.

[0059] The display unit 372 has a function equivalent to the display unit 312 of the client terminal 121. The login unit 374 has a function equivalent to the login unit 314 of the client terminal 121. The virtual object information transmission / reception unit 376 has a function equivalent to the virtual object information transmission / reception unit 316 of the client terminal 121. The local virtual object management unit 378 has a function equivalent to the local virtual object management unit 318 of the client terminal 121.

[0060] The photographic data acquisition unit 380 has a function of acquiring photographic data from the photographic data provision unit 364 of the virtual space management server 111, but can only acquire photographic data that has been set as shared, as described above.

[0061] The virtual object generation unit 382 has the same function as the virtual object generation unit 326, and has a function of generating a virtual object that can be projected into a virtual space projected on the client terminal 131 based on the shooting data acquired from the shooting data acquisition unit 380. However, the client terminals 121 and 131 may be used in different real spaces. In that case, the virtual object generation unit 326 generates a virtual object so as to be superimposed on a photographed physical object using a spatial anchor, but it is difficult for the virtual object generation unit 382 to use the function of the spatial anchor. In that case, the virtual object is projected onto an appropriate position (for example, a table in the virtual space) in the virtual space projected on the display 206 of the client terminal 131.

[0062] Collaboration application control unit 384 is software that controls services provided by collaboration application management unit 366 so that they can be used by client terminal 131, and is a dedicated application or a Web browser. Collaboration application control unit 384 is used together with collaboration application management unit 366 in the second embodiment.

[0063] In the above explanation, a case has been shown in which the photographed data is transmitted from the client terminal 122 to the client terminal 121 and displayed thereon, but a configuration in which the photographed data is transmitted from the client terminal 122 to the client terminal 121 via the virtual space management server 111 and displayed thereon is also possible.

[0064] Next, a method of importing a physical object in the real space into a virtual space as a virtual object and sharing it with other users (avatars) in the same virtual space will be described with reference to Figures 4A to 4C, 5, and 6. Hereinafter, Figures 4A to 4C will be collectively referred to as "Figure 4."

[0065] 4(a) to 4(e) are diagrams showing an example of an image of a virtual space projected on the display 206 of the client terminal 121. FIG. FIG. 4( f ) is a diagram showing an example of an image in the virtual space displayed on the display 206 of the client terminal 131 . 5(a) and 5(b) are diagrams showing an example of a method of operating the client terminal 122 for capturing a physical object in real space as a virtual object. FIG. 6 is a diagram showing an example of an overall processing sequence in the first embodiment.

[0066] FIG. 4(a) shows an image of a virtual space projected onto the display 206 of the client terminal 121 when the see-through mode is OFF. Reference numeral 412 denotes a display on the display 206 of the client terminal 121 . Virtual object 414 is a work monitor projected into the virtual space. Avatars 416 and 418 are avatars of other users who exist (participate) in the same virtual space as the user operating client terminal 121. Virtual object 420 is a desk placed in the virtual space.

[0067] FIG. 4(b) shows an image projected onto the display 206 of the client terminal 121 when the see-through mode is ON. Reference numeral 432 denotes a desk that exists in the real space of the user who operates the client terminal 121. Reference numeral 434 denotes a physical object to be captured into the virtual space. When the see-through mode is turned ON, both the real space and the virtual object are projected onto the display 206, but it is possible to set in advance the virtual object to be projected when the see-through mode is turned ON.

[0068] 4(c) shows an image in which a user operating the client terminal 121 with the see-through mode in ON state captures an image of a physical object 434. FIG. 4(c) will be described below in conjunction with FIG. Reference numeral 442 denotes a client terminal 122 connected to the client terminal 121, and it is assumed that the client terminal 121 and the client terminal 122 are operated by the same user.

[0069] Fig. 5(a) is a UI displayed on the display 206 of the client terminal 122. Reference numeral 512 denotes the client terminal 122. Before photographing a physical object in Fig. 4(c), a photographing mode is selected on the UI as shown in Fig. 5(a). At this time, since the see-through mode of the client terminal 121 is ON, the user operating the client terminal 121 can operate the client terminal 122 even while wearing the HMD.

[0070] Button 514 is a button that is pressed when a user wishes to capture a physical object as 3D data, i.e., a three-dimensional virtual object, into a virtual space. In the first embodiment, the description will proceed on the assumption that a user presses button 514 to capture a physical object as 3D data. When buttons 516 and 518 are pressed, photographed data is generated as image data and text data of the physical object, respectively. The operations performed when buttons 516 and 518 are pressed will be described in the second embodiment.

[0071] FIG. 5(b) is a diagram showing how a user photographs a physical object after pressing button 514. Reference numeral 512 denotes the client terminal 122, which is the same as 442 in Fig. 4(c) . Similarly, reference numeral 532 denotes a physical object to be photographed, which is the same as 434 in Fig. 4(b) .

[0072] As shown in 512 of FIG. 5(b), a physical object 532 is photographed from multiple angles, and the photographed data generating unit 338 of the client terminal 122 generates 3D data and acquires the feature amount of the space around the physical object. In this embodiment, the means for generating 3D data is not particularly limited, and a means that can be realized by a general smartphone is assumed. For example, it can be realized by using means such as photogrammetry or LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0073] FIG. 4(d) shows an example of a display when a virtual object is generated by a user's operation as described in FIG. 5(b) and projected onto the display 206 of the client terminal 121 with the see-through mode ON. Reference numeral 452 denotes a virtual object obtained by capturing a photograph of a physical object and projecting it into a virtual space.

[0074] 5(b), the image capturing data generating unit 338 of the client terminal 122 generates image capturing data, and the image capturing data is transmitted to the client terminal 121 via the image capturing data transmitting unit 340. Then, the client terminal 121 generates a virtual object in the virtual object generating unit 326 from the 3D data included in the image capturing data received by the image capturing data receiving unit 324. After that, the virtual object generating unit 326 detects a position similar to a feature amount in the real space from the feature amount included in the image capturing data, and instructs the display unit 312 to project the virtual object onto the display 206 by superimposing it on the captured physical object. By the client terminal 121 performing the above-mentioned processing, the user can seamlessly capture the captured physical object into the virtual space.

[0075] FIG. 4(e) shows an operation of sharing a virtual object 452 with another user in the same virtual space on a client terminal 122 in which the see-through mode is ON. Reference numeral 462 denotes a virtual object showing the fingers of a user avatar operating client terminal 121, or an image of actual fingers. The user operating client terminal 121 performs a sharing operation by grabbing virtual object 452 with the fingers and dropping it onto the avatar of the user with whom the user wishes to share (avatar 416 in this example). Virtual object 464 is a virtual object that is displayed when sharing settings are made by the above operation.

[0076] Although the above method has been given as an operation for sharing a virtual object with other users, the operation method is not limited to this. For example, a method may be used in which a virtual object to be shared is tapped to display a UI such as a user list, and a user to be set as a sharing destination is selected from the list. Furthermore, the list may include not only a single user but also multiple users, or an option to share with everyone in the virtual space, making it a selection method that can be supplied to multiple users.

[0077] 4(f) shows an image of the virtual space projected on the display 206 of the client terminal 131. In this embodiment, the client terminal 131 does not take images of physical objects, so there is no need to switch the see-through mode to ON.

[0078] Reference numeral 472 denotes a display on the display 206 of the client terminal 131. Reference numeral 474 denotes an avatar of a user who operates the client terminal 121, who is the sharing source user. Reference numeral 476 denotes a virtual object projected onto the display 206 of the client terminal 131 as a result of the virtual object 452 being shared by the client terminal 121.

[0079] Next, the overall processing sequence will be described with reference to Fig. 6. The processing of the client terminal 121 shown in Fig. 6 is realized by the CPU 202 of the client terminal 121 reading and executing a program stored in the ROM 204 or HDD / SSD 205. The processing of the client terminal 122 is realized by the CPU 202 of the client terminal 122 reading and executing a program stored in the ROM 204 or HDD / SSD 205. The processing of the virtual space management server 111 is realized by the CPU 202 of the virtual space management server 111 reading and executing a program stored in the ROM 204 or HDD / SSD 205. The processing of the client terminal 131 is realized by the CPU 202 of the client terminal 131 reading and executing a program stored in the ROM 204 or HDD / SSD 205.

[0080] First, in S612, the login unit 314 of the client terminal 121 transmits a user ID and a password to the login processing unit 354 of the virtual space management server 111 in response to a login operation by the user. When the login processing unit 354 of the virtual space management server 111 receives the user ID and password, in S614, it confirms that the received user ID and password match the user ID and password managed by the user management unit 352, and if they match, returns a login result indicating that the login was successful.

[0081] When the login processing unit 354 of the client terminal 121 receives information indicating successful login as a login result, the process proceeds to S616. In S616, the virtual object information transmitting / receiving unit 316 of the client terminal 121 makes a request to the virtual object information transmitting / receiving unit 358 of the virtual space management server 111 to acquire content for constructing the virtual space.

[0082] When the virtual object information transmitting / receiving unit 358 of the virtual space management server 111 receives the request to acquire the content for constructing the virtual space, the process proceeds to S618. In S618, the virtual object information transmitting / receiving unit 358 of the virtual space management server 111 returns the contents such as the virtual object in response to the acquisition request in S616. The virtual object information transmitting / receiving unit 316 , which acquires the content from the virtual space management server 111 , stores the content, such as a virtual object, in the local virtual object management unit 318 . By asynchronously and repeatedly executing steps S616 and S618 as described above, the client terminal 121 can reflect changes in the virtual space on the display 206 in real time.

[0083] In S620, for example, in response to a see-through mode switching operation from the user, the see-through mode switching unit 320 of the client terminal 121 turns on the see-through mode. That is, the see-through mode switching unit 320 instructs the display unit 312 to project an image of the real space that can be acquired via the camera 207 of the client terminal 121 onto the display 206 of the client terminal 121. This realizes control that allows the real space to be confirmed through the virtual space projected onto the display 206. In S622 , in response to an operation of the client terminal 122 by the user, the connection request unit 334 of the client terminal 122 makes a connection request to the connection processing unit 322 of the client terminal 121 . When the connection processing unit 322 of the client terminal 121 receives the connection request, in S624, it permits the connection request and returns a response to the client terminal 122.

[0084] When the response of connection permission is received, the connection request unit 334 of the client terminal 122 advances the process to S626. 5, the image capturing data generating unit 338 of the client terminal 122 sets the camera 207 to an image capturing state via the image capturing control unit 336 to start capturing an image of the physical object, and continues capturing an image until data necessary for generating 3D data can be acquired. Then, when the image capturing data generating unit 338 determines that data necessary for generating 3D data has been acquired, it stops capturing an image and generates image capturing data that combines the 3D data of the physical object and the feature amount of the surrounding space including the physical object.

[0085] Next, in S630, the photographing data transmitting unit 340 of the client terminal 122 transmits the photographing data generated in S626 to the photographing data receiving unit 324 of the client terminal 121. When the photographing data receiving unit 324 of the client terminal 121 receives the photographing data, in S632, it returns a response to the client terminal 122 notifying the client terminal 122 that the photographing data has been received.

[0086] Next, in S634, the virtual object generation unit 326 of the client terminal 121 generates a virtual object that can be projected into a virtual space from the 3D data included in the captured data, determines a projection position from the feature amount of the space, and projects the virtual object onto the display 206 via the display unit 312. When the processing of S634 is completed, the state shown in FIG. 4(d) is reached.

[0087] When a sharing operation is performed by the user, in S638, the photographed data sharing unit 328 of the client terminal 121 generates shared photographed data in which the photographed data to be shared and the sharing destination user are set. Thereafter, in S640, the photographed data sharing unit 328 of the client terminal 121 transmits the shared photographed data generated in S638 to the shared photographed data receiving unit 360 of the virtual space management server 111. When the shared photographing data receiving unit 360 of the virtual space management server 111 receives the shared photographing data, in S642, the shared photographing data receiving unit 360 stores the received shared photographing data in the shared photographing data management unit 362 and returns a response to the effect that the shared photographing data has been received to the photographing data sharing unit 328 of the client terminal 121. When the process of S642 is completed, the state shown in Fig. 4(e) is reached.

[0088] Next, a description will be given of the processing on the side of the client terminal 131 operated by the share destination user. Note that S644 to S650 are the same as the processing of S612 to S618 except that the client terminal operated is different, and therefore a description thereof will be omitted. In S652, the photographing data acquisition unit 380 of the client terminal 131 makes a request to the photographing data provision unit 364 of the virtual space management server 111 to acquire photographing data.

[0089] At the stage when a request to acquire photographic data is made in S652 above, it is not clear on the client terminal 131 side whether sharing settings have been made or not. When the photographing data providing unit 364 of the virtual space management server 111 receives the request to acquire the photographing data, the process proceeds to S654.

[0090] In S654, the photographing data providing unit 364 of the virtual space management server 111 checks whether the logged-in user of the client terminal 131 that requested the acquisition of the photographing data is included in the shared user list of the shared photographing data management table shown in Table 4. Next, in S656, the photographed data providing unit 364 of the virtual space management server 111 returns all photographed data set as sharing destination users based on the result of the comparison in S654. Note that, if the photographed data providing unit 364 determines in S654 that there is no photographed data set as sharing destination users, it does not return any photographed data in S656, but returns only a response to that effect.

[0091] When the photographing data acquisition unit 380 of the client terminal 131 receives the response from the virtual space management server 111, the process proceeds to S658. In S658, if the shooting data acquisition unit 380 was able to acquire at least one piece of shooting data in S656, the virtual object generation unit 382 of the client terminal 131 generates a virtual object that can be projected into a virtual space based on the acquired shooting data. Then, the virtual object generation unit 382 additionally projects the generated virtual object onto the display 206 of the client terminal 131 via the display unit 372.

[0092] As described above, the client terminals 121 and 131 may be used in different real spaces, and therefore the virtual object generation unit 382 of the client terminal 131 may not be able to determine the projection position using a spatial anchor. In that case, the virtual object generation unit 382 of the client terminal 131 determines the projection position so as to project the virtual object onto an appropriate position (for example, a tabletop in the virtual space). When the process of S658 is completed, the state shown in FIG. 4(f) is reached. The processes of S652 to S658 are executed asynchronously and repeatedly, similar to the processes of S648 and S650.

[0093] Note that some of the processes of the virtual space management server 111, the client terminal 121, and the client terminal 131 described in this embodiment can be realized on each other's server and client terminal. For example, the process of generating a virtual object from the photographed data executed by the virtual object generating unit 382 may generate a virtual object in advance on the virtual space management server 111 side so that the virtual object can be projected immediately after acquisition on the client terminal to which the photographed data is shared. Also, the processes of S654 and S656 of the virtual space management server 111 can be realized on the client terminal 121. For example, a configuration is conceivable in which the client terminals 121 and 131 work in the same real space and share the photographed data by directly communicating with each other without going through the virtual space management server 111. In that case, the client terminal 121 has the functions of the shared photographed data receiving unit 360, the shared photographed data managing unit 362, and the photographed data providing unit 364 of the virtual space management server 111, thereby realizing the sharing of the photographed data. That is, the processing described in this embodiment can also be realized in a cooperative manner, with the virtual space management server 111, the client terminal 121, and the client terminal 131 each sharing a portion of the processing.

[0094] By configuring the client terminals 121, 122, 131 and the virtual space management server 111 as described above, it is possible to seamlessly incorporate physical objects from the real world into the virtual space with simple operations, without having to put on or take off a head-mounted display, and further to share this with other users.

[0095] For example, in a virtual office or a VR conference, there are cases where a user wants to take a physical object in hand as a virtual object into a virtual space and share the virtual object so that other users in the virtual space can also handle it. In such a case, it is necessary to generate a virtual object from the physical object and make it possible to handle it in the virtual space, rather than simply projecting a physical object photographed by a camera onto the virtual space displayed by the HMD worn by the user. In addition, in order to generate a virtual object from a physical object, it is necessary to use a camera to photograph the physical object from multiple angles and generate three-dimensional 3D data. However, in the conventional technology, when photographing a physical object with a camera, the HMD must be attached and detached, and the operation for capturing and sharing the virtual object is complicated, resulting in a poor user experience. In contrast, in this embodiment, with the configuration as described above, a physical object in the real space can be easily captured into the virtual space and shared with other users without complicated operations such as attaching and detaching the head-mounted display. In other words, a virtual space management system can be provided in which a physical object in the real space photographed by a camera can be seamlessly captured into the HMD worn by the user, and the virtual object can be easily shared with other users.

[0096] Second Embodiment In the first embodiment, a method was described in which a three-dimensional physical object (3D data) based on images taken from multiple angles by a camera is captured into a virtual space and shared with other users who exist in the same virtual space. Note that there may be cases in which users wish to share not only 3D data but also planar physical objects (2D data) such as images and text data with other users. In that case, as in the first embodiment, it is possible to capture the data as a virtual object into the virtual space, but in the case of 2D data, it may be more convenient to handle it in a form that can be handled by a PC or a chat app, for example, rather than as a virtual object. In the second embodiment, as an application example of the method described in the first embodiment, a method is described in which a planar physical object in the real space is imported into a virtual space and shared with other users by posting it in a chat provided by the collaboration app management unit 366.

[0097] The configuration of the second embodiment will be described in detail below with reference to Figures 7A to 7C and 8. Hereinafter, Figures 7A to 7C will be collectively referred to as "Figure 7." 7(a) to 7(d) are diagrams showing an example of an image of a virtual space projected onto the display 206 of the client terminal 121 in the second embodiment. FIG. 7(e) is a diagram showing an example of an image in a virtual space displayed on the display 206 of the client terminal 131 in the second embodiment. FIG. 8 is a diagram showing an example of an overall processing sequence in the second embodiment.

[0098] FIG. 7(a) shows an image of a virtual space projected onto the display 206 of the client terminal 121 in which the see-through mode is ON. Reference numeral 712 denotes a display on the display 206 of the client terminal 121 . The avatar 714 is an avatar of another user who exists in the same virtual space as the user operating the client terminal 121 . Reference numeral 716 denotes a desk that exists in the real space of the user who operates the client terminal 121 . Reference numeral 718 denotes a physical object to be captured into the virtual space, which is a printed matter capable of being subjected to OCR (Optical Character Recognition) in this case.

[0099] FIG. 7(b) is an image in which a user operating a client terminal 121 with the see-through mode turned on is photographing a physical object 718, similar to FIG. 4(c). Reference numeral 722 denotes a client terminal 122 connected to the client terminal 121, and the client terminal 121 and the client terminal 122 are operated by the same user.

[0100] In the second embodiment, when photographing with the client terminal 122, the button 516 or the button 518 in FIG. 5(a) is pressed to photograph. By pressing the button 516 or the button 518 to photograph, the photographing data generating unit 338 generates photographing data of the physical object as image data or text data, respectively. In detail, when the button 516 is pressed (when the photographing mode is "image"), the photographing data generating unit 338 photographs the physical object and generates photographing data including image data. Also, when the button 518 is pressed (when the photographing mode is "text"), the photographing data generating unit 338 generates photographing data including text data obtained by character recognition by OCR processing from the image data generated by photographing the physical object. Note that, since the physical object is captured as 2D data in the second embodiment, it is not necessarily necessary to photograph it from multiple angles as in FIG. 5(b). Also, in the second embodiment, the description will proceed assuming that the user presses the button 518 to capture the physical object as text data, but the processing sequence when the button 516 is pressed can also be described using the same FIG. 8.

[0101] FIG. 7(c), like FIG. 4(d), shows an example of a display when a virtual object is generated by photographing with the client terminal 122 in the see-through mode ON state and projected onto the display 206 of the client terminal 121. The virtual object 732 is the content of the text data of the physical object 718 projected onto the display 206 of the client terminal 121, and at this point in time, is still in the state of a virtual object.

[0102] FIG. 7(d), like FIG. 4(e), illustrates an operation of sharing the contents of the text data of a virtual object 732 with another user in the same virtual space on a client terminal 122 in which the see-through mode is ON. Reference numeral 742 denotes a virtual object showing the fingers of a user avatar operating client terminal 121, or an image of actual fingers. The user operating client terminal 121 performs a sharing operation by grabbing virtual object 732 with the fingers and dropping it onto the avatar of the user with whom the user wishes to share (avatar 714 in this case). Virtual object 744 is a virtual object that is displayed when sharing settings are made by the above operation. Although the above method has been given as an operation for sharing a virtual object with other users in the first embodiment and on Saturday, the operation method is not limited to this.

[0103] Fig. 7(e) is an image of the virtual space projected on the display 206 of the client terminal 131, similar to Fig. 4(f). In the second embodiment, the client terminal 131 does not need to switch the see-through mode to ON because it does not capture images of physical objects.

[0104] Reference numeral 752 denotes the display 206 of the client terminal 131 . An avatar 754 shows the avatar of the user who operates the client terminal 121, which is the sharing source user. Reference numeral 756 denotes a virtual object projecting a UI for the operating user of the client terminal 131 to work in the virtual space, and in this embodiment, it is described as a virtual object simulating a PC used in the real space. Furthermore, a chat screen is projected onto an area corresponding to a display within the virtual object 756. This chat corresponds to a chat function provided by the collaboration application management unit 366 included in the virtual space management server 111. When the operating user of the client terminal 121 performs a sharing operation of the virtual object 732 on the avatar of the operating user of the client terminal 131, the shared text data is projected onto the chat projected onto the display area of ​​the virtual object 756.

[0105] Next, the overall processing sequence will be described with reference to Fig. 8. The processing of the client terminal 121 shown in Fig. 8 is realized by the CPU 202 of the client terminal 121 reading and executing a program stored in the ROM 204 or HDD / SSD 205. The processing of the client terminal 122 is realized by the CPU 202 of the client terminal 122 reading and executing a program stored in the ROM 204 or HDD / SSD 205. The processing of the virtual space management server 111 is realized by the CPU 202 of the virtual space management server 111 reading and executing a program stored in the ROM 204 or HDD / SSD 205. The processing of the client terminal 131 is realized by the CPU 202 of the client terminal 131 reading and executing a program stored in the ROM 204 or HDD / SSD 205.

[0106] First, S812 to S824 are the same processes as S612 to S624, and therefore the description will be omitted. Next, in S826, as shown in FIG. 5(a), the user presses any one of the buttons 514, 516, and 518, and the photographing control unit 336 of the client terminal 122 determines the photographing mode. The user can select "3D", "text", "image", etc. as the photographing mode, but it is possible to increase the variation of the photographing mode according to the purpose of use. In addition, the determination of the photographing mode is not limited to the UI in which the user presses a button as shown in FIG. 5(a). For example, the photographing mode may be determined by a gesture by the user, or the photographing control unit 336 of the client terminal 122 can automatically determine that the photographing mode is text or image by performing OCR or edge extraction of the photographing object. As described above, in this case, the photographing mode will be described as being selected as "text" in order to take the physical object into the virtual space as text data by pressing the button 518.

[0107] Next, in S828, the shooting data generation unit 338 of the client terminal 122 photographs the physical object and generates shooting data, similar to S626 in Figure 6, and assigns an attribute of the shooting data type corresponding to the shooting mode selected in S826 above to the generated shooting data.

[0108] Table 5 shows an example of a shared photographing data management table managed by the shared photographing data management unit 362 of the second embodiment. The shared photographing data management table shown in Table 5 is an extension of the shared photographing data management table of Table 4 described in the first embodiment. In the photographing data type column of Table 5, attributes corresponding to the selection in the above-mentioned photographing mode are entered. As shown in the photographing data type column of Table 5, in the subsequent processing, the photographing data described in the first embodiment to which this photographing data type has been added is treated as new "photographing data". [Table 5]

[0109] Moreover, steps S832 to S842 are the same as steps S630 to S640, and therefore the description thereof will be omitted. When the shared captured image data receiving unit 360 of the virtual space management server 111 receives the shared captured image data from the client terminal 121, in S844, the shared captured image data receiving unit 360 refers to the captured image data type of the received shared captured image data. When the shared captured image data receiving unit 360 determines that the captured image data type is "text" or "image", it instructs the collaboration app managing unit 366 to post the shared captured image data (described as "text data" here) to the account of the sharing destination user. Then, the collaboration app managing unit 366 of the virtual space management server 111 manages the text data (chat message projected on the UI of the virtual object 756) so that the operating user of the client terminal 131, who is the sharing destination user, can obtain the text data.

[0110] Also, in S846, the shared photographing data receiving unit 360 of the virtual space management server 111 returns a response indicating that the shared photographing data has been received to the photographing data sharing unit 328 of the client terminal 121. Note that, in S842, when the photographing data sharing unit 328 transmits the shared photographing data of the photographing data type "3D", the subsequent processing is the same as the processing subsequent to S640 in Fig. 6 described in the first embodiment.

[0111] Next, a description will be given of the processing on the side of the client terminal 131. Steps S852 to S858 are the same as steps S644 to S650, and therefore will be omitted. In S860, collaboration application control unit 384 of client terminal 131 makes a request to collaboration application management unit 366 of virtual space management server 111 to acquire content such as chat messages. When the collaboration application management unit 366 of the virtual space management server 111 receives the content acquisition request, the process proceeds to S862. In S 862 , if there is a chat message in the account of the user who logs in to the client terminal 131 , collaboration app management section 366 returns the chat message to collaboration app control section 384 of client terminal 131 .

[0112] When collaboration application control unit 384 of client terminal 131 receives the response from virtual space management server 111, the process proceeds to S864. In S864, if collaboration application control unit 384 of client terminal 131 was able to acquire the chat message in S862, collaboration application control unit 384 projects the acquired chat message on display 206 via display unit 372 of client terminal 131. When the processing of S864 is completed, the state shown in Fig. 7(e) is reached. Note that, although text data is posted to the chat in Fig. 7(e), image data is posted to the chat as an attachment.

[0113] As described above, by switching the sharing method with the destination user depending on the data type of the physical object corresponding to the virtual object, i.e., by sharing in a form depending on the data type of the physical object corresponding to the virtual object, it is possible to share the photographed physical object using a more appropriate projection method.

[0114] According to each of the above embodiments, it is possible to seamlessly incorporate physical objects from the real world into a virtual space with simple operations, without the need for complicated operations such as attaching and detaching a head-mounted display, and further to share this with other users.

[0115] It goes without saying that the configurations and contents of the various data described above are not limited to those described above, and the data may have various configurations and contents depending on the application and purpose. Although one embodiment has been described above, the present invention can be embodied, for example, as a system, an apparatus, a method, a program, a storage medium, etc. Specifically, the present invention may be applied to a system composed of multiple devices, or may be applied to an apparatus composed of a single device. Furthermore, any combination of the above-described embodiments is also included in the present invention.

[0116] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. Furthermore, the present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device. The present invention is not limited to the above-mentioned embodiment, and various modifications (including organic combinations of the embodiments) are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. In other words, the present invention includes all configurations that combine the above-mentioned embodiments and their modifications.

[0117] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A head mounted display device, A display control means for projecting a virtual space into the field of view of a user wearing the head mounted display device; and an execution means for executing control to enable a see-through mode so that a real space can be confirmed through the projected virtual space, the display control means displays a virtual object generated from shooting data relating to a physical object in a real space within a virtual space projected into a field of view of the user; A head-mounted display device, characterized in that the virtual object is shared so that it can be viewed by other members participating in the virtual space. (Configuration 2) 2. The head mounted display device according to configuration 1, wherein the virtual object is shared in a form according to a type of a physical object corresponding to the virtual object. (Configuration 3) The head mounted display device according to configuration 2, characterized in that in the case of a virtual object generated from three-dimensional data of the physical object, the virtual object is shared in a form in which it is displayed in a virtual space projected into the field of view of the sharing member. (Configuration 4) In the case of a virtual object generated from image data of the physical object or text data obtained by character recognition of the image data, the image data or text data is shared by posting it to an account related to the shared member of a service having a message posting function. (Configuration 5) The head-mounted display device according to configuration 4, characterized in that the image data or text data posted to the account of the member to be shared is viewed via a screen related to the service that is displayed in a virtual space projected into the field of vision of the member. (Configuration 6) 4. The head mounted display device according to configuration 3, wherein the display control means stereoscopically displays a virtual object generated from three-dimensional data of the physical object in the virtual space. (Configuration 7) 5. The head mounted display device according to configuration 4, wherein the display control means displays a virtual object generated from the image data or text data in the virtual space. (Configuration 8) the photographing data relating to the physical object includes data indicating a feature amount of a surrounding real space including the physical object; The head mounted display device according to any one of configurations 1 to 7, wherein the display control means displays a virtual object generated from the shooting data at a position in the virtual space according to a feature included in the shooting data. (Configuration 9) The head mounted display device according to any one of configurations 1 to 8, further comprising an instruction means for instructing that the virtual object generated from the shooting data be shared with one or more other members participating in the virtual space. (Configuration 10) The head mounted display device according to configuration 9, wherein the instruction means instructs other members participating in the virtual space to share the virtual object generated from the shooting data with the other members by overlaying the virtual object on the other members. (Configuration 11) A head mounted display device according to any one of configurations 1 to 10, characterized in having at least one of an imaging means for imaging the physical object and generating the imaging data, and a receiving means for receiving the imaging data from another device. (Method 1) A method for controlling a head mounted display device, comprising: a first display step of projecting a virtual space into the field of view of a user wearing the head mounted display device; an execution step of enabling a see-through mode and executing control to enable confirmation of a real space through the projected virtual space; a second display step of displaying a virtual object generated from shooting data related to a physical object in a real space within the virtual space projected into the user's field of view; A method for controlling a head-mounted display device, wherein the virtual object is shared so that it can be viewed by other members participating in the virtual space. (Program 1) A program for causing a computer to execute the control method described in Method 1.

Claims

1. A head mounted display device, A display control means for projecting a virtual space into the field of view of a user wearing the head mounted display device; and an execution means for executing control to enable a see-through mode so that a real space can be confirmed through the projected virtual space, the display control means displays a virtual object generated from shooting data relating to a physical object in a real space within a virtual space projected into a field of view of the user; A head-mounted display device, characterized in that the virtual object is shared so that it can be viewed by other members participating in the virtual space.

2. The head mounted display device according to claim 1 , wherein the virtual object is shared in a form according to a type of a physical object corresponding to the virtual object.

3. 3. The head mounted display device according to claim 2, wherein in the case of a virtual object generated from three-dimensional data of the physical object, the virtual object is shared in a form in which it is displayed in a virtual space projected into the field of view of the sharing member.

4. The head-mounted display device according to claim 2, characterized in that in the case of a virtual object generated from image data of the physical object or text data obtained by character recognition of the image data, the image data or text data is shared by posting it to an account related to the shared member of a service having a message posting function.

5. The head-mounted display device according to claim 4, characterized in that the image data or text data posted to an account related to the shared member is viewed via a screen related to the service that is displayed in a virtual space projected into the member's field of vision.

6. 4. The head mounted display device according to claim 3, wherein the display control means stereoscopically displays a virtual object generated from the three-dimensional data of the physical object in the virtual space.

7. 5. The head mounted display device according to claim 4, wherein the display control means displays a virtual object generated from the image data or the text data in the virtual space.

8. the photographing data relating to the physical object includes data indicating a feature amount of a surrounding real space including the physical object; The head mounted display device according to any one of claims 1 to 7, characterized in that the display control means displays a virtual object generated from the shooting data at a position in the virtual space according to a feature included in the shooting data.

9. The head mounted display device according to any one of claims 1 to 7, further comprising an instruction means for instructing the virtual object generated from the shooting data to be shared with one or more other members participating in the virtual space.

10. The head mounted display device according to claim 9, characterized in that the instruction means instructs the sharing of the virtual object generated from the shooting data with the other members participating in the virtual space by an operation of overlaying the virtual object on a virtual object that resembles the other members.

11. The head mounted display device according to any one of claims 1 to 7, characterized in that it has at least one of an imaging means for imaging the physical object and generating the imaging data, and a receiving means for receiving the imaging data from another device.

12. A method for controlling a head mounted display device, comprising: a first display step of projecting a virtual space into the field of view of a user wearing the head mounted display device; an execution step of enabling a see-through mode and executing control to enable confirmation of a real space through the projected virtual space; a second display step of displaying a virtual object generated from shooting data related to a physical object in a real space within the virtual space projected into the field of view of the user; A method for controlling a head-mounted display device, wherein the virtual object is shared so that it can be viewed by other members participating in the virtual space.

13. A program for causing a computer to execute the control method according to claim 12.

Images