Space sharing system, space sharing device, and program

Abstract

We provide a space sharing system, space sharing device, and program that enable users to experience the reliability of the operation. [Solution] The spatial sharing server 1 includes a virtual view generation unit 11 that generates a virtual view in a virtual space VSP that reproduces the real space SP, by placing virtual objects corresponding to movable real objects in the real space SP; a virtual view output unit 12 that outputs the generated virtual view to a PC used by a user who is not in the real space SP; an object motion acquisition unit 13 that detects the user's tactile actions toward real objects in the real space SP based on images acquired via a camera on an HMD worn by the user, and acquires real object motion data including the position and orientation of the real object; a synchronization processing unit 14 that synchronizes the position and orientation of virtual objects corresponding to real objects based on the acquired real object motion data; and a virtual view update unit 15 that updates the virtual view output to the virtual view output unit 12 to a synchronized state.

Description

Technical Field

[0003] ,

[0005] , , , , , , , , , , , , , ,

[0004] , , , , , , ,

[0006] ,

[0001] The present invention relates to a space sharing system, a space sharing device, and a program.

Background Art

[0002] In recent years, by using a digital twin that reproduces the real space in a virtual space and arranges virtual objects, users can perform real-time simulations that are difficult to execute in the real space and communications between users in remote locations within the virtual space. For example, a technique aimed at expanding the range of applications of the virtual space has been disclosed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Currently, operations in the virtual space are mainly performed using, for example, a controller. Therefore, the operability does not reach the certainty of operating a real object by holding it in hand.

[0005] Therefore, an object of the present invention is to provide a space sharing system, a space sharing device, and a program that enable a user to actually feel the certainty of operability.

Means for Solving the Problems

[0006] The present invention solves the above problems by the following means. The first invention is a space sharing system comprising: a first generation means for generating a first view in which virtual objects corresponding to movable real objects in the real space are placed in a virtual space that reproduces the real space; a first display means for outputting the first view generated by the first generation means to a display device used by a first user who is not in the real space; an object motion acquisition means for detecting the tactile actions of a second user in the real space towards the real objects based on images acquired via an imaging unit in a wearable device worn by the second user, and acquiring real object motion data including the position and orientation of the real objects; a synchronization means for synchronizing the position and orientation of the virtual objects corresponding to the real objects based on the real object motion data acquired by the object motion acquisition means; and a first update means for updating the first view output to the first display means to a synchronized state by the synchronization means. The second invention is a spatial sharing system of the first invention, wherein the first generation means generates the first view in which a first avatar corresponding to the first user is further placed in the virtual space. The third invention is a spatial sharing system of the first or second invention, wherein the first generation means generates the first view in which a second avatar corresponding to the second user in the real space is further placed at a predetermined position in the virtual space corresponding to the position of the second user in the real space. The fourth invention is a spatial sharing system comprising: a simulation means for performing a predetermined simulation in the virtual space, and a change visualization means for generating a visualization image of the virtual space that visualizes the simulation results by the simulation means of the virtual space, corresponding to the position and movement of each object in the virtual space, including the virtual object corresponding to the real object, wherein the first update means reflects the visualization image generated by the change visualization means in the first view output to the first display means. The fifth invention is a spatial sharing system of the fourth invention, comprising: a second generation means for generating a second view in which the visualization image generated by the change visualization means is superimposed on a predetermined position in the real space corresponding to the virtual space; and a second display means for outputting the second view generated by the second generation means to a display unit on the second user's mounting device. The sixth invention is a spatial sharing system of the fifth invention, further comprising a first avatar acquisition means for acquiring first avatar data including the position of a first avatar corresponding to the first user, and a second generation means for generating a second view in which the first avatar is further superimposed on a predetermined position in the real space corresponding to the position of the first avatar in the virtual space, based on the first avatar data acquired by the first avatar acquisition means. The seventh invention is a spatial sharing system of the sixth invention, wherein the first avatar acquisition means acquires first avatar data including the position of the first user, which is detected based on an image acquired via a camera unit on a device worn by the first user. The eighth invention is a spatial sharing system in any of the first to seventh inventions, wherein the virtual object is a 3DCG model of the real object. The ninth invention is a spatial sharing system of the fifth or sixth invention, wherein the virtual object is a 3DCG model of an object having an approximate shape different from the actual object, and the second generation means generates the second view in which the virtual object is further superimposed on the actual object. The tenth invention is a spatial sharing system in any of the first to ninth inventions, wherein the device, which is a physical object located in the physical space, is equipped with a control unit and an operating unit, and is connected to the network so as to be communicative, and comprises an operation receiving means for receiving operations on the virtual object located in the virtual space, and a command transmitting means for transmitting a command to the control unit of the device to operate the operating unit of the device corresponding to the operation of the virtual object received by the operation receiving means. The eleventh invention is a space-sharing device that is communicably connected to a display device and a wearable device, comprising: a first generation means for generating a first view in which virtual objects corresponding to movable real objects in the real space are arranged in a virtual space that reproduces the real space; a first display means for outputting the first view generated by the first generation means to the display device of a first user who is not in the real space; an object motion acquisition means for detecting the tactile actions of a second user in the real space towards the real objects based on images acquired from a wearable device worn by the second user, and acquiring real object motion data including the position and orientation of the real objects; a synchronization means for synchronizing the position and orientation of the virtual objects corresponding to the real objects based on the real object motion data acquired by the object motion acquisition means; and a first update means for updating the first view output to the first display means to a synchronized state by the synchronization means. The twelfth invention is a program for causing a computer to function as: a first generation means for generating a first view in which virtual objects corresponding to movable real objects in the real space are placed in a virtual space that reproduces the real space; a first display means for outputting the first view generated by the first generation means to a display device of a first user who is not in the real space; an object motion acquisition means for detecting the tactile actions of a second user in the real space towards the real objects based on images acquired from a wearable device worn by the second user, and acquiring real object motion data including the position and orientation of the real objects; a synchronization means for synchronizing the position and orientation of the virtual objects corresponding to the real objects based on the real object motion data acquired by the object motion acquisition means; and a first update means for updating the first view output to the first display means to a synchronized state by the synchronization means. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide a space sharing system, a space sharing device, and a program that enable users to experience the certainty of operability. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing the overall outline of the space sharing system according to this embodiment. [Figure 2] This is a diagram showing the functional blocks of the space sharing server according to this embodiment. [Figure 3] This is a diagram showing the functional blocks of the HMD and the home appliance according to this embodiment. [Figure 4] This is a flowchart showing the virtual space processing of the space sharing server according to this embodiment. [Figure 5] This is a flowchart showing the preprocessing of the space sharing server according to this embodiment. [Figure 6] This is a diagram for explaining the preprocessing of the space sharing server according to this embodiment. [Figure 7] This is a diagram for explaining the preprocessing of the space sharing server according to this embodiment. [Figure 8] This is a flowchart showing the received image processing of the space sharing server according to this embodiment. [Figure 9] This is a diagram showing the state of the real space and an example of the display on the PC according to this embodiment. [Figure 10] This is a diagram showing an example of the display on the PC according to this embodiment. [Figure 11] This is a diagram showing the state of the real space and an example of the display on the PC according to this embodiment. [Figure 12] This is a flowchart showing the image output processing to the HMD of the space sharing server according to this embodiment. [Figure 13] This is a diagram showing an example of the display of the HMD and the PC according to this embodiment. [Figure 14] This is a diagram showing an example of the display of the HMD according to this embodiment. [Figure 15] This is a diagram showing an example of the display of the HMD and the PC according to this embodiment.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments for implementing the present invention will be described with reference to the drawings. Note that this is merely an example, and the technical scope of the present invention is not limited thereto. (Embodiment) <Space sharing system 100> FIG. 1 is an overall schematic diagram of a space sharing system 100 according to the present embodiment. The space sharing system 100 shown in FIG. 1 is a system including a space sharing server 1, HMDs (head-mounted displays) 4A (wearable devices) and 4B, a PC (personal computer) (display device) 5, and a home appliance 9 (a device that is a physical object).

[0010] The space sharing server 1, the HMDs 4A and 4B, the PC 5, and the home appliance 9 are communicably connected via a communication network N. The communication network N is, for example, an Internet line, a mobile terminal communication network, or the like. Note that the communication network N is not limited to the above as a communication line connecting each device. For example, it may partially include a LAN (Local Area Network) or the like, and may be wired or wireless.

[0011] In the example shown in FIG. 1, a user P1 in the real space SP is wearing the HMD 4A. The real space SP is, for example, a space equipped with a pot 7 (a physical object), a desk 8 (a physical object), and a home appliance 9. Also, a user P2 in another space different from the real space SP where the user P1 is located is wearing the HMD 4B. Furthermore, a user P3 in another space different from the real space SP where the user P1 is located is operating the PC 5. The user P2 and the user P3 may be in the same space or in different spaces. Hereinafter, when the HMDs 4A and 4B are not distinguished, they will simply be described as the HMD 4.

[0012] The spatial sharing server 1 outputs a virtual view to the PC 5 operated by user P3, which includes the virtual space VSP, a digital twin that reproduces the real space SP, and the avatars of users P1, P2, and P3. The spatial sharing server 1 also outputs the avatars of users P2 and P3 (excluding user P1) superimposed on the real space SP to the HMD 4A worn by user P1. Furthermore, the spatial sharing server 1 outputs a virtual view to the HMD 4B worn by user P2, which includes the virtual space VSP and the avatars of other users (excluding user P2's avatar). Thus, the spatial sharing system 100 generates a virtual space VSP that reproduces the real space SP, and then displays the avatars of each user of HMD4A, HMD4B, and PC5, so that each user can share one virtual space VSP.

[0013] Then, user P1, while wearing the HMD4A, moves, for example, the vase 7. In this way, the image 7v of the virtual space VSP corresponding to the vase 7 in the spatial sharing system 100 moves in conjunction with user P1's actual operation. In this way, the spatial sharing system 100 makes the operation performed by user P1 on a real object, the vase 7, so that the user P1 can feel the tactile sensation and weight of the real object, and can experience the certainty of operation.

[0014] <Space sharing server 1> Next, we will explain the spatial sharing server 1. Figure 2 shows the functional blocks of the spatial sharing server 1 according to this embodiment. As shown in Figure 2, the spatial sharing server 1 comprises a control unit 10, a storage unit 30, and a communication interface unit 39. The control unit 10 is a CPU (Central Processing Unit) that controls the entire space-sharing server 1. The control unit 10 works in cooperation with the aforementioned hardware to perform various functions by appropriately reading and executing the OS (Operating System) and application programs stored in the memory unit 30.

[0015] The control unit 10 includes a virtual view generation unit 11 (first generation means), a virtual view output unit 12 (first display means), an object motion acquisition unit 13 (object motion acquisition means), a synchronization processing unit 14 (synchronization means), a virtual view update unit 15 (first update means), a virtual operation reception unit 16 (operation reception means), an operation command transmission unit 17 (command transmission means), an environment simulation unit 18 (simulation means), a change visualization unit 19 (change visualization means), a virtual avatar acquisition unit 21 (first avatar acquisition means), a superimposed view generation unit 22 (second generation means), a superimposed view output unit 23 (second display means), and a superimposed view update unit 24.

[0016] The virtual view generation unit 11 generates a virtual view (first view) in which virtual objects (for example, images 7v to 9v) corresponding to movable real objects (for example, vase 7, desk 8, and home appliance 9) in the real space SP are placed in the virtual space VSP which reproduces the real space SP. The virtual view generation unit 11 further places avatars P2v and avatar P3v (first avatar) in the virtual space VSP, corresponding to users P2 and P3 (first users) who are not in the real space SP. The virtual view generation unit 11 also places avatar P1v (second avatar) in the virtual space VSP, corresponding to user P1 (second user) who is in the real space SP.

[0017] The virtual view output unit 12 outputs the virtual view generated by the virtual view generation unit 11 to the PC 5 of user P3 (first user), who is not in the physical space SP. Furthermore, the virtual view output unit 12 outputs the virtual view generated by the virtual view generation unit 11, which is a virtual view from the perspective of avatar P2v, to the HMD4B of user P2 (first user), who is not in the real space SP.

[0018] The object motion acquisition unit 13 detects the tactile movements of user P1 in real space SP towards real objects (e.g., vase 7, desk 8, and home appliance 9) based on images acquired via a camera 46 on the HMD 4A worn by user P1 (see Figure 3(A) described later), and acquires real object motion data including the position and orientation of the real objects. The synchronization processing unit 14 synchronizes the position and orientation of the virtual object corresponding to the real object based on the real object motion data acquired by the object motion acquisition unit 13.

[0019] The virtual view update unit 15 updates the virtual view output to the virtual view output unit 12 to a synchronized state by the synchronization processing unit 14. The virtual view update unit 15 also reflects the visualization image generated by the change visualization unit 19 (described later) in the virtual view output to the virtual view output unit 12.

[0020] The virtual operation reception unit 16 receives operations on virtual objects (for example, image 9v) placed in the virtual space VSP. The operation command transmission unit 17 transmits a command to the control unit 90 (see Figure 3(B) below) of the device (for example, a home appliance 9) to operate the device's operation unit 97 (see Figure 3(B) below), which corresponds to the operation of the virtual object received by the virtual operation reception unit 16.

[0021] The environment simulation unit 18 performs predetermined simulations in the virtual space VSP. These predetermined simulations include, for example, simulations of the flow of air, smoke, water, etc., in the virtual space VSP, as well as simulations of temperature distribution, light distribution of sunlight and lighting, acoustic characteristics, etc. The environment simulation unit 18 then performs simulations that are influenced by each virtual object and avatar object in the virtual space VSP. More specifically, when the environment simulation unit 18 performs, for example, a simulation of airflow in the virtual space VSP, the simulation will be influenced by the positions of the virtual objects and avatars. The change visualization unit 19 generates a visualization image that visualizes the results of the simulation performed by the environment simulation unit 18 on the virtual space VSP, which includes virtual objects corresponding to real objects, and the position and movement of each object in the virtual space VSP.

[0022] The virtual avatar acquisition unit 21 acquires virtual avatar data (first avatar data) including the positions of avatars P2v and avatar P3v (first avatars) corresponding to users P2 and P3 (first users) who are not in the physical space SP. More specifically, the virtual avatar acquisition unit 21 acquires virtual avatar data including the position of user P2, detected based on an image acquired via a camera 46 on the HMD4 worn by user P2.

[0023] The superimposed view generation unit 22 generates a superimposed view by further superimposing avatars P2v and P3v (first avatar) at predetermined positions in real space SP based on the virtual avatar data acquired by the virtual avatar acquisition unit 21. Furthermore, if the real object is a substitute for the original object, and the virtual object is an approximate shape of the real object and is generated using a 3DCG model of the original object, the superimposed view generation unit 22 generates a superimposed view in which the virtual object is further superimposed on the real object. Furthermore, the superimposed view generation unit 22 generates a superimposed view by superimposing the visualization image generated by the change visualization unit 19 onto a predetermined position in the real space SP.

[0024] The superimposed view output unit 23 outputs the superimposed view generated by the superimposed view generation unit 22 to the display 48 on the HMD4A of user P1 (second user) (see Figure 3(A) described later). The superimposed view update unit 24 reflects the visualization image generated by the change visualization unit 19 onto the superimposed view.

[0025] Next, the memory unit 30 will be explained. The storage unit 30 is a storage area such as a hard disk or semiconductor memory element for storing programs, data, etc., necessary for the control unit 10 to perform various processes. The storage unit 30 includes a program storage unit 31. The program storage unit 31 is a memory area for storing various programs. The program storage unit 31 stores the spatially shared program 31a. The spatially shared program 31a is a program for executing each of the functions of the control unit 10 described above.

[0026] The communication interface unit 39 is an interface for communicating with, for example, the HMD4, PC5, and home appliances 9. Here, "computer" refers to an information processing device equipped with a control unit, memory device, etc., and the space-sharing server 1 is an information processing device equipped with a control unit 10, a memory unit 30, etc., and is included in the concept of a computer. Furthermore, there is no limit to the number of hardware components that make up the spatial sharing server 1; it may be configured with one or more components as needed. Also, the spatial sharing server 1 may, for example, be a cloud service.

[0027] <hmd4> Figure 3 shows the functional blocks of the HMD4 and home appliance 9 according to this embodiment. The HMD4 shown in Figure 1 is a wearable device that, when worn on the heads of users P1 and P2, has a display 48 positioned to face the eyes of users P1 and P2.

[0028] User P1, located in the real space SP, can view objects such as the vase 7 located in the real space SP through the display 48. The HMD4A can provide an augmented reality (AR) environment in which user P1 can directly touch the vase 7. Thus, the HMD4A utilizes, for example, an optical see-through display. With an optical see-through display, user P1 can view the surrounding scenery through the display 48 and see the image from the electronic display superimposed using optical systems such as prisms and half-mirrors. Furthermore, the HMD4A is not limited to AR; for example, it could also be MR (Mixed Reality). In either case, user P1 can see both the real space SP and virtual objects.

[0029] Furthermore, user P2, who is in another space, can view a virtual view of the virtual space VSP output to display 48. HMD4B outputs a virtual view of the virtual space VSP from user P's perspective to display 48 for user P2. HMD4B can provide a virtual view, for example, a VR (Virtual Reality) environment. Furthermore, the HMD4B is not limited to VR; for example, it could also be MR. In either case, user P2 can view the virtual view of the virtual space VSP.

[0030] As shown in Figure 3(A), the HMD4 comprises a control unit 40, a storage unit 45, a camera 46 (image capture unit), a display 48 (display unit), and a communication interface unit 49. The control unit 40 is a CPU that controls the entire HMD4. The control unit 40 works in cooperation with the aforementioned hardware to perform various functions by appropriately reading and executing the OS and application programs stored in the memory unit 45.

[0031] The control unit 40 includes an image acquisition processing unit 41 and an image reception processing unit 42. The image acquisition processing unit 41 acquires the captured image from the camera 46 and transmits it to the spatial sharing server 1. The image receiving processing unit 42 receives superimposed views and virtual views from the spatial sharing server 1 and outputs the received superimposed views and virtual views to the display 48. The memory unit 45 is a memory area such as a semiconductor memory element for storing programs, data, etc., necessary for the control unit 40 to perform various processes. The memory unit 45 includes a program memory unit 45a. The program storage unit 45a is a storage area that stores various programs, including programs for performing various functions executed by the control unit 40 of the HMD4.

[0032] Camera 46 is a camera that captures images of the area around the HMD4. Camera 46 is located on the outer surface opposite to the surface facing the heads of users P1 and P2 wearing the HMD4, and is provided, for example, near the positions corresponding to the left and right eyes of users P1 and P2 wearing the HMD4. Furthermore, there are not limited to two cameras 46, and there may be more than two cameras to capture images of the area around users P1 and P2. The display 48 is a display device such as an LCD (Liquid Crystal Display) or an organic EL display. The display 48 is provided on the inner surface that faces the heads of users P1 and P2 wearing the HMD4. The communication interface unit 49 is, for example, an interface for communicating with the spatial sharing server 1, etc. Here, "computer" refers to an information processing device equipped with a control unit, memory device, etc., and HMD4 is an information processing device equipped with a control unit 40, a memory unit 45, etc., and is included in the concept of a computer.

[0033] <pc5> The PC5 shown in Figure 1 is a notebook PC operated by user P3. The PC5 is not limited to the form shown in Figure 1; it could be a laptop PC, a smartphone, a tablet, or other mobile device. PC5 is a general-purpose computer, and although not shown in the diagram, it includes a control unit, a memory unit, an input / output unit, a communication interface unit, and the like.

[0034] <Home appliances 9> Next, I will explain about home appliances. The home appliance 9 shown in Figure 1 is a fan located in the real-world space SP. The home appliance 9 can be operated directly by user P1 in the real-world space SP, and can also be operated by avatar Pv3 in the virtual space VSP by manipulating the image 9v corresponding to the home appliance 9. As shown in Figure 3(B), the home appliance 9 comprises a control unit 90, a storage unit 95, an operation unit 96, an operating unit 97, and a communication interface unit 99. The control unit 90 is a CPU that controls the entire home appliance 9. The control unit 90 works in cooperation with the aforementioned hardware to perform various functions by appropriately reading and executing the OS and application programs stored in the memory unit 95.

[0035] The memory unit 95 is a storage area such as a semiconductor memory element for storing programs, data, etc., necessary for the control unit 90 to perform various processes. The memory unit 95 includes a program memory unit 95a. The program storage unit 95a is a storage area that stores various programs, including programs for performing various functions executed by the control unit 90 of the home appliance 9. The control unit 96 consists of buttons, etc., for operating the operating unit 97 of the home appliance 9. The operating unit 97 is, for example, a motor that powers the home appliance 9. The communication interface unit 99 is, for example, an interface for communicating with the spatial sharing server 1, etc.

[0036] Next, we will explain the processing in the spatial sharing system 100 with specific examples. Figure 4 is a flowchart showing the virtual space processing of the space sharing server 1 according to this embodiment. Figure 5 is a flowchart showing the preprocessing of the spatial sharing server 1 according to this embodiment. Figures 6 and 7 are diagrams illustrating the preprocessing of the spatial sharing server 1 according to this embodiment. Figure 8 is a flowchart showing the received image processing of the spatial sharing server 1 according to this embodiment. Figure 9 shows an example of the state of the real-space SP and the display on the PC5 according to this embodiment. Figure 10 shows an example of the display of PC5 according to this embodiment. Figure 11 shows an example of the state of the real-space SP and the display on the PC5 according to this embodiment.

[0037] In order to perform processing using the spatial sharing system 100, in step S (hereinafter referred to simply as "S") 11 of Figure 4, the control unit 10 of the spatial sharing server 1 performs pre-processing. Here, we will explain the pre-processing based on Figure 5. In step S31 of Figure 5, the control unit 10 (virtual view generation unit 11) generates a virtual space VSP. The control unit 10 generates a virtual space VSP, for example, a 3DCG virtual space VSP that reproduces the real space SP, using, for example, an image taken of the real space SP. The control unit 10 can generate a virtual space VSP from an image of the real space SP using known techniques. In S32, the control unit 10 (virtual view generation unit 11) places virtual objects (images 7v and 8v) corresponding to real objects (for example, the vase 7 and the desk 8) that are operated in the real space SP into the virtual space VSP.

[0038] Here, we will explain the process with a specific example. Figure 6 shows an example where the real space SP has a vase 7 and a desk 8. The control unit 10 uses a 3D scanner or the like to generate images 7v and 8v, which are virtual objects, from the vase 7 and desk 8 in the real space SP, and places them at the location in the virtual space VSP corresponding to the real space SP. Images 7v and 8v are 3DCG models of the vase 7 and desk 8.

[0039] Furthermore, real objects and virtual objects may be different things. Figure 7 shows an example where the real space SP contains a cylinder 6 (real object) and a desk 8. Here, the cylinder 6 is a substitute for the vase 7 in Figure 6. It is desirable that the cylinder 6, within the range that user P1 can touch with their hand, approximates the contour shape, surface texture, and weight of the vase 7. The control unit 10 places images 7v and 8v in the virtual space VSP at positions corresponding to the cylinder 6 and desk 8 in the real space SP. Furthermore, the superimposed view that user P1 sees when wearing the HMD4A is a superimposed view of image 7v placed on the cylinder 6 in the real space SP.

[0040] In step S33 of Figure 5, the control unit 10 associates the virtual object operated in the virtual space VSP with the actual object (device). Specifically, the control unit 10 associates the image 9v corresponding to the home appliance 9 with the home appliance 9 in the real space SP. In S34, the control unit 10 (environment simulation unit 18) performs environmental simulation processing in the virtual space VSP. As part of the environmental simulation processing, the control unit 10 simulates, for example, airflow. After that, the control unit 10 moves the processing to S12 in Figure 4.

[0041] In S12 of Figure 4, the control unit 10 accepts access to the virtual space VSP. For example, the control unit 10 accepts access from user P3 via PC 5. In this way, the control unit 10 (virtual view generation unit 11) generates an avatar P3v corresponding to user P3 and generates a virtual view in the virtual space VSP with avatar P3v placed therein. In S13, the control unit 10 (virtual view output unit 12) outputs the generated virtual view to the PC 5. In this way, user P3 can use the PC 5 to view the virtual space VSP, which includes his avatar P3v.

[0042] In S14, the control unit 10 determines whether or not it has received an image from the HMD4. For example, if user P1 is wearing the HMD4A in the real-world SP, the control unit 40 (image acquisition processing unit 41) of the HMD4 transmits the image acquired via the camera 46 to the space sharing server 1, so the control unit 10 can receive an image from the real-world SP. Also, for example, if user P2 is wearing the HMD4B in a different space from the real-world SP, the control unit 40 (image acquisition processing unit 41) of the HMD4 transmits the image acquired via the camera 46 to the space sharing server 1, so the control unit 10 can receive an image from the other space. If the control unit 10 has received an image from the HMD4 (S14: YES), it moves the process to S15. On the other hand, if the control unit 10 has not received an image from the HMD4 (S14: NO), it moves the process to S16. In S15, the control unit 10 performs received image processing.

[0043] Here, we will explain the received image processing based on Figure 8. In step S41 of Figure 8, the control unit 10 recognizes the position of user P1 in the real space SP based on the image received from HMD4A. The control unit 10 also recognizes the position of user P2 in the other space based on the image received from HMD4B. In S42, the control unit 10 (virtual view generation unit 11) generates a virtual view in which avatar P1v corresponding to user P1 and avatar P2v corresponding to user P2 are placed in the virtual space VSP based on the recognized position. Then, the control unit 10 (virtual view output unit 12) outputs the generated virtual view to the PC 5.

[0044] Figure 9(A) shows an example of screen 50 displayed on PC5. In this screen 50, avatar P1v corresponding to user P1, which is recognized based on the image received from HMD4A, and avatar P3v corresponding to user P3 are displayed, and avatar P1v is shown touching image 7v. Furthermore, Figure 9(B) shows the state of the real-world SP corresponding to Figure 9(A). In the real-world SP, user P1 is touching the vase 7. The state that user P1 is seeing at this time will be described later.

[0045] Furthermore, although screen 50 in Figure 9(A) does not visualize the environmental simulation, it may also visualize the environmental simulation. Figure 10 shows screen 51, which is displayed on PC5, another example. This screen 51 is a virtual view of the environment simulation in progress, and the visualization image Ev is displayed. Here, the environmental simulation is performed in a virtual space (VSP) that replicates the real-world SP. Therefore, it is possible to verify how things change in the real-world SP within the virtual space (VSP). Furthermore, PC5 may allow the display of the visualization image Ev during the environmental simulation to be turned ON / OFF, and it may also allow switching between screen 50 in Figure 9(A) and screen 51 in Figure 10.

[0046] In S43 of Figure 8, the control unit 10 determines whether or not the image contains a real object. Here, a real object refers to something that corresponds to a virtual object synchronized with the virtual space VSP, specifically such as the vase 7, the desk 8, the cylinder 6, etc. If a real object is present (S43: YES), the control unit 10 moves the process to S44. On the other hand, if a real object is not present (S43: NO), the control unit 10 moves the process to S47. In S44, the control unit 10 (object motion acquisition unit 13) detects the position and orientation of the real object from the image and acquires real object motion data. In S45, the control unit 10 (synchronization processing unit 14) synchronizes the position and orientation of the virtual object corresponding to the real object.

[0047] In S46, the control unit 10 (virtual view update unit 15) updates the virtual view to be output to the PC5 based on the synchronized virtual space VSP. In S47, the control unit 10 performs image output processing to the HMD4. The image output processing to the HMD4 will be described later. After that, the control unit 10 moves the processing to S16 in Figure 4.

[0048] In S16 of Figure 4, the control unit 10 determines whether or not it has received an operation in the virtual space VSP. When user P3 performs movement operations, etc., in the virtual space VSP using avatar P3v, the control unit 10 determines that it has received an operation in the virtual space VSP. If an operation in the virtual space VSP has been received (S16: YES), the control unit 10 moves the process to S17. On the other hand, if an operation in the virtual space VSP has not been received (S16: NO), the control unit 10 moves the process to S14.

[0049] In S17, the control unit 10 (virtual operation reception unit 16) determines whether the operation corresponds to a virtual object corresponding to a real object (device). Specifically, the control unit 10 determines, for example, whether the operation corresponds to an image 9v corresponding to a home appliance 9. Figure 11(A) shows an example of screen 52 displayed on PC5. In this screen 52, avatar P1v corresponding to user P1 and avatar P3v corresponding to user P3 are arranged, and it shows a scenario in which avatar P3v is operating the switch 9vs of image 9v. In this case, the control unit 10 is receiving an operation corresponding to the virtual object (image 9v) corresponding to the real object (home appliance 9). If the operation corresponds to a virtual object corresponding to a real object (S17:YES), the control unit 10 moves the process to S18. On the other hand, if the operation does not correspond to a virtual object corresponding to a real object (S17:NO), the control unit 10 moves the process to S19.

[0050] In S18, the control unit 10 (operation command transmission unit 17) transmits a command signal corresponding to the operation to the actual object (device). In this way, the operation is performed by the actual object in the real space SP. Specifically, when an operation on image 9v is received, the control unit 10 transmits a command signal to the home appliance 9 in the real space SP. The control unit 90 of the home appliance 9 receives the command signal and operates the home appliance 9, for example, by operating the operation unit 97 (see Figure 3(B)). Alternatively, it stops the operation of the home appliance 9, for example, by stopping the operation of the operation unit 97. After that, the control unit 10 moves the process to S14. Figure 11(B) shows the state of the real-world SP corresponding to Figure 11(A). In the real-world SP, the stopped home appliance 9 is shown to be operating without any operation by user P1.

[0051] On the other hand, in S19 of Figure 4, the control unit 10 updates the virtual view output to the PC 5 in response to the operation. The operation here refers to, for example, a movement operation on avatar P3v. The update of the virtual view in response to the operation refers to, for example, updating the virtual view to reflect the change in the position of avatar P3v due to the movement operation on avatar P3v. The update of the virtual view in response to the operation may also include a change in the visualization image Ev (see Figure 10) due to the change in the position of avatar P3v. After that, the control unit 10 moves the processing to S14.

[0052] Next, we will explain the image output processing for the HMD4. This section explains the processing for HMD4A worn by user P1 in the physical space SP, as well as the processing for HMD4B worn by user P2 in a different space from the physical space SP. Figure 12 is a flowchart showing the image output process to the HMD4 by the spatial sharing server 1 according to this embodiment. Figure 13 shows an example of the display between HMD4A and PC5 according to this embodiment. Figure 14 shows examples of displays for HMD4A and HMD4B according to this embodiment. Figure 15 shows an example of the display between the HMD4B and PC5 according to this embodiment.

[0053] In S51 of Figure 12, the control unit 10 of the spatial sharing server 1 determines, for example, whether the source of the image is user P1 of the real space SP, based on the image received from the HMD4. If it is user P1 of the real space SP (S51: YES), the control unit 10 moves the process to S52. On the other hand, if it is not user P1 of the real space SP (S51: NO), the control unit 10 moves the process to S53.

[0054] In S52, the control unit 10 (superimposed view generation unit 22, superimposed view output unit 23) generates a superimposed view by superimposing the avatar P3v etc. that the user P1 has in the virtual space VSP based on the position of the user P1 in the real space SP, and outputs the superimposed view to the HMD4A. As a result, the control unit 40 (image reception processing unit 42) of the HMD4A outputs the image received from the space sharing server 1 to the display 48. After that, the control unit 10 moves the processing to S54.

[0055] Figure 13(A) shows an example of screen 60 displayed on the HMD4A of user P1 in real space SP. Figure 13(B) shows an example of screen 50 on PC5 corresponding to Figure 13(A). The screen 60 in Figure 13(A) includes an image from the user P1's perspective. The avatar P3v is positioned directly in front of user P1, corresponding to the screen 50 in Figure 13(B). Furthermore, user P1's hands and the vase 7 are not displayed on the display 48, but are directly visible through the display 48.

[0056] On the other hand, in S53 of Figure 12, the control unit 10 (virtual view output unit 12) outputs a virtual view from the perspective of the avatar P2v in the virtual space VSP, based on the position of user P2 in a space other than the real space SP, to the HMD4B. In doing so, the control unit 40 (image reception processing unit 42) of the HMD4B outputs the image received from the space sharing server 1 to the display 48.

[0057] Figure 14(A) shows an example of screen 71 displayed on the HMD4B of user P2, who is in a space other than the real space SP. Figure 14(B) shows an example of screen 61 displayed on the HMD4A of user P1, corresponding to Figure 14(A). In screen 71 of Figure 14(A), avatars P1v and P3v are positioned corresponding to the location of user P2. Screen 61 in Figure 14(B) shows the avatar P2v, which corresponds to user P2.

[0058] In S54 of Figure 12, the control unit 10 (virtual view update unit 15, superimposed view update unit 24) reflects the environment simulation. After that, the control unit 10 moves the processing to Figure 8. Figure 15(A) shows screen 72, which reflects the visualization image Ev onto screen 71 in Figure 14(A). Figure 15(B) shows an example of screen 55 output to PC5. Comparing screen 55 shown in Figure 15(B) with screen 51 shown in Figure 10, the visualization image Ev in screen 55 reflects the changes in the environment simulation due to the presence of avatar P2v corresponding to user P2. Therefore, both the presence or absence of avatar P2v and the visualization image Ev differ between screen 55 and screen 51.

[0059] Thus, the spatial sharing system 100 of this embodiment has the following effects. (1) A virtual view is generated in a virtual space VSP that reproduces the real space SP, with virtual objects corresponding to movable real objects in the real space SP placed therein. The generated virtual view is output to a PC5 used by user P3, who is not in the real space SP. The user P1's tactile actions toward the real objects in the real space SP are detected based on images acquired via a camera 46 on the HMD4A worn by user P1, and real object motion data including the position and orientation of the real objects is acquired. Based on the acquired real object motion data, the position and orientation of the virtual objects corresponding to the real objects are synchronized, and the output virtual view is updated to the synchronized state. Therefore, user P1 can manipulate real objects in the real space SP using user P1's sense of touch, and the position and orientation of the real object corresponding to the manipulation can be synchronized with the virtual object in the virtual space VSP that corresponds to the real object. As a result, user P1 can experience the certainty of the operability.

[0060] (2) The system now generates a virtual view by further placing the avatar P3v, which corresponds to user P3, in the virtual space VSP. Therefore, it is possible to place an avatar corresponding to user P3 in the virtual space VSP.

[0061] (3) A virtual view is generated by further placing the avatar P1v, which corresponds to the user P1 in the real space SP, at a predetermined position in the virtual space VSP corresponding to the position of user P1 in the real space SP. Therefore, it is possible to place the avatar P1v, which corresponds to the user P1 in the real-world space SP, at the same location as the real-world space SP.

[0062] (4) A predetermined simulation is performed in the virtual space VSP, and a visualization image is generated that visualizes the simulation results of the virtual space VSP corresponding to the position and movement of each object in the virtual space VSP, including virtual objects corresponding to real objects. The generated visualization image is then reflected in the output virtual view. Therefore, the virtual view will reflect a visualization image that visualizes the simulation results. Furthermore, the simulation can be made to correspond to the position and movement of each object in the virtual space (VSP).

[0063] (5) The generated visualization image is superimposed on a predetermined position in the real space SP corresponding to the virtual space VSP to create a superimposed view, and the generated superimposed view is output to the display 48 on the user P1's HMD4A. Therefore, user P1, who is in the real space SP, can see the visualization image superimposed on the real space SP.

[0064] (6) Avatar data including the position of avatar P3v corresponding to user P3 is acquired, and based on the acquired avatar data, a superimposed view is generated in which avatar P3v is further superimposed on a predetermined position in the real space SP corresponding to the position of avatar P3v in the virtual space VSP. Therefore, user P1, who is in real space SP, can see avatar P3v, which corresponds to user P3 superimposed on real space SP.

[0065] (7) Avatar data including the position of user P2, which is detected based on the image acquired via the camera 46 on the HMD4B worn by user P2, is acquired. Therefore, the position of user P2 can be obtained based on the image acquired by HMD4B. In addition, user P1, who is in real space SP, can see avatar P2v, which corresponds to user P2 superimposed on real space SP.

[0066] (8) The virtual objects were made to be 3DCG models of real objects. Therefore, the same real objects that user P1 operates in the real space SP can be placed as images in the virtual space VSP.

[0067] (9) A virtual object is a 3DCG model of an object with an approximate shape different from that of a real object, and a superimposed view is generated in which the virtual object is superimposed on the real object. Therefore, the real object that user P1 manipulates in the real space SP can be displayed to user P1 as the same image placed as an image in the virtual space VSP. Thus, even if the real object is a substitute for the original, it can be displayed to user P1 as if it were the original, and because it is an approximate shape, it can also be perceived through touch.

[0068] (10) A home appliance 9, which is a real object located in the real space SP, is equipped with a control unit 90 and an operating unit 97, and is connected to the communication network N so as to be communicative, and receives operations on a virtual object located in the virtual space VSP, and transmits a command to the control unit 90 of the home appliance 9 to operate the operating unit 97 of the home appliance 9 corresponding to the operation of the virtual object that was received. Therefore, by performing operations on virtual objects in the virtual space VSP, it is possible to operate the home appliance 9 corresponding to the virtual object.

[0069] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments. The embodiments described above and the modified forms described later can be used in combination as appropriate, but a detailed explanation is omitted.

[0070] (Transformed form) (1) In this embodiment, the spatial sharing server 1 is responsible for processing, and the HMD 4 only handles the transmission of images and the output of received images, but the embodiment is not limited to this. For example, the spatial sharing server and the HMD may share the processing of the control unit and storage unit. (2) In this embodiment, an HMD4 was used as an example of a mounting device, but it is not limited to this. For example, glasses or the like may be used.

[0071] (3) In this embodiment, the vase 7 and cylinder 6 are described as physical objects and are operated by a user P1 in the physical space SP, but the embodiment is not limited to this. For example, the desk 8 may be the physical object to be operated on, or the home appliance 9 may be the physical object to be operated on. Also, the home appliance 9 was described as a fan, but the embodiment is not limited to this, and may be other IoT home appliances, for example. (4) In this embodiment, an example of environmental simulation has been described in which the airflow in the virtual space VSP is simulated, but it is not limited to this. For example, the simulation may be about how the airflow and temperature distribution change when the placement of heating and cooling equipment is changed, or how the flow changes when the position of people and furniture in the space or the layout of the room is moved. Alternatively, the simulation may be about how the light distribution changes when the placement of lighting is changed, or how the sunlight changes depending on the position and size of the windows, or how it changes when the position of people and furniture in the space or the layout of the room is moved. [Explanation of Symbols]

[0072] 1 Space sharing server 4, 4A, 4B HMD 5 PC 6 tubes 7 pot 7V, 8V, 9V images 8 desk 9 Home appliances 10, 40, 90 Control Unit 11. Virtual View Generation Unit 12 Virtual View Output Section 13 Object motion acquisition section 14 Synchronization Processing Unit 15 Virtual View Update Section 16 Virtual Operation Reception Unit 17 Operation command transmission unit 18. Environment Simulation Department 19 Change Visualization Department 21 Virtual Avatar Acquisition Section 22 Overlay View Generation Unit 23. Superimposed View Output Section 24 Overlay View Update Section 30, 45, 95 storage section 31a Space Sharing Program 41 Image acquisition processing unit 42 Image receiving processing unit 46 Cameras 48 displays Screens 50, 51, 52, 55, 60, 61, 71, 72 96 Control section 97 Operating section 100 Space Sharing System Ev visualization image P1, P2, P3 users P1v, P2v, P3v Avatars SP Real Space VSP Virtual Space

Claims

1. A first generation means generates a first view in which virtual objects corresponding to movable real objects in the real space are placed in a virtual space that reproduces the real space, A first display means that outputs the first view generated by the first generation means to a display device used by a first user who is not in the physical space, Object motion acquisition means detects the tactile actions of a second user in the real space towards the real object based on images acquired via an imaging unit on a device worn by the second user, and acquires real object motion data including the position and orientation of the real object. A synchronization means for synchronizing the position and orientation of the virtual object corresponding to the real object based on the real object motion data acquired by the object motion acquisition means, A first update means updates the first view output to the first display means to a synchronized state by the synchronization means, A space-sharing system equipped with these features.

2. In the spatial sharing system described in claim 1, The first generation means generates the first view in which a first avatar corresponding to the first user is further placed in the virtual space, and is a space-sharing system.

3. In the spatial sharing system described in claim 1, The first generation means generates the first view by further placing a second avatar corresponding to the second user in the real space at a predetermined position in the virtual space corresponding to the position of the second user in the real space.

4. In the spatial sharing system according to any one of claims 1 to 3, A simulation means for performing a predetermined simulation in the virtual space, A change visualization means that generates a visualization image of the simulation results of the virtual space, corresponding to the position and movement of each object in the virtual space, including the virtual object corresponding to the real object, by the simulation means, Equipped with, The first update means is a spatial sharing system that reflects the visualization image generated by the change visualization means in the first view output to the first display means.

5. In the spatial sharing system described in claim 4, A second generation means generates a second view by superimposing the visualization image generated by the change visualization means onto a predetermined position in the real space corresponding to the virtual space, A second display means outputs the second view generated by the second generation means to a display unit on the second user's mounting device, A space-sharing system equipped with these features.

6. In the spatial sharing system described in claim 5, The system includes a first avatar acquisition means for acquiring first avatar data, including the position of the first avatar corresponding to the first user, The second generation means generates a second view by further superimposing the first avatar at a predetermined position in the real space corresponding to the position of the first avatar in the virtual space, based on the first avatar data acquired by the first avatar acquisition means, in a space sharing system.

7. In the spatial sharing system described in claim 6, The first avatar acquisition means is a spatial sharing system that acquires first avatar data, including the position of the first user, based on an image acquired via a camera unit on a device worn by the first user.

8. In the spatial sharing system described in claim 1, The virtual object is a 3D CG model of the real object, and the system is a spatial sharing system.

9. In the spatial sharing system described in claim 5, The virtual object is a 3D CG model of an object with an approximate shape different from that of the actual object. The second generation means is a spatial sharing system that generates the second view in which the virtual object is further superimposed on the real object.

10. In the spatial sharing system described in claim 1, The device, which is a physical object located in the physical space, comprises a control unit and an operating unit, and is connected via a network so as to be communicative. An operation receiving means for receiving operations on the virtual object placed in the virtual space, A command transmission means transmits a command to the control unit of the device to operate the operating unit of the device corresponding to the operation of the virtual object received by the operation reception means, A space-sharing system equipped with these features.

11. A space-sharing device that is communicatively connected to a display device and a mounting device, A first generation means generates a first view in which virtual objects corresponding to movable real objects in the real space are placed in a virtual space that reproduces the real space, A first display means that outputs the first view generated by the first generation means to the display device of a first user who is not in the physical space, Object motion acquisition means for detecting the actions of a second user in the real space on the real object using their sense of touch, based on images acquired from the wearable device worn by the second user, and acquiring real object motion data including the position and orientation of the real object; A synchronization means for synchronizing the position and orientation of the virtual object corresponding to the real object based on the real object motion data acquired by the object motion acquisition means, A first update means updates the first view output to the first display means to a synchronized state by the synchronization means, A space-sharing device equipped with the following features.

12. Computers, A first generation means generates a first view in which virtual objects corresponding to movable real objects in the real space are placed in a virtual space that reproduces the real space, A first display means that outputs the first view generated by the first generation means to a display device of a first user who is not in the physical space, Object motion acquisition means for detecting the tactile actions of a second user in the aforementioned physical space towards the aforementioned physical object based on images acquired from a wearable device worn by the second user, and acquiring physical object motion data including the position and orientation of the physical object; A synchronization means for synchronizing the position and orientation of the virtual object corresponding to the real object based on the real object motion data acquired by the object motion acquisition means, A first update means updates the first view output to the first display means to a synchronized state by the synchronization means, A program to make it work.

Images