Information processing system and program

Abstract

To provide an information processing system expanding a range of use of a virtual space.SOLUTION: An information processing system comprises: first display means for providing a first user with a first view, the view of a mixed reality space formed by disposing a virtual object in a prescribed real space; second display means for providing a second user with a second view of an avatar of the first user disposed in a virtual space imitating the prescribed real space; first input receiving means for receiving input from the first user; and first control means for controlling the avatar of the first user in the virtual space on the basis of the input from the first user received by the first input receiving means.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present invention relates to an information processing system and a program. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there are known techniques for providing a mixed reality (MR) space or a virtual reality (VR) space to a user (see, for example, Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6860488 [Patent Document 2] Patent No. 6800599 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, there has been a demand for expanding the range of uses for virtual space.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to broaden the range of uses for virtual spaces. [Means for solving the problem]

[0006] According to one embodiment shown in the present disclosure, a first display means for providing a first user with a first view, the first view being a view of a mixed reality space in which a virtual object is arranged in a predetermined real space; a second display means for providing a second user with a second view in which an avatar of the first user is placed in a virtual space simulating the predetermined real space; a first input receiving means for receiving an input from the first user; a first control means for controlling an avatar of the first user in the virtual space based on the input from the first user received by the first input receiving means. An information processing system is provided. Effect of the Invention

[0007] According to the present invention, the range of uses for virtual space can be expanded. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a schematic configuration of an information processing system. [Diagram 2] FIG. 1 is a diagram showing a schematic configuration of a VR system. [Diagram 3] FIG. 1 is a diagram showing a schematic configuration of an MR system. [Figure 4] FIG. 1 is a diagram conceptually illustrating a virtual space. [Diagram 5] 1 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from an X direction. [Figure 6] 13 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from the Y direction. [Figure 7] FIG. 2 is a diagram illustrating a schematic configuration of a controller. [Figure 8] FIG. 2 is a diagram illustrating a mixed reality space and a virtual space. [Figure 9] FIG. 2 is a diagram illustrating a functional configuration of the information processing system. [Figure 10] 1A and 1B are diagrams illustrating a real object and a virtual object corresponding to the real object. [Figure 11] 13 is a flowchart illustrating an example of processing executed by the VR system. [Figure 12] 13 is a flowchart illustrating an example of processing executed by the MR system. [Figure 13] 13 is a flowchart showing an example of a process for reflecting an action of an MR user in a virtual space. [Figure 14]13 is a flowchart showing an example of a process for reflecting the action of a VR user in a mixed reality space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <Configuration of information processing system> As shown in FIG. 1, the information processing system 100 of the present embodiment includes a VR (Virtual Reality) system 200, an MR (Mixed Reality) system 400, a server 600, and an external device 700. The VR system 200 is configured to be able to communicate with the server 600, the MR system 400, and the external device 700 via a network 2. The MR system 400 is configured to be able to communicate with the server 600, the VR system 200, and the external device 700 via the network 2. The number of MR systems 400 constituting the information processing system 100 is not limited to one, and may be multiple. The number of VR systems 200 constituting the information processing system 100 is not limited to one, and may be multiple. In the present embodiment, communication between the VR system 200 and the MR system 400 via the network 2 is performed via the server 600, but may be performed without the server 600. The network 2 may be composed of, for example, the Internet, a mobile communication system (e.g., 3G, 4G, 5G, LTE (Long Term Evolution), etc.), Wi-Fi (registered trademark), Bluetooth (registered trademark), other communication lines, or a combination of these.

[0011] (VR system configuration) 2, the VR system 200 includes a VR device 210, a computer 300, a detection device 260, a display 270, and a controller 280. The VR device 210 includes a display 211, a gaze sensor 212, a first camera 213, a second camera 214, a microphone 215, a speaker 216, and a sensor 217. Hereinafter, a user who uses the VR device 210 is referred to as a VR user.

[0012] The computer 300 can be connected to the Internet or other network 2, and can communicate with, for example, a server 600, a computer of the MR system 400, and other computers connected to the network 2. Examples of other computers include computers of other VR systems 200 and external devices 700.

[0013] The VR device 210 is mounted on the head of a VR user and can provide a virtual space to the VR user during operation. The VR device 210 can be, for example, a so-called head-mounted display equipped with a display, or a head-mounted device equipped with a smartphone or other terminal having a display. The VR device 210 displays, for example, an image for the right eye and an image for the left eye on the display 211. When each eye of the VR user views the respective image, the VR user can recognize the image as a three-dimensional image based on the parallax between the two eyes.

[0014] The display 211 is realized, for example, as a non-transparent display device. The display 211 is disposed on the main body of the VR device 210 so as to be located, for example, in front of both eyes of the VR user. Therefore, when the VR user visually recognizes the three-dimensional image displayed on the display 211, the VR user can be immersed in the virtual space. The display 211 may be realized by a display provided in a so-called smartphone or other terminal.

[0015] The detection device 260 detects the movement of a VR user who uses the VR device 210. The detection device 260 may have a position tracking function for detecting the movement of the VR device 210, thereby detecting the movement of the VR user. Specifically, the detection device 260 may have a sensor for reading light (e.g., infrared light) from the VR device 210 as a sensor for detecting the movement of the VR user, and may detect the position, inclination, etc. of the VR device 210 in the real space. In this case, the VR device 210 may have a plurality of light sources not shown. Also, each light source may be realized by, for example, an LED (Light Emitting Diode) that emits infrared light.

[0016] The detection device 260 may be realized by, for example, a camera. Specifically, the detection device 260 may have an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor) as a sensor for detecting the movement of the VR user. In other words, the movement of the VR user may be detected by a camera. The detection device 260 may be, for example, a depth camera equipped with a device that emits predetermined light such as infrared light or light of a predetermined pattern, and an image sensor (for example, a depth sensor), and may detect the reflected light of the light emitted from the device by the image sensor, and may detect the position, posture, etc. of the VR user based on the output from the image sensor. The detection device 460 may be equipped with such a depth camera and a camera that can acquire an RGB image, and may detect the position, posture, etc. of the VR user based on the output from these cameras. The detection device 260 may be, for example, a stereo camera equipped with multiple image sensors, and may detect the position, posture, etc. of the VR user based on the output from the multiple image sensors. In addition, detection of the position, posture, etc. of the VR user may be detection of the position, posture, etc. of the VR user's body, or detection of the position, inclination, etc. of the VR device 210.

[0017] The VR system 200 may have one or more types of detection devices as the detection device 260, and may have a plurality of each type of detection device. The VR system 200 may not have the detection device 260. A part of the detection device 260 may be configured by the computer 300 or the like. For example, analysis of the output from the image sensor (e.g., image recognition, etc.) may be performed by the computer 300 or the like.

[0018] In addition to the above-mentioned various sensors, the detection device 260 may have a sensor capable of detecting the position, inclination, etc. of the detection device 260 itself. Specifically, for example, the detection device 260 may have an angular velocity sensor (for example, a three-axis angular velocity sensor), an acceleration sensor (for example, a three-axis acceleration sensor), or a geomagnetic sensor (for example, a three-axis geomagnetic sensor). In addition, the output from these sensors may be sent to the computer 300 or the like, and may be used, for example, when a predetermined process is performed based on the output from an image sensor included in the detection device 260.

[0019] Furthermore, the VR device 210 may include a sensor 217 instead of or in addition to the detection device 260 as a detection means for detecting the movement of the VR user. The VR device 210 may detect the position and inclination of the VR device 210 itself using the sensor 217. The sensor 217 may be, for example, an angular velocity sensor (for example, a three-axis angular velocity sensor), an acceleration sensor (for example, a three-axis acceleration sensor), or a geomagnetic sensor (for example, a three-axis geomagnetic sensor). Furthermore, the VR device 210 may have one or more types of sensors as the sensor 217, and may have a plurality of sensors of each type. As an example, when an angular velocity sensor is used as the sensor 217, the VR device 210 can detect the angular velocity of the VR device 210 around the three axes in the real space over time. Then, the VR device 210 can calculate the change over time of the angle of the VR device 210 around the three axes based on each angular velocity, and further calculate the inclination of the VR device 210 based on the change over time of the angle. Furthermore, the sensor 217 may be, for example, an image sensor. Then, the position, posture, etc. of the VR user may be detected based on the output from the image sensor. In other words, the position, posture, etc. of the VR user may be detected based on information from a camera that captures the surroundings of the VR device 210 provided in the VR device 210. In other words, tracking of the VR device 210 may be performed by an outside-in method or an inside-out method.

[0020] The gaze sensor 212 detects the direction in which the gaze of the right eye and the left eye of the VR user are directed. That is, the gaze sensor 212 detects the gaze of the VR user (in other words, the movement of the eyes). The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 212 is realized by a sensor having the eye tracking function. The gaze sensor 212 may include a sensor for the right eye and a sensor for the left eye. The gaze sensor 212 may be, for example, a sensor that irradiates the right eye and the left eye of the VR user with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and the iris in response to the irradiated light. In this case, the gaze sensor 212 can detect the gaze of the VR user based on each detected rotation angle.

[0021] The first camera 213 photographs the lower part of the face of the VR user. More specifically, the first camera 213 photographs the nose and mouth of the VR user. The second camera 214 photographs the eyes and eyebrows of the VR user. Here, the VR user side of the housing of the VR device 210 is defined as the inside of the VR device 210, and the opposite side of the housing of the VR device 210 from the VR user is defined as the outside of the VR device 210. The first camera 213 may be disposed outside the VR device 210, and the second camera 214 may be disposed inside the VR device 210. Images photographed by the first camera 213 and the second camera 214 are input to the computer 300. The first camera 213 and the second camera 214 may be realized as one camera, and the face of the VR user may be photographed by this one camera.

[0022] The microphone 215 as a sound input means converts the voice of the VR user into a sound signal (in other words, an electrical signal) and outputs it to the computer 300. The speaker 216 as a sound output means converts the sound signal into sound and outputs it to the VR user. Note that the VR device 210 may include earphones instead of the speaker 216 as a sound output means.

[0023] The display 270 displays an image similar to the image displayed on the display 211. This allows users other than the VR user wearing the VR device 210 to view the same image (in other words, a virtual space) as the VR user. The image displayed on the display 270 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the VR device 210. Examples of the display 270 include a liquid crystal display and an organic EL display.

[0024] The controller 280 is connected to the computer 300 by wire or wirelessly. The controller 280 accepts an input operation related to an instruction from the VR user to the computer 300. The controller 280 also accepts an input operation by the VR user for controlling the position and movement of a virtual object arranged in a virtual space. The controller 280 may be configured to be held by the VR user, for example. The controller 280 may also be configured to be worn on a part of the body or clothing of the VR user, for example. Specifically, the controller 280 may be, for example, a glove type controller. The controller 280 may also be configured to be capable of outputting at least one of vibration, sound, and light based on a signal transmitted from the computer 300.

[0025] The controller 280 may also have a plurality of light sources. Each light source may be realized, for example, by an LED that emits infrared light. The detection device 260 may read the infrared light from the controller 280 and detect the position and inclination of the controller 280 in the real space. In other words, the detection device 260 may have a position tracking function that detects the movement of the controller 280, thereby detecting the movement of the VR user.

[0026] It should be noted that the VR system 200 does not necessarily have to include the controller 280.

[0027] The information processing system 100 may also have a sensor 286 that detects the movement of the VR user. The sensor 286 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. The sensor 286 may also be provided in the controller 280 (see FIG. 7). The information processing system 100 may have one or more types of sensors as the sensor 286, and may have a plurality of sensors of each type. The device including the sensor 286 (for example, the controller 280 or a predetermined camera) and the computer 300 may be connected to each other, for example, wirelessly. Information acquired by the sensor 286 may also be transmitted to the computer 300, for example, by wireless communication.

[0028] In addition, the position and movement of the VR user may be detected by the VR device 210 being equipped with a device that performs a predetermined wireless communication (e.g., short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB (Ultra Wide Band) communication), and the device performing wireless communication with surrounding devices to obtain position information of the VR device 210.

[0029] The position and movement of the VR user may be detected using an external device 700 (and a device that tracks the movement of the VR user in cooperation with the device via short-range wireless communication or the like) that can track the movement of the VR user, such as a device worn by the VR user (for example, a wearable device such as a watch type, wristband type, ring type, or clothing type, or an implantable device, etc.). These external devices 700 may also be used as the controller 280. The position and movement of the VR user may also be detected by a GPS sensor or the like.

[0030] (Configuration of computer 300) 2, the computer 300 includes, as its main components, a processor 301, a memory 302, a storage 303, an input / output interface 304, and a communication interface 305. The components are connected to each other via a bus.

[0031] The processor 301 controls the operation of the VR device 210. The processor 301 reads a program from the storage 303 and loads it in the memory 302. The processor 301 executes the loaded program. The processor 301 may be configured to include, for example, one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), and a field-programmable gate array (FPGA).

[0032] The memory 302 is a main storage device. The memory 302 is configured by storage devices such as a read only memory (ROM) and a random access memory (RAM). The memory 302 temporarily stores the programs and various data that the processor 301 reads from the storage 303, thereby providing a working area for the processor 301. The memory 302 also temporarily stores various data generated while the processor 301 is operating according to the programs, various data input to the computer 300, and the like.

[0033] The storage 303 is an auxiliary storage device. The storage 303 is configured by a storage device such as a flash memory or a hard disk drive (HDD). The storage 303 stores a program for providing a service in the information processing system 100. The storage 303 also stores various data for providing a service in the information processing system 100. The data stored in the storage 303 includes data for defining a virtual space, data related to virtual objects, and the like.

[0034] The storage 303 may be realized as a removable storage device such as a memory card. Also, instead of the storage 303 built into the computer 300, the computer 300 may be configured to use programs and data stored in an external storage device.

[0035] The input / output interface 304 is an interface for the computer 300 to receive input of data and is also an interface for the computer 300 to output data. The input / output interface 304 can transmit or receive data between the VR device 210, the detection device 260, and the display 270. The input / output interface 304 may also transmit or receive data between the display 211, the gaze sensor 212, the first camera 213, the second camera 214, the microphone 215, the speaker 216, and the sensor 217 included in the VR device 210.

[0036] The input / output interface 304 may transmit or receive data to or from the controller 280. For example, the input / output interface 304 may receive input of signals output from the controller 280 and the sensor 286. The input / output interface 304 may also transmit an instruction output from the processor 301 to the controller 280. The instruction may instruct the controller 280 to vibrate, output sound, emit light, or the like. Upon receiving the instruction, the controller 280 performs vibration, output sound, emit light, or the like in accordance with the instruction.

[0037] It should be noted that each of the VR device 210, the detection device 260, the display 270, the controller 280, and the sensor 286 may be connected to the computer 300 via a wired or wireless connection.

[0038] The communication interface 305 controls transmission and reception of various data via the network 2 with other computers (eg, the server 600, the computer 500, or other computers 300, etc.).

[0039] The processor 301 accesses the storage 303, loads a program stored in the storage 303 into the memory 302, and executes a series of instructions included in the program. The processor 301 also sends a signal for providing a virtual space to the VR device 210 via the input / output interface 304.

[0040] The computer 300 may be provided outside the VR device 210, or a part or the whole of the computer 300 may be built into the VR device 210. Also, a portable terminal (e.g., a smartphone) including a display 211 may function as at least a part of the computer 300. When there are multiple VR devices 210, the computer 300 may be provided for each VR device 210, or may be used in common for the multiple VR devices 210.

[0041] (Coordinate System) A real coordinate system, which is a coordinate system in real space, is set in advance in the VR system 200. The real coordinate system has three reference directions (axes) that are parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-rear direction perpendicular to both the vertical and horizontal directions.

[0042] The position and tilt of the VR device 210 in the real space can be detected by the detection device 260 and the sensor 217. The detected tilt of the VR device 210 corresponds to, for example, each tilt around the three axes of the VR device 210 in the real coordinate system. The computer 300 sets a uvw field of view coordinate system for the VR device 210 based on the tilt of the VR device 210 in the real coordinate system (see FIG. 4). The uvw field of view coordinate system set for the VR device 210 corresponds to a viewpoint coordinate system when a VR user wearing the VR device 210 views an object in a virtual space.

[0043] Here, the uvw visual coordinate system will be described. The computer 300 sets a three-dimensional uvw visual coordinate system with the head of the VR user wearing the VR device 210 as the center (origin). More specifically, the computer 300 sets three directions newly obtained by tilting the horizontal direction, vertical direction, and front-rear direction that define the real coordinate system around each axis by the tilt around each axis of the VR device 210 in the real coordinate system as the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the uvw visual coordinate system in the VR device 210.

[0044] For example, when a VR user wearing the VR device 210 stands upright and looks straight ahead, the processor 301 sets a visual field coordinate system parallel to the real coordinate system in the VR device 210. In this case, the horizontal direction, vertical direction, and front-back direction in the real coordinate system coincide with the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the visual field coordinate system in the VR device 210.

[0045] After the uvw viewing coordinate system is set in the VR device 210, the detection device 260 or the sensor 217 can detect the tilt of the VR device 210 in the set uvw viewing coordinate system based on the movement of the VR device 210. In this case, the detection device 260 or the sensor 217 detects the pitch angle, yaw angle, and roll angle of the VR device 210 in the uvw viewing coordinate system as the tilt of the VR device 210. The pitch angle represents the tilt angle of the VR device 210 around the pitch axis in the uvw viewing coordinate system. The yaw angle represents the tilt angle of the VR device 210 around the yaw axis in the uvw viewing coordinate system. The roll angle represents the tilt angle of the VR device 210 around the roll axis in the uvw viewing coordinate system.

[0046] The computer 300 sets, in the VR device 210, the uvw visual field coordinate system of the VR device 210 after the VR device 210 moves, based on the detected inclination of the VR device 210. The relationship between the VR device 210 and the uvw visual field coordinate system of the VR device 210 is always constant, regardless of the position and inclination of the VR device 210. When the position and inclination of the VR device 210 change, the position and inclination of the uvw visual field coordinate system of the VR device 210 in the real coordinate system change in conjunction with the change in the position and inclination.

[0047] The detection device 260 may specify the position of the VR device 210 in the real space as a relative position with respect to the detection device 260. The processor 301 may also determine the origin of the uvw field of view coordinate system of the VR device 210 in the real space (actual coordinate system) based on the specified relative position.

[0048] (Virtual space) The virtual space will be further described with reference to FIG. 4. FIG. 4 is a diagram conceptually illustrating one mode of expressing the virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure covering the entire 360-degree direction of the center 12. In FIG. 4, in order to avoid complicating the description, the upper half of the celestial sphere in the virtual space 11 is illustrated. Each mesh is defined in the virtual space 11. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system, which is a global coordinate system defined in the virtual space 11. The computer 300 associates each partial image constituting the panoramic image 13 (still image, video, etc.) that can be deployed in the virtual space 11 with each corresponding mesh in the virtual space 11.

[0049] In the virtual space 11, an XYZ coordinate system is defined with a predetermined point as the origin 12. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis of the XYZ coordinate system is parallel to the horizontal direction of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the vertical direction of the real coordinate system, and the Z-axis (front-back direction) of the XYZ coordinate system is parallel to the front-back direction of the real coordinate system.

[0050] When the VR device 210 is started up, that is, in the initial state of the VR device 210, the virtual camera 14 is placed at a predetermined position (for example, the center) in the virtual space 11. Furthermore, the processor 301 displays an image captured by the virtual camera 14 on the display 211 of the VR device 210. The virtual camera 14 moves in the virtual space 11 in conjunction with the movement of the VR device 210 in the real space. This allows changes in the inclination and position of the VR device 210 in the real space to be reproduced in the virtual space 11 in the same manner.

[0051] A uvw visual coordinate system is defined for the virtual camera 14, similarly to the case of the VR device 210. The uvw visual coordinate system of the virtual camera 14 in the virtual space 11 is defined so as to be linked to the uvw visual coordinate system of the VR device 210 in the real space (actual coordinate system). Therefore, when the inclination of the VR device 210 changes, the inclination of the virtual camera 14 also changes accordingly. Note that the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in the real space, but in this embodiment, even if the VR user moves in the real space, it does not move in the virtual space 11.

[0052] The processor 301 of the computer 300 defines a viewing area 15 in the virtual space 11 based on the position and inclination of the virtual camera 14. The viewing area 15 corresponds to an area of ​​the virtual space 11 that is visually recognized by a VR user wearing the VR device 210. In other words, the position of the virtual camera 14 can be said to be the viewpoint of the VR user in the virtual space 11.

[0053] The uvw field of view coordinate system of the VR device 210 is equal to the viewpoint coordinate system when the VR user views the display 211. The uvw field of view coordinate system of the virtual camera 14 is linked to the uvw field of view coordinate system of the VR device 210. Therefore, the line of sight of the VR user detected by the gaze sensor 212 can be regarded as the line of sight of the VR user in the uvw field of view coordinate system of the virtual camera 14.

[0054] (Visual area) The field of view 15 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing a YZ cross section of the field of view 15 in the virtual space 11 as viewed from the X direction. Fig. 6 is a diagram showing an XZ cross section of the field of view 15 in the virtual space 11 as viewed from the Y direction.

[0055] 5, the field of view 15 in the YZ cross section includes an area 18. The area 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 301 defines as the area 18 a range including the polar angle α centered on the reference line of sight 16 in the virtual space.

[0056] 6, the field of view 15 in the XZ cross section includes an area 19. The area 19 is defined by the position of the virtual camera 14, the reference line of sight 16, and the XZ cross section of the virtual space 11. The processor 301 defines a range including an azimuth angle β centered on the reference line of sight 16 in the virtual space 11 as the area 19. The polar angles α and β are determined according to the position of the virtual camera 14 and the inclination (direction) of the virtual camera 14.

[0057] The VR system 200 provides the VR user with a field of view in the virtual space 11 by displaying a field of view image 17 on the display 211 based on a signal from the computer 300 (see FIG. 4). The field of view image 17 is an image corresponding to a portion of the panoramic image 13 that corresponds to the field of view area 15. When the VR user moves the VR device 210 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the field of view area 15 in the virtual space 11 changes. Thereby, the field of view image 17 displayed on the display 211 is updated to an image of the panoramic image 13 that is superimposed on the field of view area 15 in the direction in which the VR user is facing in the virtual space 11. The VR user can view a desired direction in the virtual space 11.

[0058] While wearing the VR device 210, the VR user can view only the panoramic image 13 deployed in the virtual space 11, without viewing the real world. Therefore, the information processing system 100 can provide the VR user with a highly immersive feeling in the virtual space 11.

[0059] The virtual camera 14 may include two virtual cameras, that is, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In this case, an appropriate parallax is set for the two virtual cameras so that the VR user can recognize the three-dimensional virtual space 11. The virtual camera 14 may be realized by one virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by one virtual camera.

[0060] (controller) An example of the controller 280 will now be described with reference to FIG.

[0061] As shown in FIG. 7, the controller 280 may include a right controller 280R and a left controller (not shown). In this case, the right controller 280R is operated by the right hand of the VR user. The left controller is operated by the left hand of the VR user. In one aspect, the right controller 280R and the left controller are configured as separate devices. Therefore, the VR user can freely move the right hand holding the right controller 280R and the left hand holding the left controller. In another aspect, the controller 280 may be an integrated controller that accepts operations from both hands. The right controller 280R will be described below.

[0062] The right controller 280R includes a grip 281, a frame 282, a top surface 283, buttons 284 and 285, a sensor (e.g., a motion sensor) 286, an infrared LED 287, buttons 288 and 289, and an analog stick 290. The grip 281 is configured to be held by the right hand of a VR user. For example, the grip 281 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the VR user's right hand.

[0063] Button 284 is disposed on a side surface of grip 281 and is operated by the middle finger of the right hand. Button 285 is disposed on the front surface of grip 281 and is operated by the index finger of the right hand. Buttons 284 and 285 are also provided with switches that detect the movement of the user pressing buttons 284 and 285. Buttons 284 and 285 may be configured as trigger-type buttons.

[0064] The sensor 286 is built into the housing of the grip 281. The sensor 286 detects the movement of the VR user. Specifically, the sensor 286 detects the movement of the hand of the VR user. For example, the sensor 286 detects the rotation speed, number of rotations, etc. of the hand. The controller 280 does not necessarily have to include the sensor 286.

[0065] A plurality of infrared LEDs 287 are arranged along the circumferential direction of the frame 282. The infrared LEDs 287 emit infrared light in accordance with the progress of a program that uses the controller 280 while the program is being executed. The infrared light emitted from the infrared LEDs 287 can be used to detect the positions and attitudes (tilt, direction) of the right controller 280R and the left controller.

[0066] The top surface 283 includes buttons 288, 289 and an analog stick 290. The buttons 288, 289 are operated by the thumb of the right hand of the VR user. The buttons 288, 289 are also equipped with switches that detect the movement of the user pressing the buttons 288, 289. The analog stick 290 is operable in any direction within 360 degrees from an initial position (neutral position). The operation includes, for example, an operation for moving an object placed in the virtual space 11. The analog stick 290 is also equipped with a sensor that detects the movement of the user operating the analog stick 290.

[0067] Information regarding the input operation of the VR user (in other words, the movement of the VR user) detected by the controller 280 is sent to the computer 300.

[0068] 7, for example, the yaw, roll, and pitch directions are defined for the right hand of a VR user in the controller 280. Specifically, for example, when a VR user extends his or her thumb and index finger, the direction in which the thumb extends is defined as the yaw direction, the direction in which the index finger extends is defined as the roll direction, and the direction perpendicular to the plane defined by the axis of the yaw direction and the axis of the roll direction is defined as the pitch direction.

[0069] (MR system configuration) 3, the MR system 400 includes an MR device 410, a computer 500, a detection device 460, a display 470, and a controller 480. The MR device 410 includes a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417. Hereinafter, a user who uses the MR device 410 is referred to as an MR user 6.

[0070] The computer 500 can be connected to the Internet or other network 2, and can communicate with, for example, a server 600, a computer of the VR system 200, and other computers connected to the network 2. Examples of other computers include computers of other MR systems 400, external devices 700, etc.

[0071] The MR device 410 is worn on the head of the MR user 6 and can provide a mixed reality space to the MR user 6 during operation. The MR device 410 may be, for example, a glasses-type device (for example, so-called MR glasses) equipped with a display. The MR device 410 may also be a contact lens-type device or the like.

[0072] The display 411 is realized as, for example, a transmissive display device. The transmissive display 411 may temporarily function as a non-transmissive display device by adjusting its transmittance. The display 411 is disposed on the main body of the MR device 410 so as to be located in front of both eyes of the MR user 6. Therefore, the MR user 6 is provided with a display in which a virtual object displayed by the display 411 is superimposed on the real space seen through the transmissive display 411. That is, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which a virtual object is disposed in the real space. In other words, the MR user 6 can simultaneously see a virtual object and an object (hereinafter referred to as a "real object") existing in the real space by the MR device 410.

[0073] The detection device 460 detects the movement of the MR user 6 using the MR device 410. The detection device 460 may have a position tracking function for detecting the movement of the MR device 410, thereby detecting the movement of the MR user 6. Specifically, the detection device 460 may have, for example, a sensor for reading light (e.g., infrared light) from the MR device 410 as a sensor for detecting the movement of the MR user 6, and may detect the position, inclination, etc. of the MR device 410 in the mixed reality space. In this case, the MR device 410 may have multiple light sources not shown. Also, each light source may be realized, for example, by an LED (Light Emitting Diode) that emits infrared light.

[0074] The detection device 460 may be realized by, for example, a camera. Specifically, the detection device 460 may have an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor) as a sensor for detecting the movement of the MR user 6. In other words, the movement of the MR user 6 may be detected by a camera. The detection device 460 may be, for example, a depth camera equipped with a device that emits a predetermined light such as infrared light or light of a predetermined pattern, and an image sensor (for example, a depth sensor), and may detect the reflected light of the light emitted from the device by the image sensor, and may detect the position, posture, etc. of the MR user 6 based on the output from the image sensor. The detection device 460 may be equipped with such a depth camera and a camera that can acquire an RGB image, and may detect the position, posture, etc. of the MR user 6 based on the output from these cameras. The detection device 460 may be, for example, a stereo camera equipped with multiple image sensors, and may detect the position, posture, etc. of the MR user 6 based on the output from the multiple image sensors. Note that detection of the position, posture, etc. of the MR user 6 may be detection of the body position, posture, etc. of the MR user 6, or detection of the position, inclination, etc. of the MR device 410.

[0075] The MR system 400 may have one or more types of detection devices as the detection device 460, and may have a plurality of each type of detection device. For example, the MR system 400 may have a plurality of cameras (in other words, a plurality of sensors), and the plurality of cameras may function as the detection device 460. In this embodiment, the MR system 400 is described as having three detection devices 460, and each detection device 460 is described as having a depth camera and a camera capable of acquiring an RGB image. A part of the detection device 460 may be configured by the computer 500 or the like. For example, analysis of the output from the image sensor (e.g., image recognition, etc.) may be performed by the computer 500 or the like. The MR system 400 may not have the detection device 460.

[0076] In addition to the above-mentioned various sensors, the detection device 460 may have a sensor capable of detecting the position, inclination, etc. of the detection device 460 itself. Specifically, for example, the detection device 460 may have an angular velocity sensor (for example, a three-axis angular velocity sensor), an acceleration sensor (for example, a three-axis acceleration sensor), or a geomagnetic sensor (for example, a three-axis geomagnetic sensor). In addition, the output from these sensors may be sent to the computer 500 or the like, and may be used, for example, when a predetermined process is performed based on the output from an image sensor included in the detection device 460.

[0077] As will be understood from the configuration of the MR device 410 described later, the MR device 410 of another MR user 6 may be used as the detection device 460.

[0078] Furthermore, the MR device 410 may include a sensor 417 instead of or in addition to the detection device 460 as a detection means for detecting the movement of the MR user 6. The MR device 410 may detect the position and inclination of the MR device 410 itself using the sensor 417. The sensor 417 may be, for example, an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). Furthermore, the MR device 410 may include one or more types of sensors as the sensor 417, and may include a plurality of sensors of each type.

[0079] The gaze sensor 412 detects the direction in which the gaze of the right eye and the left eye of the MR user 6 is directed. That is, the gaze sensor 412 detects the gaze of the MR user 6 (in other words, the movement of the eyes). The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 412 is realized by a sensor having the eye tracking function. The gaze sensor 412 may include a sensor for the right eye and a sensor for the left eye. The gaze sensor 412 may be, for example, a sensor that irradiates the right eye and the left eye of the MR user 6 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and the iris of the irradiated light. In this case, the gaze sensor 412 can detect the gaze of the MR user 6 based on each detected rotation angle.

[0080] The camera 413 captures the surroundings (for example, in front) of the MR user 6. The MR device 410 may include, for example, a depth camera and a camera capable of acquiring RGB images as the camera 413. Based on the output from the camera 413, the computer 500 may detect the shape of objects around the MR device 410 (in other words, the MR user 6), the relative distance between the MR device 410 and the surrounding objects, and the like. The position and inclination of the MR device 410 may also be detected based on the output from the camera 413. In other words, tracking of the MR device 410 may be performed by an outside-in method or an inside-out method.

[0081] The microphone 415 as a sound input means converts the voice of the MR user 6 into a sound signal (in other words, an electrical signal) and outputs it to the computer 500. The speaker 416 as a sound output means converts the sound signal into sound and outputs it to the MR user 6. Note that the MR device 410 may include earphones instead of the speaker 416 as a sound output means.

[0082] The display 470 displays, for example, an image similar to the image displayed on the display 411. This allows users other than the MR user 6 wearing the MR device 410 to view the same image as the MR user 6. The image displayed on the display 470 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the MR device 410. Examples of the display 470 include a liquid crystal display and an organic EL display. Note that the display 470 may display an image showing a view of the mixed reality space provided by the MR device 410 to the MR user 6.

[0083] The controller 480 is connected to the computer 500 by wire or wirelessly. The controller 480 accepts an input operation related to an instruction from the MR user 6 to the computer 500. The controller 480 also accepts an input operation by the MR user 6 for controlling the position and movement of a virtual object placed in the mixed reality space. The controller 480 may be configured to be held by the MR user 6, for example. The controller 480 may also be configured to be wearable on a part of the body or clothing of the MR user 6, for example. The controller 480 may also be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 500.

[0084] The controller 480 may also have a plurality of light sources. Each light source may be realized, for example, by an LED that emits infrared light. The detection device 460 may read the infrared light from the controller 480 and detect the position and inclination of the controller 480 in the mixed reality space. In other words, the detection device 460 may have a position tracking function that detects the movement of the controller 480, thereby detecting the movement of the MR user 6.

[0085] It should be noted that the MR system 400 does not necessarily have to include the controller 480. In addition, the controller 480 may include, for example, a part or all of the components included in the controller 280.

[0086] The information processing system 100 may have a sensor 486 that detects the movement of the MR user 6. The sensor 486 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. The sensor 486 may be provided in the controller 480, for example. The information processing system 100 may have one or more types of sensors as the sensor 486, and may have a plurality of sensors of each type. A device including the sensor 486 (for example, the controller 480 or a predetermined camera) and the computer 500 may be connected to each other, for example, wirelessly. Information acquired by the sensor 486 may be transmitted to the computer 300, for example, by wireless communication.

[0087] In addition, the position and movement of the MR user 6 may be detected by the MR device 410 being equipped with a device that performs a predetermined wireless communication (e.g., short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB communication) and having the device perform wireless communication with surrounding devices to obtain position information of the VR device 210.

[0088] The position and movement of the MR user 6 may be detected using an external device 700 capable of tracking the movement of the MR user 6 (and a device that tracks the movement of the MR user 6 in cooperation with the device via short-range wireless communication or the like), such as a device worn by the MR user 6 (for example, a wearable device such as a watch type, wristband type, ring type, or clothing type, or an implantable device, etc.). These external devices 700 may also be used as the controller 480. The position and movement of the MR user 6 may also be detected by a GPS sensor or the like.

[0089] (Configuration of computer 500) 3, the computer 500 includes, as its main components, a processor 501, a memory 502, a storage 503, an input / output interface 504, and a communication interface 505. The components are connected to each other via a bus.

[0090] The processor 501 controls the operation of the MR device 410. The processor 501 reads a program from the storage 503 and loads it in the memory 502. The processor 501 executes the loaded program. The processor 501 may be configured to include, for example, one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), and a field-programmable gate array (FPGA).

[0091] The memory 502 is a main storage device. The memory 502 is configured by storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 502 temporarily stores the programs and various data read by the processor 501 from the storage 503, thereby providing a working area for the processor 501. The memory 502 also temporarily stores various data generated while the processor 501 is operating according to the programs, various data input to the computer 500, and the like.

[0092] The storage 503 is an auxiliary storage device. The storage 503 is configured by a storage device such as a flash memory or a hard disk drive (HDD). The storage 503 stores a program for providing a service in the information processing system 100. The storage 503 also stores various data for providing a service in the information processing system 100. The data stored in the storage 503 includes data for defining a mixed reality space, data related to virtual objects, and the like.

[0093] Storage 503 may be realized as a removable storage device such as a memory card. Also, instead of storage 503 built into computer 500, programs and data stored in an external storage device may be used.

[0094] The input / output interface 504 is an interface for the computer 500 to receive input of data and also an interface for the computer 500 to output data. The input / output interface 504 may transmit or receive data between the MR device 410, the detection device 460, and the display 470. The input / output interface 504 may also transmit or receive data between the display 411, the gaze sensor 412, the camera 413, the microphone 415, the speaker 416, and the sensor 417 included in the MR device 410.

[0095] The input / output interface 504 may transmit or receive data to or from the controller 480. For example, the input / output interface 504 may receive input of signals output from the controller 480 and the sensor 486. The input / output interface 504 may also transmit an instruction output from the processor 501 to the controller 480. The instruction may instruct the controller 480 to vibrate, output sound, emit light, or the like. Upon receiving the instruction, the controller 480 performs vibration, output sound, emit light, or the like in accordance with the instruction.

[0096] The MR device 410, the detection device 460, the display 470, the controller 480, and the sensor 486 may be connected to the computer 500 via a wired connection or a wireless connection.

[0097] The communication interface 505 controls transmission and reception of various data via the network 2 with other computers (for example, the server 600, the computer 300, or other computers 500, etc.).

[0098] The processor 501 accesses the storage 503, loads a program stored in the storage 503 into the memory 502, and executes a series of instructions included in the program. The processor 501 also sends a signal for providing a mixed reality space to the MR device 410 via the input / output interface 504.

[0099] The computer 500 may be provided outside the MR device 410, or a part or the whole of the computer 500 may be built into the MR device 410. When there are a plurality of MR devices 410, the computer 500 may be provided for each MR device 410, or may be used in common for the plurality of MR devices 410. In this case, a part of the computer 500 may be built into each of the plurality of MR devices 410. In this embodiment, a case where a plurality of MR devices 410 are connected to the computer 500 will be described as an example.

[0100] A method for acquiring the position and inclination of the MR device 410 in the real space (in other words, mixed reality space) is well known, and can be detected by a method similar to that for the VR device 210 described above, so a description thereof will be omitted.

[0101] (Server configuration) The server 600 may transmit a program to the computer 300. The server 600 may also transmit a program to the computer 500. The server 600 also enables communication between the computer 300 and the computer 500. The server 600 also enables communication between the computer 300 and other computers 300. For example, when the information processing system 100 provides a service in which multiple VR users can participate, each computer 300 of a VR user may communicate with the other computers 300 via the server 600, thereby enabling multiple VR users to share an experience in the same virtual space. Note that each computer 300 may communicate with the other computers 300 without going through the server 600.

[0102] 1, the server 600 includes a processor 601, a memory 602, a storage 603, an input / output interface 604, and a communication interface 605. Each of the components is connected to each other via a bus.

[0103] The processor 601 controls the overall operation of the server 600. The processor 601 reads a program from the storage 603 and loads it in the memory 602. The processor 601 executes the loaded program. The processor 601 may be configured to include, for example, one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), and a field-programmable gate array (FPGA).

[0104] The memory 602 is a main storage device. The memory 602 is configured by storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 602 temporarily stores the programs and various data read by the processor 601 from the storage 603, thereby providing a working area for the processor 601. The memory 602 also temporarily stores various data generated while the processor 601 is operating according to the programs, various data input to the server 600, and the like.

[0105] The storage 603 is an auxiliary storage device. The storage 603 is configured by a storage device such as a flash memory or a hard disk drive (HDD). The storage 603 stores a program for providing a service in the information processing system 100. The storage 603 also stores various data for providing a service in the information processing system 100. The data stored in the storage 603 includes data for defining a virtual space, data for defining a mixed reality space, data related to virtual objects, and the like.

[0106] The storage 603 may be realized as a removable storage device such as a memory card. Also, instead of the storage 603 built into the server 600, the server 600 may be configured to use programs and data stored in an external storage device.

[0107] The input / output interface 604 is an interface through which the server 600 receives input of data, and is also an interface through which the server 600 outputs data. The input / output interface 604 can transmit or receive data to or from input devices such as a mouse or a keyboard, and output devices such as a display.

[0108] The communication interface 605 controls the transmission and reception of various data to and from other computers (eg, the computer 300 and the computer 500, etc.) via the network 2.

[0109] The external device 700 may be any device capable of communicating with the computer 300, the computer 500, or the server 600. The external device 700 may be, for example, a device capable of communicating with the computer 300 via the network 2, or may be a device capable of communicating with the computer 300 via short-distance wireless communication or a wired connection. The external device 700 may be, for example, a device capable of communicating with the computer 500 via the network 2, or may be a device capable of communicating with the computer 500 via short-distance wireless communication or a wired connection. The external device 700 may be, for example, a device capable of communicating with the server 600 via the network 2. Examples of the external device 700 include, but are not limited to, a smart device, a PC (Personal Computer), and peripheral devices of the computer 300 or the computer 500.

[0110] <Information processing system control> In the MR system 400 of this embodiment, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which a virtual object is arranged in a real space, similar to a well-known MR device. In the VR system 200 of this embodiment, the VR device 210 provides the VR user with a view of a virtual space, similar to a well-known VR device. In the MR system 400 of this embodiment, the MR user 6 is able to operate a virtual object in the mixed reality space. In the VR system 200 of this embodiment, the VR user is able to operate a virtual object in the virtual space. In addition, in the information processing system 100 of this embodiment, the movement of the VR user using the VR system 200 is reflected in a virtual object included in the view of the mixed reality space provided by the MR system 400. In the information processing system 100 of this embodiment, the movement of the MR user 6 using the MR system 400 is reflected in a virtual object included in the view of the virtual space provided by the VR system 200. In this embodiment, the reflection of the movement of the VR user on the virtual space and the virtual object in the mixed reality space is performed in real time. Furthermore, in this embodiment, the movement of the MR user 6 and the like are reflected in the virtual objects in the virtual space and the mixed reality space in real time.

[0111] In the following, a case where the configuration according to this embodiment is applied to a service (in other words, an application) that enables communication between an MR user 6 and a VR user will be described as an example. Specifically, a case where the configuration according to this embodiment is applied to a service that enables an MR user 6 in a conference room as a predetermined real space and a VR user in a place away from the conference room to hold a conference while simultaneously viewing an object such as a product mockup as a virtual object will be described as an example. Note that the configuration according to this embodiment can also be applied to a system that enables a user to take a school class from home, for example. Also, the configuration according to this embodiment can also be applied to a system that enables a user to participate in a home party held in a specific house from another place. Note that the application of the configuration according to this embodiment is not limited to these services. Also, in the following, a description will be given assuming that a plurality of detection devices 460 are installed in advance at predetermined positions in the conference room as shown in FIG. Also, in the following, a description will be given assuming that three MR users 6, ie, an MR user 6A, an MR user 6B, and an MR user 6C, exist in the conference room. Note that, in the following, an avatar object of a VR user is called a VR avatar 25. Moreover, the avatar object of the MR user 6 is called an MR avatar 26. Specifically, the avatar object of the MR user 6A is called an MR avatar 26A, the avatar object of the MR user 6B is called an MR avatar 26B, and the avatar object of the MR user 6C is called an MR avatar 26C.

[0112] With reference to FIG. 8, a field of view provided to MR users 6A, 6B, and 6C in a real conference room 21 (in other words, a mixed reality space 21) and a field of view provided to a VR user who enters a virtual space 11 simulating the conference room 21 through a VR device 210 will be described. FIG. 8 is a schematic diagram showing the state of the real conference room 21 and the state of the virtual space 11 when the MR users 6A, 6B, and 6C and the VR users are using a service. The upper part of FIG. 8 shows the state of the real conference room 21, and the lower part of FIG. 8 shows the state of the virtual space 11. Note that in FIG. 8, the VR avatar 25 is wearing the VR device 210, but this is for the purpose of making the explanation easier to understand, and in reality, the VR avatar 25 may not be wearing the VR device 210 (virtual VR device 210). Also, in FIG. 8, the MR avatar 26 is not wearing the MR device 410 (virtual MR device 410), but the MR avatar 26 may be wearing the MR device 410.

[0113] As shown in FIG. 8, a desk 8 and the like are arranged in the conference room 21 as real objects. Three detection devices 460 are also arranged in the conference room 21. Three MR users 6A, 6B, and 6C are also present in the conference room 21. Meanwhile, no VR user is present in the conference room 21. Here, a VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as a "virtual model 30") are displayed as virtual objects on the display 411 of the MR device 410 used by each of the MR users 6A, 6B, and 6C. That is, the MR device 410 displays the VR avatar 25 and the virtual model 30 in the real conference room 21 seen through the transparent display 411. Therefore, the MR users 6A, 6B, and 6C are provided with a view as if the VR avatar 25 and the virtual model 30, which do not actually exist in the conference room 21, exist in the conference room 21.

[0114] Moreover, the virtual space 11 viewed by the VR user through the VR device 210 is modeled after the conference room 21, and includes a virtual object 31 of a desk 8 existing in the real conference room 21, etc. Furthermore, the MR avatars 26A, 26B, 26C of the MR users 6A, 6B, 6C, a virtual model 30, etc. are arranged in the virtual space 11. Therefore, the VR user is provided with a view as if he or she were in the conference room 21 where the MR users 6A, 6B, 6C are present.

[0115] In this way, in the information processing system 100 of this embodiment, the MR user 6 can view other MR users 6 and real objects that are actually present in the same place through the transparent display 411. In addition, from the viewpoint of the MR user 6, the VR avatar 25 of the VR user that appears as a virtual object can be viewed. Therefore, the MR user 6 can communicate with the VR user and other MR users 6 without losing the sense of being in the real world. Furthermore, the VR user is in a different place from the MR user 6 in the real world, and in such a case, the shape of the room where the VR user is located is often different from the place where the MR user 6 is located. Therefore, if the VR user also uses the MR device 410 or the like to display the MR avatar 26 in the real space, there is a risk that the display will be unnatural. In this embodiment, the VR user can enter the virtual space 11 and communicate with the MR user 6 or the like, so that such an unnatural feeling can be prevented. In addition, since the virtual space 11 provided to the VR user is a simulation of the location of the MR user 6, when the VR user or the MR user 6 moves around or moves a virtual object in the virtual space 11 or real space, respectively, the movement of the avatar or the virtual model 30 displayed to the other party becomes natural. In addition, when the VR user or the MR user 6 operates a virtual object such as the virtual model 30, it becomes easy to imagine how it appears to the other party, and smooth communication can be achieved.

[0116] Hereinafter, the functional configuration of the information processing system 100 will be described with reference to FIG. 9. The functional configuration shown in FIG. 9 is merely an example. Each of the VR system 200, the computer 500, and the server 600 may have at least a part of the functions of the other devices. In other words, the computer 300, the computer 500, the server 600, or other devices may have a part or all of the functional blocks of the computer 300, the computer 500, and the server 600 in this embodiment. In addition, each of the devices such as the computer 300, the computer 500, and the server 600 may not be realized by an integrated device, and may be realized by, for example, a plurality of devices connected via a network or the like.

[0117] In the present embodiment, the processor 301, the processor 501, or the processor 601 executes a program stored in the information processing system 100 to perform each process described later. However, at least a part of the process described later and performed by the processor 301 may be executed by a processor other than the processor 301. Also, at least a part of the process described later and performed by the processor 501 may be executed by a processor other than the processor 501. Also, at least a part of the process described later and performed by the processor 601 may be executed by a processor other than the processor 601. In other words, the computer that executes the program in the present embodiment may be any computer including the computer 300, the computer 500, and the server 600, or may be realized by a combination of multiple devices.

[0118] 9 is a block diagram showing a functional configuration of the information processing system 100. As shown in FIG. 9, the computer 300 (in other words, the VR system 200) functions as a control unit 310 and a storage unit 311 through cooperation between the processor 301, the memory 302, the storage 303, the input / output interface 304, and the communication interface 305. The computer 500 (in other words, the MR system 400) functions as a control unit 510 and a storage unit 511 through cooperation between the processor 501, the memory 502, the storage 503, the input / output interface 504, and the communication interface 505. The server 600 functions as a control unit 610 and a storage unit 611 through cooperation between the processor 601, the memory 602, the storage 603, the input / output interface 604, and the communication interface 605.

[0119] The control unit 510 of the MR system 400 includes a virtual space generation unit 810, a coordinate definition unit 812, an MR side input reception unit 816, a user information acquisition unit 818, an MR side object control unit 820, a display control unit 840, a sound control unit 845, and a communication control unit 850.

[0120] The virtual space generating unit 810 generates a virtual space (in other words, virtual space data representing a virtual space) based on a predetermined real space. In other words, the virtual space generating unit 810 generates a virtual space that imitates a predetermined real space. Here, the predetermined real space is a predetermined place that exists in the real world, and may be, for example, a specific room such as a conference room or a school classroom. The predetermined real space may be, for example, a school or a specific building such as a house. The predetermined real space may be, for example, a specific town. In other words, the predetermined real space does not have to be a space separated by walls or the like. Furthermore, simulating a predetermined real space means that the generated virtual space has a structure similar to the basic structure of the real space. Specifically, for example, a virtual space that imitates a real-world conference room can be one in which the shape of the virtual room as a virtual space (for example, the shape of the walls, floor, etc.) is approximately the same as the shape of the conference room in the real world, and virtual objects corresponding to real objects such as desks and chairs that exist in the conference room in the real world are arranged. In this case, virtual objects such as desks, chairs, walls, and floors do not need to be exact copies of the shapes of real objects, and may be simplified in shape, pattern, color, etc. Also, real objects such as posters on the wall or small items placed on a desk or floor may be objects that do not appear in the virtual world (real objects with no corresponding virtual objects).

[0121] In this embodiment, as described above, the predetermined real space is a specific conference room 21, and a virtual space 11 that imitates the specific conference room 21 is generated.

[0122] A known method can be used for generating a virtual space that imitates a real space, and is not particularly limited, but may be, for example, as follows. That is, the virtual space generation unit 810 generates the virtual space 11 based on information from a sensor that can detect the shape and position of real objects such as a wall, floor, desk, and chair that constitute the real space. Specifically, the virtual space generation unit 810 may acquire three-dimensional information of the real space based on information from an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor) provided in the detection device 460 or the MR device 410, and generate the virtual space. In addition, the virtual space generation unit 810 may use information from, for example, an acceleration sensor, an angular velocity sensor, or a geomagnetic sensor to generate the virtual space 11.

[0123] The virtual space generating unit 810 stores data indicating the generated virtual space 11 (hereinafter referred to as "virtual space data") in, for example, the storage unit 611 serving as a virtual space data storage unit.

[0124] The virtual space generation unit 810 may generate virtual space data indicating the virtual space 11 in advance before the VR user or MR user 6 starts using the service. Also, the virtual space generation unit 810 may generate virtual space data indicating the virtual space 11 in real time while the VR user or MR user 6 is using the service, for example.

[0125] The coordinate definition unit 812 defines the coordinates of the real space (in other words, the coordinates of the mixed reality space 21, hereinafter referred to as "mixed reality coordinates") and the coordinate definition unit 812 defines the coordinates of the virtual space 11 (hereinafter referred to as "virtual space coordinates").

[0126] The mixed reality coordinates have three mutually orthogonal axes, for example, an x-axis, a y-axis, and a z-axis. The virtual space coordinates have three mutually orthogonal axes, for example, an x-axis, a y-axis, and a z-axis. The coordinate definition unit 812 defines (in other words, associates) each coordinate for each point of the virtual space 11 (in other words, a virtual conference room) corresponding to (in other words, indicating the same point) each point of the real space (in other words, the real conference room 21) so that the mixed reality coordinates indicating each point of the real space and the virtual space coordinates indicating each point of the virtual space correspond one-to-one. For example, when the virtual space generation unit 810 generates a virtual space that imitates the real space, the coordinate definition unit 812 defines the virtual space coordinates associated with the mixed reality coordinates. At this time, the coordinate definition unit 812 may associate the mixed reality coordinates with the virtual space coordinates based on, for example, a feature point of the conference room detected by the detection device 460 or the MR device 410. Furthermore, the coordinate definition unit 812 may associate the mixed reality coordinates with the virtual space coordinates, for example, based on the position of the detection device 460 or the positions of markers or the like previously installed in the conference room.

[0127] In addition, the coordinate definition unit 812 stores information regarding the correspondence between the mixed reality coordinates and the virtual space coordinates (in other words, the correspondence between each point in the real space and each point in the virtual space) in the storage unit 611, which serves as a positional relationship storage unit.

[0128] The method for defining the mixed reality coordinates and the virtual space coordinates may be any known method, and is not particularly limited.

[0129] The timing for defining the mixed reality coordinates and the timing for defining the virtual space coordinates (in other words, the timing for associating the mixed reality coordinates with the virtual space coordinates) may be, for example, when the virtual space generation unit 810 generates a virtual space that mimics the real space, or the timing when the MR device 410 starts displaying a virtual object.

[0130] In addition, the generation of the virtual space and the definition of each coordinate may be performed in advance, for example, by a dedicated device such as external device 700 capable of scanning the real space with high accuracy, and the generated data may be stored in the memory unit 611.

[0131] The MR side input receiving unit 816 receives input from the MR user 6. In other words, the MR side input receiving unit 816 acquires input information from the MR user 6. Specifically, the MR side input receiving unit 816 receives outputs from the detection device 460, the controller 480, the camera 413, a device worn by the MR user, the gaze sensor 412, the sensor 417, the sensor 486, and image sensors provided in the detection device 460 and the MR device 410, as input from the MR user 6. In other words, the MR side input receiving unit 816 receives outputs from detection means (e.g., the detection device 460, the controller 480, the camera 413, the gaze sensor 412, the sensor 417, the sensor 486, and image sensors provided in the detection device 460 and the MR device 410) that detect the movement of the MR user 6, as input from the MR user 6. As will be described later, these devices and sensors are intended to obtain information regarding the movements of the MR user 6, which is used to control virtual objects, etc. in the virtual space 11 and the mixed reality space 21, and the MR user 6 can input information to the MR system 400 to move the virtual objects, etc., through his or her own movements.

[0132] Specifically, the MR side input receiving unit 816 may receive, for example, an output from the camera or the camera 413 as the detection device 460 as an input from the MR user 6. More specifically, the MR side input receiving unit 816 may receive, for example, position information indicating the position of the MR user 6 in the mixed reality space 21, detected by image recognition from an image captured by an image sensor provided in each of these cameras, as an input from the MR user 6. In other words, the MR side input receiving unit 816 may receive, for example, an output from a position detection means for detecting the position of the MR user 6 in the mixed reality space 21 as an input from the MR user 6. Note that the position detection means may detect the position of the MR user 6 based on, for example, positioning based on wireless communication (for example, Wi-Fi communication, Bluetooth communication, or UWB communication) between the MR device 410 and a specific device (for example, the detection device 460 or a beacon (not shown)) of the MR system 400, or positioning using a GPS (Global Positioning System) sensor provided in the MR device 410, and acquire the position information. The position information indicating the position of the MR user 6 can also be said to be information regarding the movement of the MR user 6.

[0133] Furthermore, the MR side input receiving unit 816 may receive, for example, information regarding the movement of the MR user 6 detected by image recognition from an image captured by a camera serving as the detection device 460 or an image sensor provided in the camera 413, as an input from the MR user 6. For example, the MR side input receiving unit 816 may receive, for example, information regarding the hand movement of the MR user 6 detected by image recognition, or information regarding the inclination or facing direction of the MR user 6, as an input from the MR user 6.

[0134] In addition, the MR side input receiving unit 816 may receive, as input from the MR user 6, information regarding the movements of the MR user 6 obtained using, for example, a device worn by the MR user 6 (for example, a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device) that can track the movements of the MR user 6 (and a device that tracks the movements of the MR user 6 in cooperation with the device via short-range wireless communication, etc.).

[0135] Furthermore, the MR-side input receiving unit 816 may receive, as an input from the MR user 6, information indicating an operation of the MR user 6 detected by the controller 480 (in other words, information regarding the movement of the MR user 6), for example.

[0136] In addition, the MR side input receiving unit 816 may receive output data regarding the inclination or direction of the MR device 410 (in other words, information regarding the movement of the MR user 6) from, for example, a sensor (e.g., an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) provided in the detection device 460 or the MR device 410 as input from the MR user 6.

[0137] The user information acquisition unit 818 acquires information about the MR user 6 present in the mixed reality space 21. Specifically, for example, the user information acquisition unit 818 acquires identification information that enables each MR user 6A, 6B, and 6C in the mixed reality space 21 to be identified (in other words, identification information that enables each MR device 410 to be identified). For example, the user information acquisition unit 818 may communicate with the MR device 410 worn by each user and acquire information about the user using the MR device 410 from the MR device 410 (for example, information about an account logged in to the MR device 410, etc.) as the identification information. Also, for example, the user information acquisition unit 818 may communicate with the MR device 410 worn by each user and acquire information unique to each MR device 410 that enables each MR device 410 to be identified from the MR device 410 as the identification information. Also, for example, the user information acquisition unit 818 may acquire the identification information by identifying the MR user 6 in the mixed reality space 21 by image recognition from an image captured by an image sensor included in the detection device 460 or the MR device 410. Furthermore, the user information acquisition unit 818 associates the information acquired by the MR side input acceptance unit 816 with the identification information. In other words, the user information acquisition unit 818 controls so that it is possible to know which MR user 6 the information is about when information about the movement of the MR user 6 is used in controlling a virtual object, which will be described later.

[0138] The MR side object control unit 820 controls virtual objects in the mixed reality space 21 .

[0139] The MR side object control unit 820 places virtual objects in the mixed reality space 21. Specifically, the MR side object control unit 820 places, for example, the VR avatar 25 of the VR user, an object to which a predetermined change is given by at least one of the VR user and the MR user 6, and the like. In this embodiment, the MR side object control unit 820 places, in the mixed reality space 21, the VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as the "virtual model 30") that can be moved by the VR user and the MR user 6, as shown in FIG. 8.

[0140] For example, the MR-side object control unit 820 places a virtual object corresponding to the virtual object at a position in the mixed reality space 21 corresponding to the position where the virtual object is placed in the virtual space 11. Here, the virtual object corresponding to the virtual object, i.e., the virtual object in the mixed reality space 21 corresponding to the virtual object in the virtual space 11, may have the same form as the virtual object in the virtual space 11 (in other words, it indicates the same object), or may have a different form. In this embodiment, a virtual model 30 having the same form as the virtual model 30 in the virtual space 11 is placed in the mixed reality space 21. Also, a VR avatar 25 having the same form as the VR avatar 25 in the virtual space 11 is placed in the mixed reality space 21.

[0141] The placement of the VR avatar 25 in the mixed reality space 21 can also be described as follows. That is, it can also be said that the MR-side object control unit 820 places the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11. Here, the position of the VR user in the virtual space 11 is, for example, the position of the VR avatar 25 in the virtual space 11. Note that in content in which the VR user enters the virtual space 11 from a first-person perspective, there may be cases in which the VR user (the VR avatar 25) cannot be visually recognized (specifically, only a part of the user's body such as a hand can be visually recognized, or even a part of the user cannot be visually recognized), but even in such cases, the VR user in the virtual space 11 recognized by the computer 300 can be said to be the VR avatar 25, and the position of the VR user in the virtual space 11 recognized by the computer 300 can be said to be the position of the VR avatar 25 in the virtual space.

[0142] The appearance (in other words, the form) of the VR avatar 25 may imitate the appearance of the VR user, but may not necessarily imitate it. For example, the appearance of the VR avatar 25 may imitate the appearance of a specific animal. The VR avatar 25 placed in the virtual space 11 and the VR avatar 25 placed in the mixed reality space 21 may have different appearances. With this configuration, the VR user can display his / her own avatar in the mixed reality space 21 while hiding the form of his / her own avatar in the virtual space 11. The appearance of the VR avatar 25 placed in the virtual space 11 may be an appearance that is visually recognized by the VR user operating the VR avatar 25, or may be an appearance that is visually recognized by another VR user operating another VR avatar 25 that exists in the virtual space 11 in which the VR avatar 25 exists.

[0143] Furthermore, the MR side object control unit 820 may place a virtual object in the mixed reality space 21, for example, in the following manner.

[0144] That is, for example, when placing the VR avatar 25, the MR side object control unit 820 determines at what position in the mixed reality space 21 the VR avatar 25 is to be placed, based on position information indicating the position of the VR user (in other words, the VR avatar 25) in the virtual space 11. Specifically, for example, when the VR side object control unit 920 described later places the VR avatar 25 in the virtual space 11, the VR side object control unit 920 transmits information on the position of the VR avatar 25 in the virtual space 11 (for example, the coordinate value of the virtual space coordinates) to the MR side object control unit 820 via the server 600. Then, based on the information, the MR side object control unit 820 places the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11 (for example, the position of the coordinate value of the mixed reality coordinates corresponding to the coordinate value of the virtual space coordinates). As a result, the VR avatar 25 is placed at the same position in the conference room in the virtual space 11 and the mixed reality space 21.

[0145] In other words, when placing a virtual object (e.g., a VR avatar 25 or a virtual model 30, etc.) in the mixed reality space 21, the MR side object control unit 820 may determine where in the mixed reality space 21 to place the virtual object based on position information (e.g., position information transmitted by the VR side object control unit 920) indicating the position of the virtual object (e.g., a VR avatar 25 or a virtual model 30, etc.) in the virtual space corresponding to the virtual object.

[0146] Also, for example, when placing a virtual object in the mixed reality space 21, the MR side object control unit 820 may determine a position in the mixed reality space 21 to place the virtual object based on information detected by a predetermined sensor of the MR system 400. Specifically, for example, a predetermined marker may be set in advance in the real space, and the MR side object control unit 820 may place the virtual model 30 as a virtual object at the position of the marker photographed by an image sensor provided in the detection device 460 or the MR device 410. Also, the MR side object control unit 820 may place the virtual model 30 as a virtual object at a position indicated by the MR user 6. The instruction may be made by a predetermined gesture or the like (for example, a gesture of pointing at a position where the virtual model 30 is to be placed), or may be a gesture to place the marker. Also, the MR side object control unit 820 may detect a flat surface in the mixed reality space 21 from an image photographed by an image sensor provided in the detection device 460 or the MR device 410, and place the virtual model 30 on the flat surface.

[0147] When arranging a virtual object in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the virtual object, and arranges the virtual object based on the object data. Here, the MR side object control unit 820 may receive the object data, for example, from the control unit 610 of the server 600, or from the VR side object control unit 920 of the VR system 200. Specifically, when arranging the VR avatar 25 in the mixed reality space 21, the MR side object control unit 820 may receive object data indicating the form of the VR avatar 25 from the control unit 610, and arrange the VR avatar 25 in the form indicated by the object data in the mixed reality space 21. When arranging the VR avatar 25 in the mixed reality space 21, the MR side object control unit 820 may receive object data indicating the form of the VR avatar 25 from the VR side object control unit 920, and arrange the VR avatar 25 in the form indicated by the object data in the mixed reality space 21.

[0148] The display control unit 840 controls image display on the display 411 of the MR device 410. The display control unit 840 generates an image for displaying a virtual object in the mixed reality space 21 arranged by the MR side object control unit 820 at a position arranged by the MR side object control unit 820. The display control unit 840 also causes the image to be displayed on the display 411. As a result, a view of the mixed reality space in which the virtual object is arranged at a desired position in the real space is provided to the MR user 6.

[0149] Furthermore, the display control unit 840 controls image display on the display 470. For example, the display control unit 840 causes the display 470 to display an image similar to the image displayed on the display 411. Note that the display control unit 840 may cause the display 470 to display an image showing a view of the mixed reality space provided by the MR device 410 to the MR user 6.

[0150] When the microphone 415 of the MR device 410 detects the speech of the MR user 6, the sound control unit 845 acquires sound data corresponding to the speech. The sound control unit 845 also transmits the acquired sound data to the computer 300 or the like via the network 2. When the sound control unit 845 receives sound data from the computer 300 via the network 2, the sound control unit 845 outputs a sound (speech) corresponding to the sound data from the speaker 416. This allows the MR user 6 to communicate with the VR user, for example, by phone call. The MR side input receiving unit 816 may receive sound data related to the speech of the MR user 6 detected by the microphone 415 as an input from the MR user 6.

[0151] The communication control unit 850 can communicate with the server 600, the computer 300, and other information communication devices via the network 2. The communication control unit 850 transmits information used by the server 600 or the computer 300 to the server 600 or the computer 300, for example. The communication control unit 850 also receives information used by the computer 500 from the server 600 or the computer 300, for example.

[0152] The control unit 310 of the VR system 200 includes a VR side input receiving unit 916, a VR side object control unit 920, a virtual camera control unit 930, a display control unit 940, a sound control unit 945, and a communication control unit 950.

[0153] The VR-side input receiving unit 916 receives input from the VR user. In other words, the VR-side input receiving unit 916 acquires input information from the VR user. Specifically, the VR-side input receiving unit 916 receives outputs from the detection device 260, the controller 280, the first camera 213, the second camera 214, a wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and image sensors provided in the detection device 260 and the VR device 210, as input from the VR user. In other words, the VR-side input receiving unit 916 receives outputs from detection means (for example, the detection device 260, the controller 280, the first camera 213, the second camera 214, a wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and image sensors provided in the detection device 260 and the VR device 210) that detect the movement of the VR user, as input from the VR user. As will be described later, these devices and sensors are intended to obtain information relating to the movements of the VR user which is used to control virtual objects, etc. in the virtual space 11 and the mixed reality space 21, and the VR user can input information to the VR system 200 to move the virtual objects, etc., through his or her own movements.

[0154] Specifically, the VR-side input receiving unit 916 may receive, for example, an output from a camera serving as the detection device 260 or a camera provided in the VR device 210 that captures the surroundings of the VR device 210 as an input from the VR user. More specifically, the VR-side input receiving unit 916 may receive, as an input from the VR user, position information indicating the position of the VR user in real space, detected by image recognition from an image captured by an image sensor provided in each of these cameras. In other words, the VR-side input receiving unit 916 may receive, as an input from the VR user, an output from a position detection means that detects the position of the VR user in real space. Note that the position detection means may detect the position of the VR user based on, for example, wireless communication (for example, Wi-Fi communication, Bluetooth communication, or UWB communication) between the VR device 210 and a specific device (for example, the detection device 260 or a beacon (not shown)) of the VR system 200, or based on positioning using a GPS (Global Positioning System) sensor provided in the VR device 210, and acquire the position information. In addition, the position information indicating the position of the VR user can also be considered information regarding the movement of the VR user.

[0155] Furthermore, the VR-side input receiving unit 916 may receive, as an input from the VR user, information on the movement of the VR user detected by image recognition from an image captured by an image sensor included in a camera serving as the detection device 260 or a camera capturing an image of the periphery of the VR device 210 included in the VR device 210. For example, the VR-side input receiving unit 916 may receive, as an input from the VR user, information on the movement of the VR user's hands detected by image recognition, or information on the inclination or facing direction of the VR user.

[0156] In addition, the VR side input receiving unit 916 may receive, as input from a VR user, information regarding the user's movements obtained using, for example, an equipment capable of tracking the movements of a VR user, such as a device worn by the VR user (for example, a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device, etc.) (and an equipment that tracks the movements of the MR user 6 in cooperation with the equipment via short-range wireless communication, etc.).

[0157] Furthermore, the VR-side input receiving unit 916 may receive, for example, information indicating an operation of the VR user detected by the controller 280 (in other words, information regarding the movement of the VR user) as input from the VR user.

[0158] In addition, the VR side input receiving unit 916 may receive output data regarding the inclination or direction of the VR device 210 (in other words, information regarding the movement of the VR user) from, for example, a sensor (e.g., an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) provided in the detection device 260 or the VR device 210 as input from the VR user.

[0159] The control unit 310 controls the virtual space 11. The control unit 310 acquires virtual space data indicating the virtual space 11 stored in the storage unit 611, and defines the virtual space 11 to be provided to the VR user based on the virtual space data.

[0160] The VR side object control unit 920 controls the virtual objects in the virtual space 11.

[0161] The VR-side object control unit 920 places a virtual object in the virtual space 11 indicated by the virtual space data. The object data indicating the virtual object is stored, for example, in the storage unit 611 or the storage unit 311 as an object data storage unit. Then, the VR-side object control unit 920 places the virtual object in the virtual space 11 using the object data.

[0162] Specifically, the VR-side object control unit 920 places, for example, MR avatars 26A, 26B, 26C of the MR users 6A, 6B, 6C, and virtual objects to which predetermined changes are given by at least one of the VR users and the MR user 6, in a virtual conference room. In this embodiment, the VR-side object control unit 920 places, in the virtual space 11, the VR avatar 25, the MR avatar 26, and a mock-up virtual object 30 (hereinafter also referred to as a "virtual model 30") that is moved by the VR user and the MR user 6. Note that a virtual object 31 corresponding to a desk 8 as a real object, etc., is also placed in the virtual space 11.

[0163] For example, when a VR system 200 (in other words, the computer 300) used by a VR user is connected to a VR system 200 (in other words, the computer 300) used by another VR user via the network 2, the VR-side object control unit 920 may place a VR avatar 25 of the other VR user in the virtual space 11. The VR avatar 25 of the other VR user may be operated by the other VR user.

[0164] For example, the VR-side object control unit 920 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 corresponding to the position of the MR user 6 in the mixed reality space 21. That is, the VR-side object control unit 920 places the MR avatars 26A, 26B, and 26C so that the positions of the MR users 6A, 6B, and 6C in the real conference room 21 are the same as the positions of the MR avatars 26A, 26B, and 26C in the virtual conference room.

[0165] The appearance (in other words, the form) of the MR avatar 26 may imitate the appearance of the MR user 6, but may not necessarily imitate it. For example, the appearance of the MR avatar 26 may imitate the appearance of a specific animal. When arranging a virtual object in the mixed reality space 21, the VR side object control unit 920 receives object data indicating the form of the virtual object, and arranges the virtual object based on the object data. Here, the VR side object control unit 920 may receive the object data from the control unit 610 of the server 600, or may receive the object data from the MR side object control unit 820 of the MR system 400. Specifically, for example, when arranging the MR avatar 26 in the virtual space 11, the VR side object control unit 920 may receive object data indicating the form of the MR avatar 26 from the control unit 610 or the MR side object control unit 820, and arrange the MR avatar 26 in the form indicated by the object data in the virtual space 11. The appearance of the MR avatar 26 can be set in advance by the MR user 6, and when the MR avatar 26 is displayed on the VR device 210, it may be displayed with the appearance set by the MR user 6. In addition, when the appearance of the MR avatar 26 imitates the appearance of the MR user 6, the VR side object control unit 920 may place the MR avatar 26 in the virtual space 11 (for example, generated by the control unit 510) based on an image of the MR user 6 acquired by a predetermined camera such as the camera 413 or the camera serving as the detection device 460.

[0166] The virtual camera control unit 930 places the virtual camera 14 in the virtual space 11. The virtual camera control unit 930 also controls the position of the virtual camera 14 in the virtual space 11 and the inclination (direction) of the virtual camera 14. In this embodiment, the virtual camera control unit 930 places the virtual camera 14 at the eye position of the VR avatar 25 in the virtual space 11. In other words, the virtual camera control unit 930 links the position of the virtual camera 14 with the position of the VR avatar 25, and moves the virtual camera 14 in the virtual space 11 when the VR avatar 25 moves in the virtual space 11. The position of the virtual camera 14 does not have to be linked with the position of the VR avatar 25. For example, it may be possible to move the VR avatar 25 in the virtual space 11 while keeping the virtual camera 14 fixed at a predetermined position.

[0167] The display control unit 940 controls image display on the display 211 of the VR device 210. The display control unit 940 defines a field of view 15 according to the position and inclination of the virtual camera 14 (in other words, the inclination of the head of the VR user wearing the VR device 210). In addition, the display control unit 940 generates a field of view image 17 to be displayed on the display 211 based on the defined field of view area 15. The field of view image 17 generated by the display control unit 940 is output to the VR device 210.

[0168] Furthermore, the display control unit 940 controls image display on the display 270. The display control unit 940 causes the display 270 to display an image similar to the image displayed on the display 211, for example.

[0169] When the sound control unit 945 detects an utterance by the VR user using the microphone 215 from the VR device 210, it acquires sound data corresponding to the utterance. In addition, the sound control unit 945 transmits the acquired sound data to the computer 500 or the computer 300 of another VR user via the network 2. In addition, when the sound control unit 945 receives sound data from the computer 500 or the computer 300 of another user via the network 2, it outputs a sound (utterance) corresponding to the sound data from the speaker 216. This allows the VR user to communicate with the MR user 6, for example, by phone. In addition, the VR side input receiving unit 916 may receive sound data related to the VR user's utterance detected by the microphone 215 as an input from the VR user.

[0170] The communication control unit 950 may communicate with the server 600, the computer 500, the computer 300 of another VR user, and other information communication devices via the network 2. The communication control unit 950 transmits, for example, information used by the server 600, the computer 500, or the computer 300 of another VR user to the server 600, the computer 500, or the computer 300 of another VR user. The communication control unit 850 also receives, for example, information used by the computer 300 from the server 600, the computer 500, or the computer 300 of another VR user.

[0171] (Integration of mixed reality and virtual space) In this embodiment, based on an input from the MR user 6 acquired by the MR side input receiving unit 816, a virtual object displayed by the MR device 410 and a virtual object displayed by the VR device 210 are controlled.

[0172] Specifically, the MR-side object control unit 820 applies a predetermined change to a virtual object in the mixed reality space 21, specifically, the virtual model 30, based on an input from the MR user 6 received by the MR-side input receiving unit 816. Here, the predetermined change may be a change to move a virtual object in the mixed reality space 21, such as moving the virtual object, changing the inclination or direction of the virtual object, or changing the relative positional relationship between parts of the virtual object. In addition, the predetermined change may be a change to the color of the virtual object, a change to information displayed by the virtual object, or a change to the form of the virtual object.

[0173] More specifically, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on, for example, information on the movement of the MR user 6 received by the MR-side input receiving unit 816. Specifically, for example, when a gesture of the MR user 6 carrying the virtual model 30 is detected by a detection means (for example, the detection device 460 or the camera 413, etc.) and the detection means outputs information related to the gesture, the MR-side input receiving unit 816 receives the output as an input from the MR user 6. Also, based on the input received by the MR-side input receiving unit 816, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 according to an instruction from the MR user 6 by the gesture. Note that the gesture of the MR user 6 that causes a predetermined change to the virtual model 30 as a virtual object does not have to be one that touches the virtual model 30 in the mixed reality space 21. For example, it may be possible to rotate the virtual object by holding a hand toward the virtual model 30 and waving the hand sideways.

[0174] The MR side object control unit 820 may also make a predetermined change to the virtual object based on, for example, information related to the operation of the MR user 6 on the controller 480 (in other words, information related to the movement of the MR user 6) received by the MR side input receiving unit 816. The MR side object control unit 820 may also make a predetermined change to the virtual object based on, for example, sound data related to the speech of the MR user 6 received by the MR side input receiving unit 816.

[0175] In addition, the VR-side object control unit 920 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 received by the MR-side input receiving unit 816.

[0176] Specifically, the VR-side object control unit 920 applies a change to the virtual model 30 in the virtual space 11 similar to the change applied to the virtual model 30 in the mixed reality space 21 based on, for example, an input from the MR user 6 received by the MR-side input receiving unit 816. That is, for example, as described above, when the virtual model 30 in the mixed reality space 21 moves due to a gesture of the MR user 6 to carry the virtual model 30, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 in the same manner.

[0177] The method of imparting a change to the virtual model 30 in the virtual space similar to the change imparted to the virtual model 30 in the mixed reality space 21 is not particularly limited, but may be, for example, as follows. That is, for example, when the MR-side object control unit 820 moves the virtual model 30 (in other words, the virtual object) in the mixed reality space 21 based on an input from the MR user 6 (in other words, at a predetermined opportunity), the MR-side object control unit 820 may send information indicating the position and tilt of the virtual model 30 after the movement (in other words, information regarding the state of the virtual model 30) to the VR-side object control unit 920. Then, the VR-side object control unit 920 may place the virtual model 30 at the position indicated by the information in the virtual space 11, or place the virtual model 30 at the tilt indicated by the information. Also, for example, the communication control unit 850 may send information based on the input from the MR user 6 (for example, information on the movement amount, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 510 based on the input) to the communication control unit 950, and the VR-side object control unit 920 may move a virtual object such as the virtual model 30 based on the information. That is, in this embodiment, the VR-side object control unit 920 gives a predetermined change to a virtual object in the virtual space 11 based on the input from the MR user 6 received by the MR-side input receiving unit 816, but "based on the input from the MR user 6" is not limited to the VR-side object control unit 920 itself receiving the input and controlling the virtual object, and may be any virtual object in the virtual space 11 that consequently undergoes a predetermined change in response to the input.

[0178] Further, the VR side object control unit 920 moves the MR avatar 26 in the virtual space 11 based on, for example, an input from the MR user 6 received by the MR side input receiving unit 816. Specifically, the VR side object control unit 920 moves the MR avatar 26 based on, for example, information on the movement of the MR user 6 received by the MR side input receiving unit 816. For example, the VR side object control unit 920 moves the MR avatar 26 based on position information indicating the position of the MR user 6 in the mixed reality space 21 received by the MR side input receiving unit 816 as an input from the MR user 6. That is, the VR side object control unit 920 places the MR avatar 26 at a position in the virtual space 11 corresponding to the position of the MR user 6 in the mixed reality space 21 based on the position information, and when the MR user 6 moves in the mixed reality space 21, places the MR avatar 26 at a position in the virtual space 11 corresponding to the position after the movement.

[0179] Further, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information on the movement of each part of the body of the MR user 6 (for example, hand movement, head movement, eye movement, change in facial expression, etc.) and information on the inclination and facing direction of the MR user 6, which are received by the MR-side input receiving unit 816 as input from the MR user 6. That is, the VR-side object control unit 920 reflects the movement of each part of the body of the MR user 6 (in other words, posture) and the inclination and facing direction of the MR user 6 in the MR avatar 26 in the virtual space 11 based on each of these pieces of information. That is, in this embodiment, the VR-side object control unit 920 controls the movement of the MR avatar 26 so that the movement of the MR avatar 26 in the virtual space 11 imitates the movement of the MR user 6 detected by a detection means for detecting the movement of the MR user 6. In other words, the VR side object control unit 920 controls the movement of the MR avatar 26, for at least parts of the body (e.g., hands, feet, head, eyes, mouth, etc.), so that the movement of the MR avatar 26 is linked to the movement of the MR user 6.

[0180] At least a part of the movement of the MR avatar 26 may be realized by the MR user 6's operation of the controller 480 (e.g., operation of an analog stick, a button, or the like). That is, for example, the VR-side object control unit 920 may move a part of the body of the MR avatar 26 based on information indicating the operation of the MR user 6 on the controller 480 as an input from the MR user 6.

[0181] At least one of the position and the movement of the MR avatar 26 in the virtual space 11 may not be linked to the position or the movement of the MR user 6 in the mixed reality space 21. That is, for example, even if the MR user 6 moves in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not move in response to this. Also, for example, even if the MR user 6 moves his / her hand in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not move in the same manner.

[0182] Furthermore, in this embodiment, based on an input from the VR user acquired by the VR-side input receiving unit 916, a virtual object displayed by the VR device 210 and a virtual object displayed by the MR device 410 are controlled.

[0183] Specifically, the VR-side object control unit 920 applies a predetermined change to a virtual object in the virtual space 11, specifically the virtual model 30, based on an input from the VR user received by the VR-side input receiving unit 916.

[0184] More specifically, the VR-side object control unit 920 applies a predetermined change to the virtual model 30 in the virtual space 11 based on, for example, information indicating the VR user's operation on the controller 280 (in other words, information regarding the movement of the VR user) received by the VR-side input receiving unit 916. Specifically, for example, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 according to an instruction from the VR user through an operation on the controller 280.

[0185] The VR-side object control unit 920 may also give a predetermined change to the virtual object based on, for example, a gesture of the VR user detected by a detection means. The VR-side object control unit 920 may also give a predetermined change to the virtual object based on, for example, sound data related to the speech of the VR user received by the VR-side input receiving unit 916.

[0186] Furthermore, the MR side object control unit 820 applies a predetermined change to a virtual object in the mixed reality space 21 based on an input from the VR user received by the VR side input receiving unit 916.

[0187] Specifically, the MR-side object control unit 820 applies a change to the virtual model 30 in the mixed reality space 21 similar to the change applied to the virtual model 30 in the virtual space 11 based on, for example, an input from the VR user received by the VR-side input receiving unit 916. That is, for example, as described above, when the virtual model 30 in the virtual space 11 moves due to an operation of the VR user on the controller 280, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 in the same manner.

[0188] The method of imparting a change to the virtual model 30 in the mixed reality space 21 similar to the change imparted to the virtual model 30 in the virtual space 11 is not particularly limited, but may be, for example, as follows. That is, for example, when the VR side object control unit 920 moves the virtual model 30 (in other words, the virtual object) in the virtual space 11 based on an input from the VR user (in other words, at a predetermined opportunity), the VR side object control unit 920 may send information indicating the position and inclination of the virtual model 30 after the movement (in other words, information regarding the state of the virtual model 30) to the MR side object control unit 820. Then, the MR side object control unit 820 may place the virtual model 30 at the position indicated by the information in the mixed reality space 21, or place the virtual model 30 at the inclination indicated by the information. Also, for example, the communication control unit 950 may send information based on an input from a VR user (for example, information on the movement amount, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 310 based on the input) to the communication control unit 850, and the MR side object control unit 820 may move a virtual object such as the virtual model 30 based on the information. That is, in this embodiment, the MR side object control unit 820 gives a predetermined change to a virtual object in the mixed reality space 21 based on an input from a VR user received by the VR side input receiving unit 916, but "based on an input from a VR user" does not necessarily mean that the MR side object control unit 820 itself receives the input and controls the virtual object, but may mean that the virtual object in the mixed reality space 21 ultimately undergoes a predetermined change in response to the input.

[0189] Furthermore, the VR-side object control unit 920 applies a predetermined change to the VR avatar 25 in the virtual space 11 based on an input from the VR user received by the VR-side input receiving unit 916. Specifically, the VR-side object control unit 920 applies a predetermined change to the VR avatar 25 in the virtual space 11 based on, for example, information indicating an operation of the VR user on the controller 280 (in other words, information regarding the movement of the VR user) received by the VR-side input receiving unit 916. Specifically, for example, the VR-side object control unit 920 moves the VR avatar 25 in the virtual space 11 according to an instruction from the VR user by operating the controller 280.

[0190] Further, the VR-side object control unit 920 moves the VR avatar 25 based on, for example, information on the movement of each part of the VR user's body (for example, hand movement, head movement, eye movement, change in facial expression, etc.) and information on the inclination and facing direction of the VR user, which the VR-side input receiving unit 916 receives as an input from the VR user. That is, the VR-side object control unit 920 reflects the movement of each part of the VR user's body (in other words, posture) and the inclination and facing direction of the VR user on the VR avatar 25 in the virtual space 11 based on each of these pieces of information. That is, in this embodiment, the VR-side object control unit 920 controls the movement of the VR avatar 25 so that the movement of the VR avatar 25 in the virtual space 11 imitates the movement of the VR user detected by a detection means for detecting the movement of the VR user. In other words, the VR side object control unit 920 controls the movement of the VR avatar 25 for at least parts of the body (e.g., hands, feet, head, eyes, mouth, etc.) so that the movement of the VR avatar 25 is linked to the movement of the VR user.

[0191] Further, the MR-side object control unit 820 gives, for example, a change to the VR avatar 25 in the mixed reality space 21 similar to the change given to the VR avatar 25 in the virtual space 11 based on an input from the VR user received by the VR-side input receiving unit 916. That is, for example, as described above, when the VR avatar 25 in the virtual space 11 moves by the VR user's operation on the controller 280, the MR-side object control unit 820 moves the VR avatar 25 in the mixed reality space 21 in the same manner. Note that the method of giving, for example, a change to the VR avatar 25 in the mixed reality space 21 similar to the change given to the VR avatar 25 in the virtual space 11 is not particularly limited, but may be the same as the case of the virtual model 30 described above.

[0192] Note that the position of the VR avatar 25 in the virtual space 11 and the position in the mixed reality space 21 may move in accordance with the movement of the VR user in the real space. That is, for example, the VR-side object control unit 920 may move the VR avatar 25 based on position information indicating the position of the VR user in the real space as an input from the VR user.

[0193] Furthermore, the movement of a part of the VR avatar 25 in the virtual space 11 and the VR avatar 25 in the mixed reality space 21 may be realized by an operation on the controller 280. That is, for example, the VR side object control unit 920 and the MR side object control unit 820 may move a part of the body of the VR avatar 25 based on information indicating an operation of the VR user on the controller 280 as an input from the VR user.

[0194] In this embodiment, the results of actions on real objects that actually exist in the conference room 21 as a predetermined real space can also be shared between the VR user and the MR user 6. In this regard, the following will explain an example of control of a real object on which a predetermined marking can be made (for example, a blackboard or a whiteboard). As shown in FIG. 8, a blackboard 35 as a real object is placed in the mixed reality space 21, and a virtual blackboard 36 as a virtual object corresponding to the blackboard 35 is placed in the virtual space 11 at a position corresponding to the blackboard 35.

[0195] In this embodiment, an action (in other words, a result of an action) performed by the MR user 6 on a blackboard 35 serving as a real object in the mixed reality space 21 is reflected on the blackboard 35 in the mixed reality space 21. Also, in this embodiment, the action is reflected on the virtual blackboard 36 in the virtual space 11.

[0196] In the mixed reality space 21, the MR user 6 can make a predetermined entry on the blackboard 35. The entry action on the blackboard 35 by the MR user 6 is detected by a detection means (hereinafter, referred to as "MR-side entry detection means") that detects the entry action, and the output from the MR-side entry detection means (in other words, information on the entry action) is received by the MR-side input receiving unit 816 as an input from the MR user 6. The MR-side entry detection means may be, for example, a detection means capable of detecting the movement of the hand of the MR user 6. Specifically, the MR-side entry detection means may be, for example, a pen-type controller as the controller 480. The MR-side entry detection means may be, for example, a camera as the detection device 460, the camera 413, or the like. The MR-side entry detection means may be, for example, a device worn by the MR user 6. The MR-side entry detection means may be a sensor provided on the blackboard 35 (for example, a touch sensor, a sensor that reads reflected infrared light from a device that emits infrared light provided on the blackboard 35, or the like). In other words, the blackboard 35 is capable of communicating with the computer 500, and information regarding the writing action of the MR user 6 detected by a sensor provided on the blackboard 35 may be received by the MR side input receiving unit 816 as input from the MR user 6.

[0197] The writing action of the MR user 6 is reflected on the blackboard 35 in the mixed reality space 21. In other words, when the MR user 6 performs a writing action, the blackboard 35 changes to a state in which the writing contents written by the MR user 6 (for example, a picture of a moon, a picture of a heart, and the word "Sample" in FIG. 10(a)) are displayed, as shown in FIG. 10(a). For example, the blackboard 35 may be an electronic blackboard that electronically displays (in other words, reflects) the writing contents written by the MR user 6, and when the MR user 6 performs a writing action, the blackboard 35 may change to a state in which the writing contents are displayed. Specifically, for example, the writing surface of the blackboard 35 itself may be a display, and the display may display the writing contents. Also, the blackboard 35 may include a projector that projects an image onto the writing surface, and the projector may project the writing contents onto the writing surface. In other words, the reflection of the user's behavior in the mixed reality space 21 (in other words, a change in the mixed reality space 21 seen through the MR device 410 due to the behavior of the MR user 6) may be realized by some change (for example, a change in the display content) occurring in the real object itself. When the written content is electronically displayed, the computer of the blackboard 35 may control the display based on the detection result of a sensor equipped in the blackboard, or the computer 500 may control the display based on the detection result of the MR side writing detection means. That is, for example, the control unit 510 of the computer 500 or the like may function as a control means (hereinafter referred to as "blackboard display control means") that controls the display of the blackboard 35 based on the input from the MR user 6 received by the MR side input receiving unit 816. Specifically, the blackboard display control means may control the image displayed by the display or projector of the blackboard 35 to be an image showing the written content based on the input from the MR user 6. The writing action of the MR user 6 on the blackboard 35 in the mixed reality space 21 may be reflected by the MR user 6 actually writing characters or pictures on the blackboard 35 with a writing implement such as chalk. In other words, the action of actually writing characters or pictures on the blackboard 35 with a writing implement may be detected as the writing action of the MR user 6 by the MR-side writing detection means.

[0198] Also, for example, the MR side object control unit 820 may generate a virtual object (hereinafter, referred to as "entry object 37") indicating the contents entered by the MR user 6, and place it at the position of the blackboard 35 in the mixed reality space 21 (see FIG. 10(a)). Specifically, the MR side object control unit 820 may generate the entry object 37 based on the input from the MR user 6 accepted by the MR side input acceptance unit 816. Then, when the blackboard 35 is viewed through the MR device 410, the state in which the entry contents (in other words, the entry object 37) are displayed on the blackboard 35 may be visually recognized. In other words, the reflection of the user's actions in the mixed reality space 21 (in other words, the change in the mixed reality space 21 viewed through the MR device 410 due to the actions of the MR user 6) may be realized by adding a predetermined virtual object to the real object.

[0199] In addition, the VR-side object control unit 920 applies a predetermined change to the virtual object corresponding to the real object based on the input from the MR user 6 received by the MR-side input receiving unit 816. Specifically, as shown in FIG. 10(b), the VR-side object control unit 920 controls the virtual blackboard 36 to reflect the input from the MR user 6, and to display the contents written by the MR user 6. In other words, the VR-side object control unit 920 reflects a change related to a real object (e.g., the blackboard 35) in the mixed reality space 21 based on the action of the MR user 6 in a virtual object (e.g., the virtual blackboard 36) in the virtual space 11 corresponding to the real object. Specifically, for example, when information related to the writing action of the MR user 6 is input to the MR-side input receiving unit 816, the VR-side object control unit 920 may place a writing object 38 indicating the contents written by the MR user 6 at the position of the virtual blackboard 36 in the virtual space 11 (see FIG. 10(b)). Also, the VR-side object control unit 920 may control the virtual blackboard 36 to display the written contents by changing the form of the virtual blackboard 36 itself, for example. Also, the virtual blackboard 36 may be a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, etc., and the VR-side object control unit 920 may be configured to display the image on the virtual blackboard 36. Also, the MR system 400 may be provided with a camera that captures the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 may be a virtual object that displays an image captured by the camera, and the VR-side object control unit 920 may be configured to display the image on the virtual blackboard 36.

[0200] In addition, in this embodiment, a behavior (in other words, a result of the behavior) performed by the VR user on a virtual blackboard 36 serving as a virtual object in the virtual space 11 is reflected on the virtual blackboard 36 in the virtual space 11. In addition, in this embodiment, the behavior is reflected on the blackboard 35 in the mixed reality space 21.

[0201] In the virtual space 11, the VR user can make a predetermined entry on the virtual blackboard 36. Furthermore, the entry action on the virtual blackboard 36 by the VR user is detected by a detection means (hereinafter referred to as "VR-side entry detection means") that detects the entry action, and the output from the VR-side entry detection means is received by the VR-side input receiving unit 916 as an input from the VR user. The VR-side entry detection means may be, for example, a detection means capable of detecting the movement of the hand of the VR user. Specifically, the VR-side entry detection means may be, for example, a controller 280 or the like. Furthermore, the VR-side entry detection means may be, for example, a camera or the like as the detection device 260. Furthermore, the VR-side entry detection means may be, for example, a device worn by the VR user.

[0202] The writing action of the VR user is reflected on the blackboard 35 in the mixed reality space 21. In other words, when the VR user performs a writing action, the blackboard 35 changes to a state in which the writing content written by the VR user is displayed. In further other words, the input from the VR user accepted by the VR side input receiving unit 916 is reflected on the blackboard 35. The reflection of the input from the VR user on the blackboard 35 may be performed, for example, as follows. That is, for example, the MR side object control unit 820 may place a writing object 37 indicating the writing content written by the VR user at the position of the blackboard 35 in the mixed reality space 21. Also, the blackboard display control means may control the image displayed by the display or projector of the blackboard 35 based on the input from the VR user so that it becomes an image indicating the writing content written by the VR user. That is, the MR side object control unit 820 or the blackboard display control means may function as a reflection means for reflecting the input from the VR user accepted by the VR side input receiving unit 916 on a real object in the mixed reality space 21.

[0203] The writing action of the VR user is also reflected on the virtual blackboard 36 in the virtual space 11. In other words, when the VR user performs a writing action, the virtual blackboard 36 changes to a state in which the writing content written by the VR user is displayed. In further other words, the input from the VR user received by the VR-side input receiving unit 916 is reflected on the virtual blackboard 36. Specifically, the VR-side object control unit 920 gives a predetermined change to the virtual blackboard 36 based on the input from the VR user received by the VR-side input receiving unit 916. More specifically, the VR-side object control unit 920 reflects the input from the VR user on the virtual blackboard 36, and controls the virtual blackboard 36 to a state in which the writing content written by the VR user is displayed. Specifically, for example, when information on the writing action of the VR user is input to the VR-side input receiving unit 916, the VR-side object control unit 920 may place a writing object 38 indicating the writing content written by the VR user at the position of the virtual blackboard 36 in the virtual space 11. Also, the VR-side object control unit 920 may control the virtual blackboard 36 to display the written contents by changing the form of the virtual blackboard 36 itself, for example. Also, the virtual blackboard 36 may be a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, etc., and the VR-side object control unit 920 may display the image on the virtual blackboard 36. Also, the MR system 400 may be provided with a camera that captures the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 may be a virtual object that displays an image captured by the camera, and the VR-side object control unit 920 may display the image on the virtual blackboard 36. In other words, the input from the VR user received by the VR-side input receiving unit 916 may be reflected in a real object in the mixed reality space 21, and thereby the virtual blackboard 36 may change based on the input from the VR user.

[0204] As described above, in this embodiment, the action taken by the MR user 6 on a real object and the action taken by the VR user on a virtual object corresponding to the real object are reflected in the mixed reality space 21 and the virtual space 11. Such reflection of each action in the mixed reality space 21 or the virtual space 11 may be as follows.

[0205] That is, for example, when the MR user 6 points an arbitrary position in the mixed reality space 21 with a predetermined pointer such as a laser pointer, the pointed position (hereinafter referred to as the "pointed position") may be known in the virtual space 11. Specifically, for example, in the mixed reality space 21, the detection device 460 or the camera 413 of the MR system 400 may function as a pointed position detection means for detecting the pointed position. Then, information on the pointed position detected by the pointed position detection means may be received by the MR side input receiving unit 816 as an input from the MR user 6. In addition, the VR side object control unit 920 may place a virtual object indicating light from a laser pointer at a position in the virtual space 11 corresponding to the pointed position in the mixed reality space 21. In other words, the control unit 310 may control the display of the virtual space 11 in the VR device 210 so that the position in the virtual space 11 corresponding to the pointed position in the mixed reality space 21 is illuminated in a predetermined color representing light from a laser pointer.

[0206] Also, for example, when the VR user points an arbitrary position in the virtual space 11 with a predetermined pointer such as a virtual laser pointer, the pointed position may be known in the mixed reality space 21. Specifically, for example, when information on the operation of the VR user on the controller 280 for operating the virtual laser pointer (in other words, information on the movement of the VR user) is input to the VR side input receiving unit 916, the VR side object control unit 920 may place a virtual object indicating light from the virtual laser pointer at a position pointed by the operation of the VR user (in other words, a position pointed by the virtual laser pointer). In other words, the control unit 310 may control the display of the virtual space in the VR device 210 so that a position in the virtual space 11 pointed by the operation of the virtual VR user is illuminated in a predetermined color representing light from the virtual laser pointer. Also, the control unit 510 or the like may control so that a position in the mixed reality space 21 corresponding to the pointed position in the virtual space 11 (specifically, a pointed position by the VR user) is illuminated in a predetermined color representing light from the virtual laser pointer. Specifically, the MR-side object control unit 820 may place a virtual object indicating light from a virtual laser pointer at a position in the mixed reality space 21 corresponding to the indicated position in the virtual space 11. Also, for example, when the indicated position by the VR user is a predetermined position on the virtual blackboard 36, the blackboard display control means may control, based on an input from the VR user, an image displayed by the display or projector of the blackboard 35 to be an image in which a position corresponding to the predetermined position on the virtual blackboard 36 is illuminated (i.e., an image indicating that a predetermined place is being pointed to by a laser pointer). Also, for example, the detection device 460, the MR device 410, etc. may include a device for emitting laser light, and the control unit 510 may control the device to irradiate the laser light at a position in the mixed reality space 21 corresponding to the indicated position in the virtual space 11 in response to an input from the VR user received by the VR-side input receiving unit 916.

[0207] Furthermore, the behavior of the MR user 6 or the VR user may be reflected in the mixed reality space 21 and the virtual space 11 as follows. That is, for example, a device (e.g., an electronic piano) capable of inputting or outputting sound as a real object may be placed in the mixed reality space 21, and the behavior of the MR user 6 or the VR user related to the device may be reflected in the mixed reality space 21 and the virtual space 11. In the following, it is assumed that an electronic piano is placed as a real object in the mixed reality space 21, and a virtual electronic piano as a virtual object corresponding to the electronic piano is placed in the virtual space 11 at a position corresponding to the electronic piano.

[0208] The electronic piano may be capable of communicating with the computer 500, for example. When the MR user 6 operates the electronic piano in the mixed reality space 21, information on the operation (for example, sound data corresponding to the operation) may be sent to the computer 300 via the computer 500. The sound control unit 945 may output a sound from the speaker 216 based on the information. As a result, for example, when the MR user 6 operates a key of the electronic piano (for example, a key of the note "C"), a sound corresponding to the operated key (for example, the note "C") may be output from the speaker 216. In other words, the MR side input receiving unit 816 may receive an output (for example, sound data corresponding to the operation) from the electronic piano (in other words, a sensor or the like that detects an operation on the key) as a detection means that detects the operation of the MR user 6 as an input from the MR user 6, and the control unit 310 (for example, the sound control unit 945) may cause the speaker 216 to output a sound corresponding to the operation of the MR user 6 on the electronic piano based on the input. In addition, the sound control unit 945 may control the direction from which the sound is emitted so that the sound corresponding to an operation on the electronic piano sounds as if it is being output from the virtual electronic piano in the virtual space 11.

[0209] Also, when the VR user operates the virtual electronic piano in the virtual space 11, information on the operation may be sent to the computer 500. Then, the sound control unit 845 may output a sound from the speaker 416 based on the information. As a result, for example, when the VR user operates a key (for example, a key of the note "C") of the virtual electronic piano, a sound (for example, the note "C") corresponding to the operated key may be output from the speaker 416. In other words, the VR-side input receiving unit 916 may receive an output from a detection means (for example, the detection device 260 or the controller 280) that detects an operation of the VR user on the virtual electronic piano as an input from the VR user, and the control unit 510 (for example, the sound control unit 845) may output a sound corresponding to the operation of the VR user on the virtual electronic piano from the speaker 416 based on the input. Note that the control unit 510 may output a sound corresponding to the operation of the VR user on the virtual electronic piano from the speaker of the electronic piano.

[0210] When the MR user 6 operates the electronic piano, the sound corresponding to the operation may be output in the mixed reality space 21 by the speaker of the electronic piano or by the speaker 416. When the VR user operates the virtual electronic piano, the sound corresponding to the operation may be output in the virtual space 11 by the speaker 216.

[0211] <Processing performed by the information processing system> Next, the flow of processing executed by the information processing system 100 will be described with reference to FIGS.

[0212] An example of the processing executed by the VR system 200 will be described with reference to FIG.

[0213] In step S101 , the processor 301 of the computer 300 identifies virtual space data and defines the virtual space 11 .

[0214] In step S102, the processor 301 places the VR avatar 25 of the VR user in the virtual space 11.

[0215] In step S103, the processor 301 acquires position information indicating the position of the MR user 6 in the mixed reality space 21 from the processor 501 of the computer 500 which is connected to the computer 300 via the network 2 and which controls the mixed reality space 21 corresponding to the virtual space 11.

[0216] In step S104, the processor 301 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21, based on the position information acquired in step S103.

[0217] In step S105, the processor 301 places the virtual model 30 as a virtual object in the virtual space 11. The processor 301 places the virtual model 30 at a position in the virtual space 11 corresponding to the position of the virtual model 30 in the mixed reality space 21. Note that placing at a corresponding position may mean that the position in the mixed reality space 21 or the position in the virtual space 11 is determined first.

[0218] In step S106, the processor 301 accepts an input from the VR user. Specifically, the processor 301 accepts, for example, an output of a detection means for detecting the movement of the VR user as the input from the VR user.

[0219] In step S107, the processor 301 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the VR user. Specifically, the processor 301 moves the VR avatar 25 or the virtual model 30 in the virtual space 11 based on, for example, the input from the VR user. For example, if the input is related to the VR avatar 25, the processor 301 moves the VR avatar 25. Also, for example, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30. Also, the processor 301 transmits information related to the input from the VR user to the processor 501.

[0220] In step S108, the processor 301 acquires information regarding the input from the MR user 6 that the processor 501 has accepted.

[0221] In step S109, the processor 301 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 acquired in step S108. Specifically, the processor 301 moves the MR avatar 26 or the virtual model 30 in the virtual space 11 based on, for example, the input from the MR user 6. For example, if the input is related to the MR avatar 26, the processor 301 moves the MR avatar 26. Also, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30.

[0222] Next, an example of processing executed by the MR system 400 will be described with reference to FIG.

[0223] In step S201, the processor 501 of the computer 500 defines the mixed reality space 21.

[0224] In step S202, the processor 501 detects the position of the MR user 6 in the mixed reality space 21. The processor 501 also transmits position information indicating the detected position of the MR user 6 to the processor 301 of the computer 300 connected to the computer 500 via the network 2 and controlling the virtual space 11 corresponding to the mixed reality space 21.

[0225] In step S203, the processor 501 acquires, from the processor 301, position information indicating the position of the VR avatar 25 in the virtual space 11.

[0226] In step S204, the processor 501 places the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR avatar 25 in the virtual space 11, based on the position information acquired in step S203. Note that placing at a corresponding position may mean that the position in the virtual space 11 or the position in the mixed reality space 21 is determined first.

[0227] In step S205, the processor 501 places the virtual model 30 as a virtual object in the mixed reality space 21. The processor 501 places the virtual model 30 at a position in the mixed reality space 21 that corresponds to the position of the virtual model 30 in the virtual space 11.

[0228] In step S206, the processor 501 receives an input from the MR user 6. Specifically, the processor 501 receives, for example, an output of a detection means for detecting the movement of the MR user 6 as the input from the MR user 6.

[0229] In step S207, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the MR user 6. Specifically, the processor 501 moves the virtual model 30 in the mixed reality space 21 based on, for example, the input from the MR user 6. In addition, the processor 501 transmits information regarding the input from the MR user 6 to the processor 301.

[0230] In step S208, the processor 501 acquires information regarding the input from the VR user that the processor 301 has accepted.

[0231] In step S209, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user acquired in step S208. Specifically, the processor 501 moves the VR avatar 25 or the virtual model 30 in the mixed reality space 21 based on, for example, the input from the VR user. For example, if the input is related to the VR avatar 25, the processor 501 moves the VR avatar 25. Also, if the input is related to the virtual model 30, the processor 501 moves the virtual model 30.

[0232] Next, an example of a process for reflecting the behavior of the MR user 6 in the virtual space 11 will be described with reference to FIG.

[0233] In step S301, the processor 501 receives an input from the MR user 6. Specifically, the processor 501 receives, as an input from the MR user 6, for example, an output of a detection means that detects an action of the MR user 6 with respect to a specific real object (for example, the blackboard 35, a laser pointer, or an electronic piano).

[0234] In step S302, the processor 501 transmits information regarding the input from the MR user 6 received in step S301 to the processor 301 (in other words, the VR system 200).

[0235] In step S303, the processor 301, based on the information transmitted in step S302, applies a predetermined change to the virtual object corresponding to the specific real object in which the MR user 6 has performed an action. For example, when the MR user 6 performs a writing action to write predetermined contents on the blackboard 35, the processor 301 changes the virtual blackboard 36 corresponding to the blackboard 35 to a state in which the predetermined written contents are displayed.

[0236] Next, an example of a process for reflecting the action of the VR user in the mixed reality space 21 will be described with reference to FIG.

[0237] In step S351, the processor 301 accepts an input from a VR user. Specifically, the processor 301 accepts, as the input from the VR user, an output of a detection means that detects an action of the VR user with respect to a virtual object (e.g., the virtual blackboard 36 or a virtual electronic piano) corresponding to a specific real object, for example.

[0238] In step S352, the processor 301 transmits information regarding the input from the VR user received in step S351 to the processor 501 (in other words, the MR system 400).

[0239] In step S353, the processor 501 reflects the action performed by the VR user on the virtual object corresponding to a specific real object on the specific real object based on the information transmitted in step S352. For example, when the VR user performs an action of writing a predetermined content on the virtual blackboard 36, the processor 501 changes the blackboard 35 to a state in which the predetermined written content is displayed.

[0240] The present invention is not limited to the above-described embodiment, and can be modified in various ways without departing from the spirit of the present invention. Within the scope of the present invention, the components can be freely combined, any components can be modified, or any components can be omitted. The process flow described in this specification is merely an example, and the order and configuration of each process may be different.

[0241] <Additional Notes> The matters described in the above embodiment can also be described as follows.

[0242] (Appendix 1) A first display means (e.g., an MR device 410) for providing a first user with a first view, which is a view of a mixed reality space in which a virtual object is arranged in a predetermined real space; a second display means (e.g., a VR device 210) for providing a second user with a second view in which an avatar of the first user is placed in a virtual space simulating the predetermined real space; A first input receiving means (e.g., an MR side input receiving unit 816) for receiving an input from the first user; and a first control means (e.g., a control unit 310 or a VR side object control unit 920) that controls an avatar of the first user in the virtual space based on the input from the first user received by the first input receiving means. Information processing system. According to this configuration, the avatar of the first user is placed in a virtual space that imitates a predetermined real space in which the first user is located, and is controlled based on an input from the first user. Therefore, in controlling the avatar of the first user in the virtual space based on an input from the first user in the mixed reality space, the gap between the mixed reality space and the virtual space can be filled, and the connection between the two can be strengthened. Therefore, the range of uses of the virtual space is expanded.

[0243] (Appendix 2) the first input receiving means receives an output from a position detection means that detects a position of the first user in the mixed reality space as an input from the first user; The first control means places an avatar of the first user at a position in the virtual space corresponding to the position of the first user in the mixed reality space detected by the position detection means. 2. An information processing system according to claim 1. According to such a configuration, the avatar of the first user is placed at a position in the virtual space corresponding to the position of the first user in the mixed reality space. That is, the position of the first user in the mixed reality space and the position of the avatar of the first user in the virtual space are linked. Therefore, from the viewpoint of the second user, it is possible to know where the first user is in reality, what kind of place he / she is in, etc. Also, from the viewpoint of the first user, it becomes easy to see how the second user sees his / her avatar. Therefore, it is possible to fill the gap between the mixed reality space and the virtual space, and the range of uses of the virtual space is expanded.

[0244] (Appendix 3) the second view is a view of the virtual space from a virtual camera (e.g., virtual camera 14) located in the virtual space; A second input receiving means (e.g., a VR-side input receiving unit 916) for receiving an input from the second user; and a virtual camera control means (e.g., a virtual camera control unit 930) that moves the position of the virtual camera in the virtual space based on the input from the second user received by the second input receiving means. 2. An information processing system according to claim 1. According to this configuration, the second user can move the position of the virtual camera to move the viewpoint from which the virtual space is viewed. Therefore, the second user can view the virtual space that imitates a specific real space and the avatar of the first user that exists in the specific real space and exists in the virtual space from various viewpoints, which expands the range of uses of the virtual space.

[0245] (Appendix 4) A second input receiving means (e.g., a VR-side input receiving unit 916) for receiving an input from the second user; A reflection means (for example, an MR side object control unit 820, a blackboard display control unit, or a sound control unit 845), The first control means is A specific virtual object (e.g., a virtual blackboard 36 or a virtual electronic piano) corresponding to a specific object (e.g., a blackboard 35 or an electronic piano) actually existing in the predetermined real space is placed in the virtual space; imparting a predetermined change to the specific virtual object based on the input from the second user received by the second input receiving means; The reflecting means reflects the input from the second user received by the second input receiving means on the specific object in the mixed reality space. 2. An information processing system according to claim 1. According to this configuration, the operation of the second user performed on a virtual object in the virtual space can be reflected on the object that actually exists in the real space that the first user is in. Therefore, the real space and the virtual space are highly integrated, and the range of uses of the virtual space is expanded.

[0246] (Appendix 5) A second input receiving means (for example, a VR-side input receiving unit 916) is provided to receive an input from the second user, The first control means is Placing an avatar of the second user in the virtual space; Controlling an avatar of the second user in the virtual space based on the input from the second user received by the second input receiving means. An information processing system according to any one of appendix 1 to 4. According to such a configuration, the virtual space contains an avatar of the first user and an avatar of the second user, and each is controlled based on an input from the first user or the second user. Therefore, in the virtual space, communication can be performed using the avatar of the first user and the avatar of the second user, and the range of uses of the virtual space is expanded.

[0247] (Appendix 6) A second control means (e.g., a control unit 510 or an MR side object control unit 820) is provided for receiving information regarding a position of an avatar of the second user in the virtual space from the first control means and placing the avatar of the second user at a position in the mixed reality space corresponding to the position of the avatar of the second user in the virtual space. 6. An information processing system according to claim 5. According to such a configuration, in the mixed reality space, the avatar of the second user is placed at a position corresponding to the position where the second user placed the avatar of the second user in the virtual space. In addition, the avatar of the second user in the mixed reality space moves in conjunction with the movement of the avatar of the second user in the virtual space. This improves the quality of communication between the virtual space and the mixed reality space via the avatar. This expands the range of uses of the virtual space.

[0248] (Appendix 7) A first display means (e.g., an MR device 410) for providing a first user with a first view, which is a view of a mixed reality space in which a virtual object is arranged in a predetermined real space; a second display means (e.g., a VR device 210) for providing a second user with a second view in which an avatar of the first user is placed in a virtual space simulating the predetermined real space; A computer of an information processing system including a first input receiving means (e.g., an MR side input receiving unit 816) for receiving an input from the first user, The first input receiving means functions as a first control means (e.g., the control unit 310 or the VR side object control unit 920) that controls an avatar of the first user in the virtual space based on the input from the first user received by the first input receiving means. program. According to this configuration, the same effects as those of the program described in Supplementary Note 1 can be achieved. [Explanation of symbols]

[0249] 11 Virtual space, 21 Mixed reality space, 25 VR avatar, 26 MR avatar, 30 Virtual model, 35 Blackboard, 36 Virtual blackboard, 100 Information processing system, 200 VR system, 210 VR device, 211 Display, 212 Gaze sensor, 213 First camera, 214 Second camera, 215 Microphone, 216 Speaker, 217 Sensor, 260 Detection device, 270 Display, 280 Controller, 286 Sensor, 300 Computer, 301 Processor, 302 Memory, 303 Storage, 304 Input / output interface, 305 Communication interface, 310 Control unit, 311 Memory unit, 400 MR system, 410 MR device, 411 Display, 412 Gaze sensor, 413 Camera, 415 Microphone, 416 Speaker, 417 Sensor, 460 Detection device, 470 Display, 480 Controller, 486 Sensor, 500 Computer, 501 Processor, 502 Memory, 503 Storage, 504 Input / Output Interface, 505 Communication Interface, 510 Control Unit, 511 Storage Unit, 600 Server, 601 Processor, 602 Memory, 603 Storage, 604 Input / Output Interface, 605 Communication Interface, 610 Control Unit, 611 Storage Unit, 700 External Device, 810 Virtual Space Generation Unit, 812 Coordinate Definition Unit, 816 MR Side Input Reception Unit, 818 User Information Acquisition Unit, 820 MR Side Object Control Unit, 840 Display Control Unit, 845 Sound Control Unit, 850 Communication Control Unit, 916 VR Side Input Reception Unit, 920 VR Side Object Control Unit, 930 Virtual Camera Control Unit, 940 Display Control Unit, 945 Sound Control Unit, 950 Communication Control Unit

Claims

1. a first display means for providing a first user with a first view, which is a view of a mixed reality space in which a virtual object is arranged in a predetermined real space; a second display means for providing a second user with a second view, the second view being a view of a virtual space generated based on three-dimensional information of the predetermined real space; a first control means for controlling a virtual object in the virtual space; a second control means for controlling a virtual object in the mixed reality space, The first control means Placing a predetermined virtual object operable by the first user and the second user in the virtual space; imparting a predetermined change to the predetermined virtual object in the virtual space based on an operation of the first user; and applying the predetermined change to the predetermined virtual object in the virtual space based on an operation of the second user; The second control means placing the predetermined virtual object in the mixed reality space; applying the predetermined change to the predetermined virtual object in the mixed reality space based on an operation of the first user; and applying the predetermined change to the predetermined virtual object in the mixed reality space based on an operation of the second user. Information processing system.

2. The first control means is capable of imparting a first change to the predetermined virtual object in the virtual space based on an operation of the first user, and of imparting a second change to the predetermined virtual object in the virtual space to which the first change has been imparted based on an operation of the second user; The second control means is capable of imparting the first change to the predetermined virtual object in the mixed reality space based on an operation of the first user, and imparting the second change to the predetermined virtual object in the mixed reality space to which the first change has been imparted based on an operation of the second user. The information processing system according to claim 1 .

3. The predetermined change includes at least one of a change in the relative positional relationship between parts of the predetermined virtual object, or a change in the shape of the predetermined virtual object. The information processing system according to claim 1 .

4. a first display means for providing a first user with a first view, which is a view of a mixed reality space in which a virtual object is arranged in a predetermined real space; a second display means for providing a second user with a second view, the second view being a view of a virtual space generated based on three-dimensional information of the predetermined real space; a first control means for controlling a virtual object in the virtual space; and second control means for controlling a virtual object in the mixed reality space, causing a computer of the information processing system to function as the first control means and the second control means; The first control means Placing a predetermined virtual object operable by the first user and the second user in the virtual space; imparting a predetermined change to the predetermined virtual object in the virtual space based on an operation of the first user; and applying the predetermined change to the predetermined virtual object in the virtual space based on an operation of the second user; The second control means placing the predetermined virtual object in the mixed reality space; applying the predetermined change to the predetermined virtual object in the mixed reality space based on an operation of the first user; and applying the predetermined change to the predetermined virtual object in the mixed reality space based on an operation of the second user. program.

Images