Information processing apparatus, information processing program, information processing method, and information processing system

Abstract

This information processing device is provided with: an acquisition unit for acquiring local space information indicating space information of a local space in which a terminal device is positioned, local position information indicating position information of the terminal device in the local space, and remote space information indicating space information of a remote space positioned at a location different from the local space; and a generation unit that, on the basis of the local space information, the local position information, and the remote space information, generates information that pertains to a virtual video to be displayed on the terminal device when the terminal device is regarded as being in a position within the remote space that corresponds to the local position information.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing program, and an information processing system.

Background Art

[0002] In recent years, virtual space technology has been rapidly developing. By using virtual space technology, a user can move a virtual space displayed on a terminal device, for example, based on movement information of a finger on a sensor.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, services have started in which users can virtually enter a space located remotely (hereinafter also referred to as a remote space) in real time. However, with conventional virtual space technology, the user can only move within a virtual reality space that copies the remote space using a sensor, and cannot effectively experience the remote space.

[0005] Therefore, the present disclosure proposes an information processing apparatus, an information processing program, and an information processing system that can effectively experience a remote space.

Means for Solving the Problems

[0006] In order to solve the above problems, an information processing apparatus according to one aspect of the present disclosure includes: local space information indicating space information of a local space where a terminal device is located; local position information indicating position information of the terminal device in the local space; and an acquisition unit that acquires remote space information indicating space information of a remote space located at a location different from the local space, and based on the local space information, the local position information, and the remote space information, a generation unit that generates information for generating a virtual video for display on the terminal device, which is a video in a case where the terminal device is assumed to be at a position corresponding to the local position information in the remote space.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] Also, in this specification and the drawings, there are cases where a plurality of components having substantially the same functional configuration are distinguished by attaching different numbers after the same reference numeral. For example, a plurality of components having substantially the same functional configuration are distinguished as terminal devices 301, 302, and 303 as needed. However, when it is not necessary to particularly distinguish each of a plurality of components having substantially the same functional configuration, only the same reference numeral is attached. For example, when it is not necessary to particularly distinguish the terminal devices 301, 302, and 303, they are simply referred to as the terminal device 30.

[0010] Also, the present disclosure will be described in accordance with the following item order. 1. Outline of this embodiment 2. Configuration of the information processing system 2-1. Configuration of the server 2-2. Configuration of the imaging device 2-3. Configuration of the Terminal Device 3. Operations of the Information Processing System 3-1. Transmission and Reception of Data 3-2. Method for Allocating the Remote Space to the Local Space 3-3. Setting Process 3-4. First Generation Process 3-5. Second Generation Process 3-6. Application Examples 4. Modification Examples 5. Conclusion

[0011] <<1. Overview of the Present Embodiment>> Services using next-generation communication technologies such as local 5G are being actively developed. In such a situation, services have started in which users can virtually enter a space different from the local space (hereinafter referred to as the remote space), such as a concert hall or a construction site, in real time from the space where they are currently located (referred to as the local space). However, in conventional virtual space technologies, users can only move within a virtual space that copies the remote space using sensors, and cannot effectively experience the remote space.

[0012] When using a service that enters from the local space to the remote space, it is desirable for the user to secure a space in the local space that is the same size as the remote space. If the remote space is a narrow space, it may be possible to secure a space in the local space that is the same size as the remote space. However, in many cases, it is difficult to secure a space in the local space that is the same size as the remote space.

[0013] FIG. 1 is a diagram showing the local space and the remote space. The local space is the space where the user is currently located, such as the user's own room or the office at the user's workplace. The remote space is a space in a remote location different from the local space, such as a concert hall, an exhibition hall, a construction site, or an operating room. In the example of FIG. 1, there are four local spaces, and one user is located in each of them. The system of the present embodiment is a system for realizing an experience as if each user in the local space were in the remote space.

[0014] FIG. 2 is a diagram for explaining the outline of the system of the present embodiment. In the present embodiment, in the remote space, devices for acquiring spatial information of the remote space (hereinafter referred to as remote space information), such as a camera and a 3D sensor, are installed. These devices are connected to the server via a network. The server acquires remote space information from these devices. Further, in the present embodiment, in each local space, devices (for example, a camera and a 3D sensor) for acquiring spatial information of the local space (hereinafter referred to as local space information) are installed. The server acquires local space information from these devices.

[0015] In addition, each user located in the local space wears a terminal device such as an xR glass. Since the user wears the terminal device and moves within the local space, it is safer to be able to see the obstacles in the home. Therefore, it is desirable that the terminal device is an AR type (optical see-through type) glass that superimposes an image on the real space. The terminal device is provided with a function of detecting the position information and / or attitude information of the user (hereinafter also referred to as position and attitude information), such as an acceleration sensor. The server acquires the position and attitude information of the user from the terminal device.

[0016] The server generates control information for causing the terminal device to display a virtual image based on the local space information, the local position information, and the remote space information. Here, the virtual image is an image when it is assumed that the user is at a position corresponding to the local position information in the remote space. The server transmits the control information to the terminal device via the network. The terminal device superimposes and displays the virtual image on the real space based on the control information. Thereby, the user can obtain a feeling as if being in the remote space while staying in the local space.

[0017] In the example of Fig. 2, the sizes of the plurality of local spaces are different from each other. However, the size of the remote space is the same. Therefore, it is necessary to match the size of the local space with the size of the local area. Fig. 3 is a diagram showing the state where the sizes of the local space and the remote space are matched. On the upper side of Fig. 3, two local spaces A and B and one remote space are shown. In the example of Fig. 3, the size of local space A is one-fourth of the size of the remote space, and the size of local space B is one-half of the size of the remote space. The server matches the scales of the local space and the remote space, but if the scales of people and objects are also matched, as shown in the upper diagram of Fig. 3, the sizes of people and objects will become extremely large. This will greatly damage the user experience quality.

[0018] Therefore, the server of the present embodiment changes the movement amount of the user according to the scale (the ratio of the local space and the remote space) without changing the scale of people and objects. For example, in the case of a scale of 4 times, 4 times the movement amount of the user in the local space becomes the movement amount of the user in the remote space. Generally, people's activities involve temporary movement of position, and they are more likely to be active in a stationary state. Changing the sizes of people and objects always causes discomfort. In the present embodiment, the scales of people and objects are not changed, but the movement speed is changed. The influence of changing the movement speed is limited to the time of movement and is also limited on the time axis. As a result, the discomfort of the user is suppressed, so that the user can effectively experience the remote space.

[0019] In the example of Fig. 3, all of the plurality of users are in the local space, but a part of the plurality of users may be in the remote space. Fig. 4 is a diagram showing the state where one of the users is in the remote space. In this case, the magnification of the movement amount of the person in the remote space remains 1 time, and only the magnification of the movement amount of the user in the local space changes according to the scale.

[0020] The overview of the present embodiment has been described above. Hereinafter, the information processing system 1 according to the present embodiment will be described in detail.

[0021] <<2. Configuration of Information Processing System>> First, the overall configuration of the information processing system 1 will be described.

[0022] FIG. 5 is a diagram showing a configuration example of an information processing system 1 according to an embodiment of the present disclosure. The information processing system 1 is a system for providing a user located in a local space with a feeling of being in a remote space. The information processing system 1 includes a server 10, an imaging device 20, and a terminal device 30. Note that the devices in the figure may be considered as devices in a logical sense. That is, a part of the devices in the figure may be realized by a virtual machine (VM), a container, Docker, etc., and they may be implemented on physically the same hardware.

[0023] The server 10, the imaging device 20, and the terminal device 30 each have a communication function and are connected via a network N. The server 10, the imaging device 20, and the terminal device 30 can be regarded as communication devices. In the example of FIG. 5, only one network N is shown, but there may be a plurality of networks N.

[0024] Here, the network N is a communication network such as a LAN (Local Area Network), a WAN (Wide Area Network), a cellular network, a fixed telephone network, a regional IP (Internet Protocol) network, or the Internet. The network N may include a wired network or a wireless network. Also, the network N may include a core network. The core network is, for example, an EPC (Evolved Packet Core) or a 5GC (5G Core network). Further, the network N may include a data network other than the core network. The data network may be a service network of a communication carrier, for example, an IMS (IP Multimedia Subsystem) network. Also, the data network may be a private network such as an enterprise internal network.

[0025] Communication devices such as server 10, imaging device 20, and terminal device 30 may be configured to connect to network N or other communication devices using wireless access technologies (RAT: Radio Access Technology) such as LTE (Long Term Evolution), NR (New Radio), Wi-Fi, Bluetooth (registered trademark), etc. At this time, the communication device may be configured to be able to use different wireless access technologies. For example, the communication device may be configured to be able to use NR and Wi-Fi. Also, the communication device may be configured to be able to use different cellular communication technologies (for example, LTE and NR). LTE and NR are a type of cellular communication technology, and by arranging a plurality of areas covered by the base station in a cell shape, mobile communication of the communication device is enabled.

[0026] Note that communication devices such as server 10, imaging device 20, and terminal device 30 may be connectable to network N or other communication devices using wireless access technologies other than LTE, NR, Wi-Fi, and Bluetooth. For example, the communication device may be connectable to network N or other communication devices using LPWA (Low Power Wide Area) communication. Also, the communication device may be connectable to network N or other communication devices using wireless communication of its own standard. Of course, the communication device may be connectable to network N or other communication devices using wireless communication of other known standards.

[0027] Hereinafter, the configurations of each device constituting the information processing system 1 will be specifically described. Note that the configurations of each device shown below are merely examples. The configurations of each device may be different from those shown below.

[0028] <2-1. Configuration of Server> First, the configuration of server 10 will be described.

[0029] Server 10 is an information processing device (computer) that performs processing to provide a user located in a local space with the feeling of being in a remote space. Any form of computer can be adopted for Server 10. For example, Server 10 may be a PC server, a mid-range server, or a mainframe server. Also, Server 10 may be an information processing device that performs data processing (edge processing) near a user or a terminal. For example, it may be an information processing device (computer) installed or built-in together with a base station or a roadside unit. Of course, Server 10 may be an information processing device that performs cloud computing.

[0030] FIG. 6 is a diagram showing a configuration example of Server 10 according to an embodiment of the present disclosure. Server 10 includes a communication unit 11, a storage unit 12, and a control unit 13. Note that the configuration shown in FIG. 6 is a functional configuration, and the hardware configuration may be different from this. Also, the functions of Server 10 may be implemented in a distributed manner in a plurality of physically separated configurations. For example, Server 10 may be configured by a plurality of server devices.

[0031] The communication unit 11 is a communication interface for communicating with other devices. For example, the communication unit 11 is a LAN (Local Area Network) interface such as a NIC (Network Interface Card). The communication unit 11 may be a wired interface or a wireless interface. The communication unit 11 communicates with the imaging device 20, the terminal device 30, etc. according to the control of the control unit 13.

[0032] The storage unit 12 is a storage device capable of reading and writing data such as DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), flash memory, and hard disk. The storage unit 12 functions as a storage means of Server 10.

[0033] Returning to FIG. 6, the control unit 13 is a controller that controls each part of the server 10. The control unit 13 is realized by a processor such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), for example. For example, the control unit 13 is realized by a processor executing various programs stored in a storage device inside the server 10, with the RAM (Random Access Memory) or the like as a working area. Note that the control unit 13 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). Any of the CPU, MPU, GPU, ASIC, and FPGA can be regarded as a controller.

[0034] The control unit 13 includes an acquisition unit 131 and a generation unit 132. Each block (acquisition unit 131 to generation unit 132) constituting the control unit 13 is a functional block indicating the function of the control unit 13. These functional blocks may be software blocks or hardware blocks. For example, each of the above-described functional blocks may be a single software module realized by software (including a microprogram), or may be a single circuit block on a semiconductor chip (die). Of course, each functional block may be a single processor or a single integrated circuit. The control unit 13 may be configured in a functional unit different from the above-described functional blocks. The method of configuring the functional blocks is arbitrary.

[0035] Note that the control unit 13 may be configured in a functional unit different from the above-described functional blocks. Also, some or all of the operations of each block (acquisition unit 131 to generation unit 132) constituting the control unit 13 may be performed by another device. For example, some or all of the operations of each block constituting the control unit 13 may be performed by the control unit 23 of the imaging device 20 or the control unit 33 of the terminal device 30. The operations of each block constituting the control unit 13 will be described later.

[0036] <2-2. Configuration of Imaging Device> Next, the configuration of the imaging device 20 will be described.

[0037] The imaging device 20 is an information processing device (computer) having an imaging function. The imaging device 20 is installed in a remote space and transmits an image of the captured remote space to the server 10.

[0038] FIG. 7 is a diagram showing a configuration example of the imaging device 20 according to an embodiment of the present disclosure. The imaging device 20 includes a communication unit 21, a storage unit 22, a control unit 23, a sensor unit 24, and an imaging unit 25. Note that the configuration shown in FIG. 7 is a functional configuration, and the hardware configuration may be different. Also, the functions of the imaging device 20 may be implemented in a plurality of physically separated configurations.

[0039] The communication unit 21 is a communication interface for communicating with other devices. For example, the communication unit 21 is a LAN interface such as a NIC. The communication unit 21 may be a wired interface or a wireless interface.

[0040] When the communication unit 21 includes a wireless interface, the communication unit 21 may be configured to connect to the network N using a wireless access technology such as LTE, NR, Wi-Fi, Bluetooth (registered trademark), etc. At this time, the communication device may be configured to be able to use different wireless access technologies. For example, the communication device may be configured to be able to use NR and Wi-Fi. Also, the communication device may be configured to be able to use different cellular communication technologies (e.g., LTE and NR). In addition, the imaging device 20 may be able to connect to the network N using a wireless access technology other than LTE, NR, Wi-Fi, and Bluetooth.

[0041] The storage unit 22 is a storage device capable of reading and writing data, such as DRAM, SRAM, flash memory, and hard disk. The storage unit 22 functions as a storage means of the imaging device 20. The captured data (e.g., image data and metadata) captured by the imaging unit 25 is stored in the storage unit 22. Note that the captured data may be in a file format.

[0042] The control unit 23 is a controller that controls each part of the imaging device 20. The control unit 23 is realized by a processor such as a CPU, MPU, or GPU, for example. For example, the control unit 23 is realized by the processor executing various programs stored in the storage device inside the imaging device 20, using the RAM or the like as a working area. Note that the control unit 23 may be realized by an integrated circuit such as an ASIC or FPGA. Any of the CPU, MPU, GPU, ASIC, and FPGA can be regarded as a controller.

[0043] The sensor unit 24 is a sensor that detects information about the remote space. For example, the sensor unit 24 is a 3D sensor that detects the shape of the remote space and the position, shape, etc. of the objects in the remote space. At this time, the sensor unit 24 may be a ToF (Time of Flight) camera type 3D sensor or a stereo camera type 3D sensor.

[0044] The imaging unit 25 is a conversion unit that converts an optical image into an electrical signal. The imaging unit 25 includes, for example, an image sensor and a signal processing circuit that processes the analog pixel signals output from the image sensor, and converts the light entering from the lens into digital data (image data). Note that the image captured by the imaging unit 25 is not limited to video (moving image), and may be a still image. Note that the imaging unit 25 can be referred to as a camera.

[0045] <2-3. Configuration of the Terminal Device> Next, the configuration of the terminal device 30 will be described. The terminal device 30 is an information processing device configured to be communicable with the server 10. For example, the terminal device 30 is an xR device such as an AR (Augmented Reality) device, a VR (Virtual Reality) device, or an MR (Mixed Reality) device. At this time, the xR device may be a glasses-type device such as AR glasses or MR glasses, or may be a head-mounted device such as a VR head-mounted display. When the terminal device 30 is an xR device, the terminal device 30 may be a stand-alone type device composed only of a user wearing part (for example, a glasses part). Further, the terminal device 30 may be a terminal-linked type device composed of a user wearing part (for example, a glasses part) and a terminal part (for example, a smart device) linked to the part.

[0046] Note that the terminal device 30 may be a mobile terminal such as a mobile phone, a smart device (smartphone or tablet), a PDA (Personal Digital Assistant), or a notebook PC. Further, the terminal device 30 may be a wearable device such as a smartwatch. Additionally, the terminal device 30 may be a portable IoT (Internet of Things) device.

[0047] FIG. 8 is a diagram showing a configuration example of the terminal device 30 according to an embodiment of the present disclosure. The terminal device 30 includes a communication unit 31, a storage unit 32, a control unit 33, an input unit 34, an output unit 35, a sensor unit 36, and an imaging unit 37. Note that the configuration shown in FIG. 8 is a functional configuration, and the hardware configuration may be different from this. Further, the functions of the terminal device 30 may be implemented in a distributed manner in a plurality of physically separated configurations.

[0048] The communication unit 31 is a communication interface for communicating with other devices. For example, the communication unit 31 is a LAN interface such as a NIC. Note that the communication unit 31 may be a wired interface or a wireless interface. When the communication unit 31 includes a wireless interface, the communication unit 31 may be configured to connect to the network N or other communication devices using wireless access technologies (RATs) such as LTE (Long Term Evolution), NR (New Radio), Wi-Fi, Bluetooth (registered trademark), etc. The communication unit 31 communicates with the server 10, the imaging device 20, etc. according to the control of the control unit 33.

[0049] The storage unit 32 is a data-readable / writable storage device such as a DRAM, SRAM, flash memory, hard disk, etc. The storage unit 32 functions as a storage means of the terminal device 30.

[0050] The control unit 33 is a controller that controls each part of the terminal device 30. The control unit 33 is realized by a processor such as a CPU, MPU, GPU, etc. For example, the control unit 33 is realized by the processor executing various programs stored in a storage device inside the terminal device 30 with a RAM or the like as a working area. Note that the control unit 33 may be realized by an integrated circuit such as an ASIC or FPGA. Any of the CPU, MPU, GPU, ASIC, and FPGA can be regarded as a controller.

[0051] The input unit 34 is an input device that receives various inputs from the outside. For example, the input unit 34 is an operating device for the user to perform various operations, such as a keyboard, a mouse, and operation keys. Note that when a touch panel is adopted in the terminal device 30, the touch panel is also included in the input unit 34. In this case, the user performs various operations by touching the screen with a finger or a stylus.

[0052] The output unit 35 is a device that performs various outputs to the outside, such as sound, light, vibration, and images. The output unit 35 includes a display device that displays various information. The display device is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. When a touch panel is adopted in the terminal device 30, the display device may be an integrated device with the input unit 34. Further, when the terminal device 30 is an xR device (for example, AR / MR glasses), it may be a transmissive device that projects an image on the glass, or a retinal projection type device that projects an image directly onto the user's retina. The output unit 35 performs various outputs to the user according to the control of the control unit 33.

[0053] The sensor unit 36 is a sensor that acquires information regarding the position or orientation of the terminal device 30. For example, the sensor unit 36 is an acceleration sensor. Note that the sensor unit 36 is not limited to an acceleration sensor. The sensor unit 36 may be an IMU (Inertial Measurement Unit), a geomagnetic sensor, or a 3D sensor. Further, the sensor unit 36 may be a GNSS (Global Navigation Satellite System) sensor. The GNSS sensor may be a GPS (Global Positioning System) sensor, a GLONASS sensor, a Galileo sensor, or a QZSS (Quasi-Zenith Satellite System) sensor. Also, the sensor unit 36 may be a combination of these multiple sensors.

[0054] The imaging unit 37 is a conversion unit that converts an optical image into an electrical signal. The imaging unit 37 includes, for example, an image sensor and a signal processing circuit that processes the analog pixel signals output from the image sensor, and converts the light entering from the lens into digital data (image data). Note that the image captured by the imaging unit 37 is not limited to a video (moving image), and may be a still image. Note that the imaging unit 37 can be referred to as a camera.

[0055] <<3. Operation of the Information Processing System>> As described above, the configuration of the information processing system 1 has been described. Next, the operation of the information processing system 1 will be explained.

[0056] <3-1. Transmission and Reception of Data> First, the transmission and reception of data will be explained. FIG. 9 is a diagram for explaining the transmission and reception of data.

[0057] First, the server 10 acquires remote space information from the imaging device 20 installed in the remote space. The remote space information is three-dimensional information of the remote space. The remote space information may include information on the area size of the remote space (for example, information on the vertical, horizontal, and height sizes of the remote space).

[0058] Also, when the terminal device 30 located in the local space A starts to connect to the server 10, the server 10 acquires local space information from the terminal device 30 located in the local space A (or a sensor installed in the local space A). The local space information is three-dimensional information of the local space. The local space information may include information on the area size of the local space (for example, information on the vertical, horizontal, and height sizes of the local space).

[0059] Then, based on the local space information and the remote space information of the local space A, the server 10 generates information regarding a virtual video, which is the first time after the start of connection. The information regarding the virtual video is spatial data of the remote space (for example, three-dimensional data of the remote space) for the terminal device 30 to generate a virtual video. Then, the server 10 transmits the spatial data of the remote space to the terminal device 30 located in the local space A. The terminal device 30 generates and displays a virtual video based on the spatial data.

[0060] Further, the server 10 acquires the position and orientation information of the user from the terminal device 30 located in the local space A (or a sensor installed in the local space A). The position and orientation information is information on the position and / or orientation of the user. In the following description, the position information of the user may be referred to as local position information.

[0061] Then, the server 10 generates information regarding a virtual image based on the local space information of the local space A, the remote space information, and the position and orientation information of the user. For example, the server 10 generates spatial data of the remote space. The virtual image created here is an image that would be the user's field of view if the user were in the remote space. Then, the server 10 transmits the spatial data of the remote space to the terminal device 30 located in the local space A. The terminal device 30 generates and displays a virtual image based on the spatial data.

[0062] Also, when the terminal device 30 located in the local space B starts connecting to the server 10, the server 10 acquires local space information from the terminal device 30 located in the local space B (or a sensor installed in the local space B).

[0063] Then, the server 10 generates information regarding a virtual image, which is the first time after the connection start, based on the local space information of the local space B and the remote space information. The information regarding the virtual image is the spatial data of the remote space for the terminal device 30 to generate a virtual image. Then, the server 10 transmits the spatial data of the remote space to the terminal device 30 located in the local space B. The terminal device 30 generates and displays a virtual image based on the spatial data.

[0064] Further, the server 10 acquires the position and orientation information of the user from the terminal device 30 located in the local space B (or a sensor installed in the local space B).

[0065] Then, based on the local space information, remote space information, and the user's position and orientation information of the local space B, the server 10 generates information regarding virtual images. For example, the server 10 generates spatial data of the remote space. Then, the server 10 transmits the spatial data of the remote space to the terminal device 30 located in the local space B. The terminal device 30 generates and displays a virtual image based on the spatial data.

[0066] Note that the server 10 repeats the transmission of spatial data and the reception of position and orientation information at regular time intervals. The transmission of spatial data is, for example, at 1-second intervals, and for this data transmission, for example, 38 Mbps is required. Also, the transmission of position and orientation data is, for example, at 1-ms intervals, and for this data transmission, for example, 0.1 Mbps is required.

[0067] Note that the transmission of spatial data does not necessarily have to be reliable. The terminal device 30 can generate a virtual image even without an update of the spatial data. However, if the position and orientation information is missing, the virtual image will become extremely unnatural. For example, problems such as the video not being updated while the user is moving will occur. Therefore, the server 10 performs priority control on the transmission of position and orientation information, for example, by QoS control, so as to realize real-time interaction even in a poor communication environment.

[0068] Note that in 5G, in order to provide communication services optimized for various communication characteristics according to use cases, the concept of network slicing is introduced. Here, network slicing refers to the concept of constituting a network slice, and a network slice refers to a logical resource partition that virtually separates network functions for management and / or operation. For example, in network slicing, operations such as changing the unit for sending data according to the use of the virtually separated slices can be considered. In the following description, a network slice may sometimes be simply referred to as a slice.

[0069] The server 10 of this embodiment may transmit or receive local position information and spatial data using different network slices. At this time, the server 10 may use a slice with lower latency for communicating local position information than the slice used for communicating spatial data. For example, the server 10 may use a slice of the URLLC (Ultra-Reliable and Low Latency Communications) type for communicating local position information and a slice of the eMBB (Enhanced Mobile Broadband) type for communicating spatial data.

[0070] <3-2. Method of Assigning Remote Space to Local Space> There are roughly two methods of assigning local space to remote space. One is the mobility method (the first method), and the other is the division method (the second method).

[0071] FIG. 10 is a diagram for explaining the mobility method. The mobility method is a method of assigning local space to remote space by enlarging or reducing the scale of the remote space or the local space. In the mobility method, the server 10 calculates the virtual movement amount of the user according to the enlargement or reduction of the scale. For example, assume that the local space is a space with a size of 1 m × 1 m and the remote space is a space with a size of 2 m × 2 m. In this case, the size of the space obtained by enlarging the vertical and horizontal dimensions of the local space by a factor of 2 each is the same as the size of the remote space. Therefore, the server 10 sets the movement amount of the user in the local space multiplied by 2 as the movement amount of the user in the remote space. When the sizes of the local space and the remote space are close, this mechanism for correcting the difference in space is easy to use. When the mobility method is used, the user has a completely free viewpoint with respect to the remote space.

[0072] FIG. 11 is a diagram for explaining the division method. The division method is a method of dividing a remote space or a local space according to the other space. In this case, the local space is assigned to one of the divided spaces of the remote space. When the remote space is divided by the size of the local space, the amount of movement of the user in the local space remains the amount of movement of the user in the remote space. Note that in the division method, the user can move only a part of the remote space. When the scales of the remote space and the local space are significantly different, only a part of the remote space can be moved. The user enjoys the remote space within the field of view. When moving outside the assigned range in the remote space, it is possible by the user performing a predetermined operation such as a button operation. Note that, for example, when the remote space is a live venue, this mechanism is easy to use when the sizes of the local space and the remote space are significantly different. When the division method is used, the user has a free viewpoint within a specific range in the remote space.

[0073] <3-3. Setting Process> Next, the setting process will be described. When there is a connection from the terminal device 30, the server 10 performs a setting process regarding the method of assigning the local space to the remote space. FIG. 12 is a flowchart for explaining the setting process. The setting process is executed by the control unit 13 of the server 10. Hereinafter, the setting process will be described with reference to the flowchart of FIG. 12.

[0074] First, the acquisition unit 131 of the server 10 acquires remote space information from the imaging device 20 installed in the remote space (or the server that manages the imaging device 20) (step S101). The remote space information includes information on the area size of the remote space. For example, the remote space information includes information on the vertical, horizontal, and height (X R , Y R , Z R ) of the remote space. In addition, the remote space information may include information on the shape and area of the remote space.

[0075] Subsequently, the acquisition unit 131 of the server 10 acquires local space information from the terminal device 30 located in the local space (or a sensor placed in the local space) (step S102). The local space information includes information on the area size of the local space. For example, the local space information includes information on the vertical, horizontal, and height dimensions (X L , Y L , Z L ) of the local space. Additionally, the local space information may include information on the shape or area of the local space.

[0076] Next, the generation unit 132 of the server 10 compares the area size of the local space with the area size of the remote space. Specifically, the generation unit 132 calculates a scaling ratio when allocating the local space to the remote space based on the information on the area size of the local space and the information on the area size of the remote space (step S103). The scaling ratio may be, for example, the ratio of the respective sizes of the vertical and horizontal dimensions of the local space and the remote space. For example, if the vertical and horizontal sizes of the remote space are X R , Y R , and the vertical and horizontal sizes of the local space are X L , Y L , then the scaling ratios D X , D Y are as shown in the following equations (1) and (2).

[0077] D X = X R / X L …(1) D Y = Y R / Y L …(2)

[0078] Subsequently, the generation unit 132 determines an allocation method for the remote space to the local space based on the comparison result between the area size of the local space and the area size of the remote space. For example, the generation unit 132 uses the scaling ratios D X , D YBased on this, it is determined whether the method for allocating the remote space to the local space is the mobility method or the division method (step S104).

[0079] For example, when the difference in size between the local space and the remote space is smaller than a predetermined threshold, the generation unit 132 sets the allocation method to the mobility method. For example, the scale factor D X 、D Y If both are less than the predetermined threshold (step S104: Yes), the generation unit 132 sets the allocation method to the mobility method. The predetermined threshold is, for example, 1.2 (120%). In this case, the generation unit 132 makes settings regarding the virtual movement amount of the terminal device 30 in the remote space (step S105). For example, the generation unit 132 sets the virtual movement amount in the remote space when the user moves in the local space to the movement amount obtained by multiplying the movement amount in the local space by the scale factor.

[0080] On the other hand, when the difference in size between the local space and the remote space is equal to or greater than the predetermined threshold (or greater than the predetermined threshold), the generation unit 132 sets the allocation method to the division method. For example, the scale factor D X 、D Y If one of them (or if both scale factors D X 、D Y are) equal to or greater than the predetermined threshold (step S104: No), the generation unit 132 sets the allocation method to the division method. The predetermined threshold is, for example, 1.2 (120%). In this case, the generation unit 132 makes settings regarding the virtual movement amount of the terminal device 30 in the remote space (step S106). For example, the generation unit 132 sets the virtual movement amount in the remote space when the user moves in the local space to the movement amount in the local space as it is.

[0081] When the settings regarding the virtual movement amount are completed, the server 10 ends the setting process.

[0082] <3-4. First Generation Process> Next, the generation process of a virtual video based on the mobility method (the first generation process) will be described. FIG. 13 is a flowchart for explaining the first generation process. In the following description, it is assumed that the server 10 generates a virtual video to be displayed on the terminal device 30. Note that the server 10 may only generate spatial data for the terminal device 30 to generate a virtual video. Here, the terminal device 30 is an xR device (for example, an AR device). The server 10 executes the first generation process when the method of allocating the local space to the remote space is the mobility method. The first generation process is executed by the control unit 13 of the server 10. Hereinafter, the first generation process will be described with reference to the flowchart of FIG. 13.

[0083] First, the generation unit 132 of the server 10 performs scale adjustment between the local space and the remote space (step S201). For example, the generation unit 132 performs scale adjustment based on the comparison result between the area size of the local space and the area size of the remote space. The generation unit 132 calculates the scale factors for the vertical, horizontal, and height based on the area size of the local space and the area size of the remote space, and enlarges or reduces the scale of the remote space or the local space based on the calculated scale factors. FIG. 14 is a diagram showing the result of scale adjustment in the mobility method. The local coordinates are the coordinates in the local space, and the remote coordinates are the coordinates in the remote space. As a result of the scale adjustment, the server 10 can handle two spaces with different sizes at the same coordinates.

[0084] Subsequently, the generation unit 132 performs mobility adjustment (step S202). The generation unit 132 adjusts the mobility based on the scale factor calculated in step S201. For example, the generation unit 132 uses the scale factor calculated in step S201 as the mobility as it is.

[0085] Subsequently, the acquisition unit 131 of the server 10 acquires the user's position and orientation information from the terminal device 30. Then, based on the scale factor calculated in step S201 and the user's position and orientation information, the generation unit 132 of the server 10 generates a virtual video to be displayed on the terminal device 30, which is a virtual video assuming that the user is at a predetermined position (coordinates specified by the position and orientation information) within the remote space. Then, the server 10 performs processing for AR overlay rendering based on this virtual video (step S203). AR overlay rendering may also be performed by the terminal device 30.

[0086] Subsequently, the generation unit 132 starts tracking the movement (movement direction and amount) of the user in the local space while wearing the terminal device 30 (step S204). The generation unit 132 executes tracking at any time based on the position and orientation information from the terminal device 30. Then, based on the result of the tracking, the generation unit 132 updates the user's field of view (that is, the virtual video displayed on the terminal device 30) (step S205). For example, when the user wearing the terminal device 30 moves in the local space, the generation unit 132 calculates the virtual movement amount of the user in the remote space. At this time, the generation unit 132 uses the value obtained by multiplying the movement amount of the user in the local space by the movement rate calculated in step S202 as the virtual movement amount of the user in the remote space. Then, the generation unit 132 updates the virtual video based on the calculated virtual movement amount.

[0087] Next, the server 10 determines whether the end condition of the first generation process is satisfied (step S206). For example, the server 10 determines whether it has received a connection termination notification from the terminal device 30. If the end condition is not satisfied (step S206: No), the server 10 returns to step S205 and continues the process of updating the field of view. If the end condition is satisfied (step S206: Yes), the server 10 ends the first generation process.

[0088] <3-5. Second Generation Process> Next, a virtual video generation process (second generation process) based on the segmentation method will be described. FIG. 15 is a flowchart for explaining the second generation process. In the following description, it is assumed that the server 10 generates a virtual video to be displayed on the terminal device 30. Note that the server 10 may only generate spatial data for the terminal device 30 to generate a virtual video. Here, the terminal device 30 is an xR device (for example, an AR device). The server 10 executes the second generation process when the method of allocating the local space to the remote space is the segmentation method. The second generation process is executed by the control unit 13 of the server 10. Hereinafter, the second generation process will be described with reference to the flowchart of FIG. 15.

[0089] First, the generation unit 132 of the server 10 performs scale adjustment between the local space and the remote space (step S301). In the case of the segmentation method, the scale of the local space and the scale of the remote space may be the same. Of course, even in the case of the segmentation method, the scale of the local space and the scale of the remote space may be different, similar to the movement rate method.

[0090] Subsequently, the server 10 performs a segmentation process (step S302). For example, the acquisition unit 131 of the server 10 acquires information on the area size of the local space and information on the area size of the remote space. Then, the generation unit 132 of the server 10 divides the remote space into a plurality of regions (hereinafter referred to as segmentation regions) according to the size of the local space. In the following description, the segmentation region assigned to the local space as an initial setting is referred to as an initial segmentation region.

[0091] Subsequently, the acquisition unit 131 of the server 10 acquires the position and orientation information of the user from the terminal device 30. Then, the generation unit 132 of the server 10 generates a virtual video to be displayed on the terminal device 30 based on the position and orientation information, assuming that the user is at a predetermined position (coordinates specified by the position and orientation information) within the initial segmentation region. Then, the server 10 performs a process for AR overlay rendering based on this virtual video (step S303). AR overlay rendering may be performed by the terminal device 30.

[0092] Subsequently, the generation unit 132 starts tracking the movement (movement direction and amount of movement) of the user wearing the terminal device 30 in the local space (step S304). The generation unit 132 executes tracking at any time based on the position and orientation information from the terminal device 30. Then, the generation unit 132 updates the user's field of view (that is, the virtual video displayed on the terminal device 30) based on the result of the tracking (step S305). For example, when the user wearing the terminal device 30 moves in the local space, the generation unit 132 calculates the virtual movement amount of the user in the divided area. At this time, the generation unit 132 uses the movement amount of the user in the local space as the virtual movement amount of the user in the divided area as it is. Then, the generation unit 132 updates the virtual video based on the calculated virtual movement amount.

[0093] Note that it is possible to move the divided area by performing a predetermined operation of the user. In the following description, the movement of the divided area is referred to as a scene change. FIG. 16 is a diagram for explaining the scene change. In the example of FIG. 16, the local space is a space of 1 m × 1 m in length and width, and the remote space is a space of 3 m × 2 m in length and width. In the example of FIG. 16, the user is located in the divided area A1. The user can move freely in 6 degrees of freedom up to the edge of the divided area A1. When the user arrives at the edge of the local space and performs a predetermined operation (for example, a gesture such as stepping on the foot or a button operation) on the spot, the server 10 performs a scene change. In the example of FIG. 16, due to the scene change, the image of the divided area A2 slides into the user's field of view.

[0094] Based on the above, the description of the flowchart in FIG. 15 is continued.

[0095] The generation unit 132 of the server 10 determines whether the user is located at the edge of the local space (whether there is no space to move in the moving direction) (step S306). If not located at the edge of the divided area (step S306: No), the generation unit 132 returns the process to step S305.

[0096] When located at the edge of the divided area (step S306: Yes), the generation unit 132 determines whether the scene change condition is satisfied (or whether an operation for scene change has been performed) (step S307). For example, the generation unit 132 determines whether a predetermined operation has been performed by the user. When the scene change condition is not satisfied (step S307: No), the generation unit 132 returns the process to step S305. When the scene change condition is satisfied (step S307: Yes), the generation unit 132 executes a scene change (step S308).

[0097] FIG. 17 is a diagram showing the state of a scene change. In the example of state S1 in FIG. 17, the local area is assigned to the divided area A1. When the user moves to the edge of the local area and performs a predetermined operation (state S2), the image of the adjacent divided area slides into the user's field of view (state S3). Here, the predetermined operation may be pressing a scene change button or continuing to step on the foot at the edge of the local area.

[0098] Returning to FIG. 15, the server 10 determines whether the end condition of the second generation process is satisfied (step S309). For example, the server 10 determines whether it has received a connection end notification from the terminal device 30. When the end condition is not satisfied (step S309: No), the server 10 returns to step S305. When the end condition is satisfied (step S309: Yes), the server 10 ends the second generation process.

[0099] <3-6. Application Example> FIG. 18 is a diagram showing an application example of the information processing system 1. The information processing system 1 shown in FIG. 18 is a system that enables medical personnel to perform surgeries from a remote location. In the example shown in FIG. 18, one person, an operating table, and an object (chair) are arranged in the remote space. Also, in the example shown in FIG. 18, there are two local spaces, and one person is arranged in each. The spaces indicated by diagonal lines are the spaces where the remote space is synthesized.

[0100] Now, pay attention to the local space 1. In AR, occlusion processing is used to avoid object overlap, but avoiding overlap at multiple points requires a common open area, which may extremely reduce the usable area. Therefore, in the application example, although the area with large obstacles is avoided as the space synthesis space, the area with small obstacles is recognized as the space synthesis space (AR surface). Here, the space synthesis space is the target area of the local space information.

[0101] On the right side of FIG. 18, when a virtual image is displayed on the terminal device 30, the state of the local space 1 as seen by the user is shown. The terminal device 30 is an xR device such as a VR device or an AR device. In the example of FIG. 18, the image of an object in the remote space overlaps the area with small obstacles. In this case, the server 10 makes the image (the image of a chair in the example of FIG. 18) overlying the small obstacle in a transparent state so that the user can see the small obstacle. When the terminal device 30 is an AR device, the image of the chair and the small obstacle appear to overlap. When the terminal device 30 is a VR device, the terminal device 30 performs a transparency process on the image of the chair so that the user can see the small obstacle. This reduces the risk of the user colliding with the small obstacle.

[0102] <<4. Modification Example>> The above-described embodiments are merely examples, and various modifications and applications are possible.

[0103] For example, in the above-described embodiment (the example of FIG. 14), the area size of the local space and the area size of the remote space were adjusted in scale. In this case, the aspect ratio of the local space and the aspect ratio of the remote space may be different. In this case, as shown in FIG. 14, even if the user moves in the local space in the 45° diagonal direction, the movement in the remote space will not be in the 45° diagonal direction. Therefore, when the server 10 allocates the local space to the remote space by the movement rate method, it may be allocated so as to maintain the aspect ratio. For example, the server 10 may make the space to be allocated to the remote space in the local space have a shape that matches the shape of the remote space.

[0104] Also, in the above-described embodiment, the shapes of the remote space and the local space were not very different, but there may be a case where the shapes of the remote space and the local space are very different. For example, one of the remote space and the local space may be an extremely long rectangle in the longitudinal direction and the other may be a square, or one may be a square or rectangle and the other may be a circle or ellipse. FIGS. 19 and 20 are diagrams showing processing examples when the shapes of the remote space and the local space are very different. In the example of FIG. 19, the remote space and the local space are not only different in shape but also greatly different in area size. In this case, the server 10 adopts the division method so as to include all regions of the remote space. On the other hand, in the example of FIG. 20, the area sizes of both are close. In this case, the server 10 adopts the movement rate method and adjusts the scale of the remote space or the local space so that all regions of the remote space are included. At this time, the server 10 may prevent the aspect ratio from being broken.

[0105] Also, in the above-described embodiment, the remote space was assumed to be a real space, but the remote space may be completely a cyber space (virtual space). When the remote space is a virtual space, the remote space information does not have to be information sensed by a device that senses the real world such as the imaging device 20. For example, the remote space information may be information of an artificial space designed by programming or the like and stored in the server 10 or the like.

[0106] The control device that controls the server 10, the imaging device 20, or the terminal device 30 in this embodiment may be realized by a dedicated computer system or may be realized by a general-purpose computer system.

[0107] For example, a communication program for executing the above-described operations is stored and distributed in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk. Then, for example, the program is installed in a computer, and the control device is configured by executing the above-described processing. At this time, the control device may be a device external to the server 10, the imaging device 20, or the terminal device 30 (for example, a personal computer). Further, the control device may be a device inside the server 10, the imaging device 20, or the terminal device 30 (for example, the control unit 13, the control unit 23, or the control unit 33).

[0108] Further, the communication program may be stored in a disk device provided in a server device on a network such as the Internet so that it can be downloaded to a computer. Further, the above-described functions may be realized by the cooperation of an OS (Operating System) and application software. In this case, the part other than the OS may be stored in a medium and distributed, or the part other than the OS may be stored in a server device so that it can be downloaded to a computer.

[0109] Further, among the respective processes described in the above embodiment, all or part of the processes described as being automatically performed can also be performed manually, or all or part of the processes described as being performed manually can be automatically performed by a known method. In addition, regarding the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings, they can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

[0110] Moreover, each component of each illustrated device is functionally conceptual and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of the distribution and integration of each device is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed and integrated in any unit according to various loads, usage situations, etc.

[0111] Also, the above-described embodiments can be appropriately combined in a region where the processing contents do not conflict. Also, the order of each step shown in the flowchart of the above-described embodiments can be appropriately changed.

[0112] Moreover, for example, this embodiment can also be implemented as any configuration constituting a device or a system, such as a processor as a system LSI (Large Scale Integration), a module using a plurality of processors, etc., a unit using a plurality of modules, etc., a set obtained by adding other functions to the unit (i.e., a part of the configuration of the device).

[0113] Note that in this embodiment, a system means a collection of a plurality of components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a single device in which a plurality of modules are housed in one housing are both systems.

[0114] Moreover, for example, this embodiment can take a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0115] <<5. Conclusion>> As described above, the server 10 generates spatial data for the terminal device 30 to generate virtual images based on the local spatial information, the local position information, and the remote spatial information. At this time, the server 10 calculates the virtual movement amount of the user in the remote space when the user moves in the local space, and generates information for updating the virtual image based on the calculated virtual movement amount. Since the virtual image is updated based on the virtual movement amount, the user can experience the feeling of moving in the remote space without discomfort. In addition, the server 10 changes the processing according to the size and shape of the remote space and the local space, so that the user can experience the movement in the remote space without any restrictions and without discomfort.

[0116] As described above, each embodiment of the present disclosure has been described. However, the technical scope of the present disclosure is not limited to the above-described embodiments as they are, and various modifications are possible without departing from the gist of the present disclosure. In addition, components across different embodiments and variations may be appropriately combined.

[0117] In addition, the effects in each embodiment described in this specification are merely examples and are not limiting, and there may be other effects.

[0118] Note that the present technology can also adopt the following configuration. (1) An acquisition unit that acquires local spatial information indicating spatial information of the local space where the terminal device is located, local position information indicating the position information of the terminal device in the local space, and remote spatial information indicating spatial information of a remote space located at a location different from the local space; A generation unit that generates information regarding a virtual image for display on the terminal device, which is a virtual image when the terminal device is assumed to be at a position corresponding to the local position information in the remote space, based on the local spatial information, the local position information, and the remote spatial information; An information processing apparatus comprising: (2) The acquisition unit acquires the posture information of the terminal device or the user of the terminal device, The generation unit generates information regarding the virtual video based on the local space information, the local position information, the remote space information, and the posture information. The information processing apparatus according to (1) above. (3) When the terminal device moves in the local space, the generation unit calculates the virtual movement amount of the terminal device in the remote space, and generates information for updating the virtual video based on the calculated virtual movement amount. The information processing apparatus according to (1) or (2) above. (4) The generation unit changes the calculation method of the virtual movement amount according to the method of allocating the local space to the remote space. The information processing apparatus according to (3) above. (5) The local space information includes information on the area size of the local space. The remote space information includes information on the area size of the remote space. The generation unit changes the method of allocating the local space to the remote space based on the comparison result between the area size of the local space and the area size of the remote space. The information processing apparatus according to (4) above. (6) When the difference in size between the local space and the remote space is smaller than a predetermined threshold, the generation unit allocates the local space to the remote space by a first method of enlarging or reducing the scale of the remote space or the local space, and calculates the virtual movement amount according to the magnitude of the enlargement or reduction of the scale. The information processing apparatus according to (5) above. (7) When the generation unit allocates the local space to the remote space by the first method, the generation unit allocates the local space to the remote space while maintaining the aspect ratio. The information processing apparatus according to (6) above. (8) When the difference in size between the local space and the remote space is greater than a predetermined threshold, the generation unit allocates the local space to the remote space by a second method of dividing one of the remote space or the local space to match the other space, and calculates the amount of movement in the local space of the terminal device as the virtual amount of movement as it is. The information processing apparatus according to (5) above. (9) The information regarding the virtual video is the spatial data of the remote space for the terminal device to generate the virtual video. The information processing apparatus according to any one of (1) to (8) above. (10) A communication unit that receives the local position information from the terminal device and transmits the spatial data to the terminal device. The communication unit transmits or receives at least the local position information and the spatial data using different network slices. The information processing apparatus according to (9) above. (11) The terminal device is an xR device. The generation unit generates the spatial data for the xR device to display the virtual video. The information processing apparatus according to (9) or (10) above. (12) The terminal device is an AR device. The generation unit generates the spatial data for the AR device to superimpose and display the virtual video on the real space. The information processing apparatus according to (11) above. (13) When there is an object in the local space, the generation unit generates the spatial data so that the video superimposed on the object is in a transparent state. The information processing apparatus according to (12) above. (14) A computer, A acquisition unit that acquires local space information indicating space information of a local space where a terminal device is located, local position information indicating position information of the terminal device in the local space, and remote space information indicating space information of a remote space located at a location different from the local space; A generation unit that generates information regarding a virtual video for display on the terminal device based on the local space information, the local position information, and the remote space information, the virtual video being a video that assumes the terminal device is at a position corresponding to the local position information in the remote space; An information processing program for causing the above to function. (15) An information processing system including an information processing device and one or more terminal devices located in a local space, wherein the terminal device includes a communication unit that transmits local space information indicating space information of the local space and local position information indicating position information of the terminal device in the local space to the information processing device; and the information processing device includes an acquisition unit that acquires the local space information, the local position information, and remote space information indicating space information of a remote space located at a location different from the local space; and a generation unit that generates information regarding a virtual video for display on the terminal device based on the local space information, the local position information, and the remote space information, the virtual video being a video that assumes the terminal device is at a position corresponding to the local position information in the remote space. Information processing system. (16) Acquire local space information indicating space information of a local space where a terminal device is located, local position information indicating position information of the terminal device in the local space, and remote space information indicating space information of a remote space located at a location different from the local space, Based on the local space information, the local position information, and the remote space information, generate information regarding a virtual video to be displayed on the terminal device, which is the video when the terminal device is assumed to be at a position corresponding to the local position information in the remote space. An information processing method comprising the above.

Explanation of Signs

[0119] 1 Information processing system 10 Server 20 Imaging device 30 Terminal device 11, 21, 31 Communication unit 12, 22, 32 Storage unit 13, 23, 33 Control unit 24, 36 Sensor unit 25, 37 Imaging unit 34 Input unit 35 Output unit 131 Acquisition unit 132 Generation unit

Claims

An acquisition unit that acquires local space information indicating space information of a local space where the terminal device is located, local position information indicating position information of the terminal device in the local space, and remote space information indicating space information of a remote space located at a location different from the local space; A generation unit that generates information regarding a virtual video for display on the terminal device based on the local space information, the local position information, and the remote space information, assuming that the terminal device is at a position corresponding to the local position information in the remote space; When the terminal device moves in the local space, the generation unit calculates a virtual movement amount of the terminal device in the remote space, and generates information for updating the virtual video based on the calculated virtual movement amount; The local space information includes information on the area size of the local space; The remote space information includes information on the area size of the remote space; Based on a comparison result between the area size of the local space and the area size of the remote space, the generation unit changes a method of allocating the local space to the remote space, and changes a method of calculating the virtual movement amount according to the method of allocating the local space to the remote space; When a difference in size between the local space and the remote space larger than the local space is greater than a predetermined threshold, the generation unit allocates the local space to the remote space by a second method of dividing the remote space into a plurality of scene-changeable spaces according to the local space, and calculates the movement amount of the terminal device in the local space as the virtual movement amount as it is; An information processing apparatus.

2. The acquisition unit acquires posture information of the terminal device or a user of the terminal device; The generation unit generates information regarding the virtual video based on the local space information, the local position information, the remote space information, and the posture information; The information processing apparatus according to claim 1.

3. When the difference in size between the local space and the remote space is smaller than a predetermined threshold, the generation unit allocates the local space to the remote space by a first method of enlarging or reducing the scale of the remote space or the local space, and calculates the virtual movement amount according to the magnitude of the enlargement or reduction of the scale. The information processing apparatus according to claim 1 or 2.

4. When the generation unit allocates the local space to the remote space by the first method, the generation unit allocates the local space to the remote space while maintaining the aspect ratio. The information processing apparatus according to claim 3.

5. The information regarding the virtual video is the spatial data of the remote space for the terminal device to generate the virtual video. The information processing apparatus according to any one of claims 1 to 4.

6. A communication unit that receives the local position information from the terminal device and transmits the spatial data to the terminal device. The communication unit transmits or receives at least the local position information and the spatial data using different network slices. The information processing apparatus according to claim 5.

7. The communication unit transmits or receives the local position information using a second network slice with lower latency than the first network slice used for communication of the spatial data. The information processing apparatus according to claim 6.

8. The first network slice is an eMBB (Enhanced Mobile Broadband) type network slice. The second network slice is a URLLC (Ultra-Reliable and Low Latency Communications) type network slice. The information processing apparatus according to claim 7.

9. The terminal device is an xR device. The generation unit generates the spatial data for the xR device to display the virtual video. The information processing apparatus according to any one of claims 5 to 8.

10. The terminal device is an AR device. The generation unit generates the spatial data for the AR device to superimpose and display the virtual video on the real space. The information processing apparatus according to claim 9.

11. When there is an object in the local space, the generation unit generates the spatial data so that the video superimposed on the object is in a transparent state. The information processing apparatus according to claim 10.

12. A computer, An acquisition unit that acquires local space information indicating spatial information of a local space where the terminal device is located, local position information indicating position information of the terminal device in the local space, and remote space information indicating spatial information of a remote space located at a location different from the local space; Function as a generation unit that generates information regarding a virtual video for display on the terminal device, assuming that the terminal device is at a position corresponding to the local position information in the remote space, based on the local space information, the local position information, and the remote space information; When the terminal device moves in the local space, the generation unit calculates a virtual movement amount of the terminal device in the remote space, and generates information for updating the virtual video based on the calculated virtual movement amount. The local space information includes information on the area size of the local space. The remote space information includes information on the area size of the remote space. Based on the comparison result between the area size of the local space and the area size of the remote space, the generation unit changes the allocation method of the local space to the remote space, and changes the calculation method of the virtual movement amount according to the allocation method of the local space to the remote space. When the difference in size between the local space and the remote space larger than the local space is greater than a predetermined threshold, the generation unit allocates the local space to the remote space by a second method of dividing the remote space into a plurality of spaces where scene changes are possible according to the local space, and calculates the movement amount of the terminal device in the local space as the virtual movement amount as it is. Information processing program.

13. An acquisition step of acquiring local space information indicating spatial information of a local space where the terminal device is located, local position information indicating position information of the terminal device in the local space, and remote space information indicating spatial information of a remote space located at a location different from the local space; A generation step of generating information regarding a virtual video for display on the terminal device, which is based on the local space information, the local position information, and the remote space information, and assumes that the terminal device is at a position corresponding to the local position information in the remote space. In the generation step, when the terminal device moves in the local space, a virtual movement amount of the terminal device in the remote space is calculated, and information for updating the virtual video is generated based on the calculated virtual movement amount. The local space information includes information on the area size of the local space. The remote space information includes information on the area size of the remote space. In the generation step, based on the comparison result between the area size of the local space and the area size of the remote space, the allocation method of the local space to the remote space is changed, and the calculation method of the virtual movement amount is changed according to the allocation method of the local space to the remote space. In the generation step, when the difference in size between the local space and the remote space, which is larger than the local space, is greater than a predetermined threshold, the local space is allocated to the remote space by a second method of dividing the remote space into a plurality of scene-changeable spaces in accordance with the local space, and the movement amount of the terminal device in the local space is calculated as the virtual movement amount as it is. Information processing method.

14. An information processing system including an information processing device and one or more terminal devices located in a local space, The terminal device includes A communication unit that transmits local space information indicating the space information of the local space and local position information indicating the position of the terminal device in the local space to the information processing device. The information processing device includes An acquisition unit that acquires the local space information, the local position information, and remote space information indicating the space information of a remote space located at a location different from the local space. A generation unit that generates information regarding a virtual video for display on the terminal device, which is based on the local space information, the local position information, and the remote space information, and assumes that the terminal device is at a position corresponding to the local position information in the remote space. The generation unit calculates a virtual movement amount of the terminal device in the remote space when the terminal device moves in the local space, and generates information for updating the virtual video based on the calculated virtual movement amount. The local space information includes information on the area size of the local space. The remote space information includes information on the area size of the remote space. Based on the comparison result between the area size of the local space and the area size of the remote space, the generation unit changes the allocation method of the local space to the remote space, and changes the calculation method of the virtual movement amount according to the allocation method of the local space to the remote space. When the difference in size between the local space and the remote space larger than the local space is greater than a predetermined threshold, the generation unit allocates the local space to the remote space by a second method of dividing the remote space into a plurality of scene-changeable spaces according to the local space, and calculates the movement amount of the terminal device in the local space as the virtual movement amount as it is. Information processing system.

Images