Information processing systems, programs, and information processing methods

Abstract

To provide a technique capable of outputting a virtual space on the basis of information obtained by sensing real space information indicating a structure of a real space and the real space, or a technique capable of allowing a system as an output destination of the sensing information to activate the sensing information, without depending on the kind of a method for acquiring the sensing information.SOLUTION: An information processing system includes at least one or more processors executing the following steps. A space reception step receives real space information. The real space information is information indicating the structure of the real space. The reception step receives sensing information. The sensing information is information obtained by sensing the real space. A space output step outputs virtual space information at real time. The virtual space information is information indicating a virtual space obtained by reflecting the result of sensing indicated by the sensing information, on the structure of the real space indicated by the real space information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, a program, and an information processing method.

Background Art

[0002] Conventionally, a digital twin analysis device includes a data reception unit that receives input of data for analysis, an evaluation index storage unit that stores an evaluation index for evaluating the accuracy of the data received by the data reception unit, a data evaluation unit that evaluates the accuracy of the received data based on the evaluation index stored in the evaluation index storage unit, a data analysis unit that performs analysis using the data selected based on the accuracy evaluated by the data evaluation unit, and an output unit that outputs the result analyzed by the data analysis means (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in recent years, a technology for outputting a virtual space based on real space information indicating the structure of the real space and information sensed from the real space, or a technology that enables the system of the output destination of the sensing information to utilize the sensing information regardless of the type of method for acquiring the sensing information has been demanded.

[0005] In view of the above circumstances, the present invention aims to provide a technology capable of outputting a virtual space based on real space information indicating the structure of the real space and information sensed from the real space, or a technology that enables the system of the output destination of the sensing information to utilize the sensing information regardless of the type of method for acquiring the sensing information. [Means for solving the problem]

[0006] According to one aspect of the present invention, an information processing system is provided. This information processing system comprises at least one processor that performs the following steps: In the spatial reception step, real space information is received. Real space information is information that indicates the structure of real space. In the reception step, sensing information is received. Sensing information is information obtained by sensing real space. In the spatial output step, virtual space information is output in real time. Virtual space information is information that indicates a virtual space in which the sensing results indicated by the sensing information are reflected in the structure of real space indicated by the real space information.

[0007] According to this disclosure, a virtual space can be output based on real-space information that shows the structure of the real space and information sensed from the real space, or the system to which the sensed information is output can utilize the sensed information regardless of the type of method used to acquire the sensed information. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example of the system configuration of Information Processing System 1. [Figure 2] This figure shows an example of the hardware configuration of server device 2. [Figure 3] This figure shows an example of the hardware configuration of the game engine device 3. [Figure 4] This figure shows an example of the hardware configuration of agent device 4. [Figure 5] This figure shows an example of the hardware configuration of camera 5a. [Figure 6] This figure shows an example of the hardware configuration of sensor 5b. [Figure 7] This figure shows an example of the hardware configuration for lighting 5c. [Figure 8] This figure shows an example of the hardware configuration of the 5D smartphone. [Figure 9] It is an example of a block diagram showing functions realized by the processor 21 of the server device 2. [Figure 10] It is an example of a block diagram showing functions realized by the processor 31 of the game engine device 3. [Figure 11] It is a diagram showing an example of the architecture of SSCP. [Figure 12] It is a diagram showing an example of a sequence diagram for explaining information processing according to an embodiment. [Figure 13] It shows data acquired from IoT equipment installed in a laboratory. [Figure 14] It shows data acquired from IoT equipment installed in a laboratory. [Figure 15] It shows data reproducing the flow of people in a laboratory. [Figure 16] It shows an example in which a wheelchair is controlled while recognizing the surrounding situation in a laboratory. [Figure 17] It shows data of the real space in which a wheelchair and a robot are controlled while recognizing the surrounding situation in a laboratory. [Figure 18] It shows data of the real space in which a wheelchair and a robot are controlled while recognizing the surrounding situation in a laboratory. [Figure 19] It is a diagram showing data of the real space in which an agent is controlled based on human movement. [Figure 20] It is a diagram showing data of the virtual space in which an agent is controlled based on human movement.

Mode for Carrying Out the Invention

[0009] [Embodiment] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic matters shown in the embodiments described below can be combined with each other.

[0010] 1. Definition of Terms First, the definition of each term will be explained.

[0011] A program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable medium readable by a computer, may be provided so as to be downloadable from an external server, or may be provided so that the program is launched on an external computer to realize its function on a client device (so-called cloud computing).

[0012] In addition, in this embodiment, the “section” may include, for example, a combination of hardware resources implemented by a circuit in a broad sense and information processing of software that can be specifically realized by these hardware resources. Also, in this embodiment, various types of information are handled, and these information are represented, for example, by physical values of signal values representing voltage and current, the level of signal values as a set of binary bits composed of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be executed on a circuit in a broad sense.

[0013] In addition, a circuit in a broad sense is a circuit realized by appropriately combining at least a circuit, circuitry, a processor, a memory, etc. That is, it includes an application specific integrated circuit (ASIC), programmable logic devices (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] This disclosure reveals an architecture called "SSCP." In SSCP (Spatial Simulation-based Cyber-Physical Model), humans, agents, and the real world are controlled through a virtual space. An example of the SSCP architecture will be shown later with reference to Figure 12.

[0015] A "virtual space" is a domain defined by information processing. A virtual space has the nature of data and does not have a physical form. In this embodiment, the virtual space is assumed to be a three-dimensional space, but it may also include a one-dimensional or two-dimensional space. A virtual space may include a centrally managed space, a space managed as common property (a so-called metaverse), etc. In this embodiment, the virtual space is represented by a digital twin.

[0016] A "digital twin" is a technology that represents a virtual space synchronized with the real space and time, based on real-time information collected from the real space and historical information from the real space. The virtual space represented by the digital twin is constructed on the premise that the spatial data matches the real physical space and that necessary changes in space are adjusted through sensors. The world of the digital twin may be constructed manually by the developer using BIM, etc., or it may be constructed from images acquired by a camera or data acquired by LiDAR (Light Detection and Ranging), provided that the purpose is taken into consideration. Furthermore, it can be constructed so that the objects themselves provide information through IoT devices such as the camera 5a and sensor 5b described later. In this embodiment, the digital twin has real-time capabilities, multi-agent orientation, multi-site symmetry, and spatiotemporal synchronization.

[0017] "Real-time performance" means that the virtual space is represented as a dynamic space with a time difference from the current real space being as close to zero as possible, and at a high frequency (for example, in units of 1 / 100th of a second). In other words, by having real-time performance, the virtual space is represented in real time.

[0018] "Multi-agent orientation" refers to a method where a digital twin is used to recreate a virtual space, assuming the presence of multiple participants in a virtual space controlled by multiple agents. Unlike typical games and motion capture systems that use scanning for small groups of people, multi-agent orientation does not assume a system with a small number of participants.

[0019] "Multi-site symmetry" means that the virtual space reproduced by the digital twin is not described in different ways depending on the participant's viewpoint. A virtual space with multi-site symmetry is described independently of the participant's location and viewpoint. In other words, a virtual space with multi-site symmetry is not intended to be described in a one-way direction from a single location.

[0020] "Spatiotemporal synchronization" means that the virtual space reproduced by the digital twin is not described in different ways by different participants. In a virtual space with spatiotemporal synchronization, multiple participants share the same virtual space.

[0021] "Real space" is the space in which various objects exist as physical entities. Real space is a concept that is the opposite of virtual space, in which no physical entities exist.

[0022] A "common ground" is a platform that enables multiple agents to share information with one another. The common ground provides a shared understanding to minimize misunderstandings among participants for information exchange, collaborative work, and coordinated actions. In the common ground, data with the same semantics but different data formats can be standardized and treated as data of the same format. Furthermore, in the common ground, data with similar semantics may also have their data formats standardized and treated as data of the same format. In this embodiment, the spatial reception unit 210 and the commonization unit 211 function as the common ground.

[0023] "Semantics" refers to the meaning, intent, or content of data. For example, even if two sets of data have the same semantics, they can be described in different forms or structures.

[0024] 2. System configuration of Information Processing System 1 Next, the system configuration of the information processing system 1 of this embodiment will be described with reference to Figure 1.

[0025] Figure 1 shows an example of the system configuration of Information Processing System 1. As shown in Figure 1, Information Processing System 1 includes a server device 2, a game engine device 3, an agent device 4, IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d), a human 6, an object 7, and a network N. Each of the server device 2, game engine device 3, agent device 4, and IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d) is configured to communicate with each other via the network N. As a result, each of the server device 2, game engine device 3, agent device 4, and IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d) can send or receive various information from each other. Note that there may be multiple instances of each of the server device 2, game engine device 3, agent device 4, IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d), human 6, and object 7. Here, the system exemplified in Information Processing System 1 consists of one or more devices or components. Therefore, even if a server device 2, a game engine device 3, an agent device 4, or an IoT device (camera 5a, sensor 5b, lighting 5c, smartphone 5d) is used alone, it is still included in the system exemplified by the information processing system 1.

[0026] 3. Hardware Configuration Next, the hardware configuration of the server device 2, game engine device 3, agent device 4, and IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d) of this embodiment will be described with reference to Figures 2 to 8.

[0027] 3.1. Hardware configuration of Server Device 2 Figure 2 shows an example of the hardware configuration of server device 2. As shown in Figure 2, server device 2 comprises a processor 21, a storage unit 22, and a communication unit 23, and these components are electrically connected within server device 2 via a communication bus. Server device 2 performs processing according to the embodiment.

[0028] The processor 21 performs processing and control of the overall operation related to the server device 2. The processor 21 is, for example, a central processing unit (CPU). The processor 21 reads a predetermined program stored in the memory unit 22 and executes processing based on the program, thereby realizing various functions related to the server device 2, such as the processing shown in Figure 12, which will be described later. Note that the processor 21 is not limited to a single unit, and may be implemented with multiple processors 21 for each function, or a combination thereof.

[0029] The storage unit 22 stores various types of information as defined above. This can be implemented, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the server device 2 executed by the processor 21, or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. The storage unit 22 stores various programs, variables, and data used by the processor 21 when it executes processing based on the programs related to the server device 2 executed by the processor 21. The storage unit 22 is an example of a storage medium.

[0030] The communication unit 23 preferably uses wired communication methods such as USB, IEEE1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as LTE / 3G / 4G / 5G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the server device 2 may communicate various information from the outside via the communication unit 23.

[0031] 3.2. Hardware configuration of game engine device 3 Next, the game engine device 3 will be described with reference to Figure 3. The game engine device 3 is hardware equipped with a game engine, which is software. The game engine is used as a development environment for creating games. The game engine has the characteristic of being able to place character data, terrain data, etc., in a virtual space, render the virtual space on the screen, and manipulate the virtual space using game-specific data. Any software such as Unity3D, Unreal Engine, GameMaker, Godot, Cocos2d-x, and PhyreEngine may be used as the game engine. Furthermore, by using a game engine, the information processing of this embodiment can be executed in real time.

[0032] Figure 3 shows an example of the hardware configuration of the game engine device 3. As shown in Figure 3, the game engine device 3 has a processor 31, a storage unit 32, and a communication unit 33, and these components are electrically connected via a communication bus inside the game engine device 3. The game engine device 3 performs processing according to the embodiment. For details on the processor 31, storage unit 32, and communication unit 33 of the game engine device 3, please refer to the processor 21, storage unit 22, and communication unit 23 of the server device 2.

[0033] 3.3. Hardware configuration of Agent Device 4 Next, the agent device 4 will be explained with reference to Figure 4. The agent device 4 is an autonomous agent other than humans 6 and animals, and is a general term for autonomously moving mobile bodies, robots, etc. The agent device 4 is a machine that can operate independently or as part of a vehicle management system. The agent device 4 is developed based on a robot system such as ROS (Robot Operating System) or Autoware. The agent device 4 can provide its own position coordinates and sensing information it has acquired to the server device 2.

[0034] Figure 4 shows an example of the hardware configuration of agent device 4. As shown in Figure 4, agent device 4 has a processor 41, a storage unit 42, a communication unit 43, a sensing unit 44, and a drive unit 45, and these components are electrically connected within agent device 4 via a communication bus. Agent device 4 performs processing according to the embodiment. For details on the processor 41, storage unit 42, and communication unit 43 of agent device 4, please refer to the processor 21, storage unit 22, and communication unit 23 of server device 2.

[0035] The sensing unit 44 operates based on specific physical or chemical principles to convert the detected input into an appropriate output signal. This conversion process often manifests as changes in electrical properties such as resistance, capacitance, voltage, and current.

[0036] The drive unit 45 is an actuator that converts control signals into physical action or motion. The drive unit 45 typically receives electrical, thermal, or pressure signals and converts them into mechanical action.

[0037] 3.4. Hardware Configuration of IoT Devices Next, IoT devices will be explained with reference to Figures 5 to 8. An IoT device is a device connected to a network N and capable of communicating with a server device 2 or a game engine device 3. In the example of information processing system 1 in Figure 1, the IoT devices include a camera 5a, a sensor 5b, lighting 5c, a smartphone 5d, etc. Since a wide variety of IoT devices can exist, the hardware configuration of the IoT device shown below is merely an example, and variations corresponding to different types of devices may exist.

[0038] Camera 5a acquires image data by capturing images of the real space. This image data is used as real space information or sensing information. Figure 5 shows an example of the hardware configuration of camera 5a. As shown in Figure 5, camera 5a has a processor 51a, a storage unit 52a, a communication unit 53a, and an imaging unit 54a, and these components are electrically connected within camera 5a via a communication bus. Camera 5a performs processing according to the embodiment. For details on the processor 51a, storage unit 52a, and communication unit 53a of camera 5a, please refer to the processor 21, storage unit 22, and communication unit 23 of server device 2.

[0039] The imaging unit 54a is equipped with an optical lens inside, which collects light from a subject located in the imaging direction of the camera 5a. This collected light is guided to an image sensor located inside the body of the camera 5a. The image sensor is configured to convert the collected light into an electronic signal, which is then processed and generated as image data.

[0040] Sensor 5b acquires sensing information by sensing the real space. Figure 6 shows an example of the hardware configuration of sensor 5b. As shown in Figure 6, sensor 5b has a processor 51b, a storage unit 52b, a communication unit 53b, and a sensing unit 54b, and these components are electrically connected via a communication bus inside sensor 5b. Sensor 5b performs processing according to the embodiment. For details on the processor 51b, storage unit 52b, and communication unit 53b of sensor 5b, please refer to the processor 21, storage unit 22, and communication unit 23 of server device 2. As IoT devices having a hardware configuration similar to sensor 5b, sensors such as LiDAR, ToF sensors, thermosensors, weight sensors, and microphones can be used. In addition, a sensor as an IoT device may include not only sensor 5b itself, which can sense from the real space, but also a device that collects data acquired by sensor 5b. For details on the sensing unit 54b of sensor 5b, please refer to the sensing unit 44 of agent device 4.

[0041] Lighting device 5c is an example of an IoT device that operates to illuminate a real space. Figure 7 shows an example of the hardware configuration of lighting device 5c. As shown in Figure 7, lighting device 5c has a processor 51c, a storage unit 52c, a communication unit 53c, and a drive unit 54c, and these components are electrically connected within lighting device 5c via a communication bus. Lighting device 5c performs processing according to the embodiment. For details of the processor 51c, storage unit 52c, and communication unit 53c of lighting device 5c, please refer to the processor 21, storage unit 22, and communication unit 23 of server device 2. Examples of IoT devices with a hardware configuration similar to lighting device 5c include automatic doors, elevators, water heaters, and air conditioners. For details of the drive unit 54c of lighting device 5c, please refer to the drive unit 45 of agent device 4.

[0042] Smartphone 5d is an example of an IoT device having a display in real space. Figure 8 shows an example of the hardware configuration of smartphone 5d. As shown in Figure 7, smartphone 5d has a processor 51d, a storage unit 52d, a communication unit 53d, an input unit 54d, and an output unit 55d, and these components are electrically connected within smartphone 5d via a communication bus. Smartphone 5d performs processing according to the embodiment. For details on the processor 51d, storage unit 52d, and communication unit 53d of smartphone 5d, please refer to the processor 21, storage unit 22, and communication unit 23 of server device 2. As IoT devices having a hardware configuration similar to smartphone 5d, devices such as speakers, screens, playback devices, and head-mounted displays may be used.

[0043] The input unit 54d may be included in the casing of the smartphone 5d or it may be external. For example, the input unit 54d may be integrated with the output unit 55d and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, a switch button, mouse, QWERTY keyboard, etc. may be used instead of a touch panel. In other words, the input unit 54d receives input based on operations performed by the user. This input is transmitted as a command signal to the processor 51d via the communication bus, and the processor 51d can perform predetermined control or calculations as needed.

[0044] The output unit 55d can function as a display unit for the smartphone 5d. The output unit 55d may be included in the casing of the smartphone 5d, for example, or it may be attached externally. The output unit 55d displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably done by using different display devices such as a CRT display, liquid crystal display, organic EL display, and plasma display, depending on the type of smartphone 5d.

[0045] Human 6 represents an individual present in the environment. Human 6 operates by using a smartphone 5d to report its state, location, etc., to server device 2. In addition, Human 6's state, location, and skeletal information are reported to server device 2 using various methods such as RFID and motion capture.

[0046] Object 7 includes any existing objects such as desks, chairs, cars, plants, buildings, luggage, benches, and traffic lights. The scope of Object 7 may also include walls, floors, rivers, and the sea. Furthermore, the scope of Object 7 may include the aforementioned IoT devices, people 6, etc.

[0047] 4. Functional Configuration This section will describe the functional configuration of this embodiment with reference to Figures 9 to 10. The information processing system 1 includes at least one or more processors 21, 31 that execute each of the functional units shown in Figures 9 to 10.

[0048] 4.1. Functional Configuration of Server Device 2 Figure 9 is an example of a block diagram showing the functions realized by the processor 21 of the server device 2. The processor 21 of the server device 2, which is an example of the information processing system 1, has the functions of a spatial reception unit 210, a common unit 211, and an information transmission / reception unit 212. As described above, the information processing by the software stored in the storage unit 22 is specifically realized by the processor 21, which is an example of hardware, and can be executed as each functional unit included in the processor 21.

[0049] The spatial reception unit 210 is configured to receive, obtain, or acquire real-space information from other devices via the network N and the communication unit 23. The spatial reception unit 210 is also configured to transmit or output real-space information to other devices via the communication unit 23 and the network N. Furthermore, the spatial reception unit 210, acting as a common ground, converts the real-space information into mesh format data (vertices, indices, materials, etc.) so that it can be used by the game engine. The spatial reception unit 210 then performs the spatial reception step.

[0050] Real-world spatial information is information that describes the state of a real space. Real-world spatial information includes information that represents spaces such as sites, buildings, and rooms. Real-world spatial information may also be based on information modeled using methods such as BIM (Building Information Modeling) and 3DCAD (3D Computer Aided Design). When real-world spatial information is based on BIM, 3DCAD, etc., it is necessary to model the connections between spaces, the information contained in each space, and the configuration of each space. Real-world spatial information may also be based on information obtained from platforms such as PLATEAU, GIS (Geographic Information System), and 3D spatial information platforms (Spatial ID). Real-world spatial information may also be based on point cloud data. When real-world spatial information is based on point cloud data, data reduction and attribute data assignment may be performed by converting or compressing the real-world spatial information as surface model information.

[0051] Real-world spatial information includes information about various objects 7 such as walls, floors, equipment, desks, chairs, and robots. This real-world spatial information contains information about changes in the position, state, and shape of various objects 7 in real space. Objects 7 may be modeled as abstract "points" and managed entirely through a BACS (Building Automation and Control System) using standard protocols such as BACnet (Building Automation and Control Networking Protocol). Furthermore, these objects may be processed using protocols such as MQTT (Message Queuing Telemetry Transport).

[0052] The commonization unit 211 converts the sensing information into a data format that can be interpreted by the system that receives the sensing information, according to the data attributes indicated by the sensing information. The commonization unit 211 functions as a common ground. The commonization unit 211 executes a commonization step.

[0053] The information transmission / reception unit 212 is configured to receive, receive, or acquire sensing information from other devices via the network N and the communication unit 23. The information transmission / reception unit 212 is also configured to transmit or output sensing information to other devices via the network N and the communication unit 23. The information transmission / reception unit 212 performs either an information transmission / reception step or a reception step.

[0054] Sensing information is information obtained by sensing or observing the real space with devices such as sensors and cameras. Sensing information may include information indicating temperature, humidity, pressure, vibration, acceleration, position, direction, velocity, gas concentration, pH, light intensity, light color, distance, image, sound, frequency, magnetic field, electric field, stress, strain, weight, radiation, etc. Sensing information may also include information defined by combining two or more of the information listed above (for example, temperature map information by combining position and temperature).

[0055] Details of the spatial reception unit 210, the commonization unit 211, and the information transmission / reception unit 212 will be described later.

[0056] 4.2. Functional Configuration of Game Engine Device 3 Figure 10 is an example block diagram showing the functions realized by the processor 31 of the game engine device 3. The processor 31 of the game engine device 3, which is an example of the information processing system 1, comprises a spatial output unit 310, a simulation unit 311, and an information transmission / reception unit 312. As described above, information processing by software stored in the memory unit 32 is concretely realized by the processor 31, which is an example of hardware, and can be executed as each functional unit included in the processor 31.

[0057] The spatial output unit 310 outputs virtual spatial information corresponding to real spatial information in real time. The spatial output unit 310 executes the spatial output step.

[0058] Virtual space information is information that represents a virtual space represented by a digital twin. In other words, virtual space information is information that represents a virtual space corresponding to the real space. In this embodiment, virtual space information is information that represents a virtual space in which the sensing results shown in the sensing information are reflected in the structure of the real space shown in the real space information. Furthermore, since virtual space information is output in real time, it is data that represents a virtual space that changes over time. In this embodiment, for example, virtual space information includes information that shows the posture of human 6, the positions and positional relationships of agent device 4, human 6, and object 7, and the acceleration, velocity, and weight they have. In this embodiment, for example, virtual space information may also include information that shows the distribution of temperature, humidity, pressure, vibration, gas concentration, pH, light intensity, light color, sound, frequency, magnetic field, electric field, radiation, etc., within the space.

[0059] The simulation unit 311 generates simulation information from real-world spatial information acquired in a time series. The simulation information is information obtained by simulating changes in real space in a virtual space. The simulation unit 311 executes the simulation step.

[0060] The information transmission / reception unit 312 is configured to receive, receive, or acquire various information from other devices via the network N and the communication unit 33. The information transmission / reception unit 312 is also configured to transmit or output various information to other devices via the communication unit 33 and the network N. The information transmission / reception unit 312 performs either an information transmission / reception step or an information reception step.

[0061] Details of the spatial output unit 310, the simulation unit 311, and the information transmission / reception unit 312 will be described later.

[0062] 5. SSCP Architecture Next, the relationship between the system, hardware, and software and the SSCP architecture described above will be explained with reference to Figure 11. Figure 11 shows an example of the SSCP architecture. The SSCP in Figure 11 includes at least a physical environment layer, an abstraction / storage / synchronization layer, a simulation layer, and a client application.

[0063] The physical environment layer includes real-world spatial information such as BIM and point cloud data, sensors (camera 5a, sensor 5b), actuators (lighting 5c), a robot (agent device 4), a human 6, and audiovisual information (smartphone 5d).

[0064] The abstraction / storage / synchronization layer includes the model server, the actor network, the IoT platform, and the coordination infrastructure. This abstraction / storage / synchronization layer is implemented by server device 2.

[0065] The model server corresponds to the spatial reception unit 210. The spatial reception unit 210 stores information on geometry present in the target space, such as buildings, facilities, equipment, and robots, and provides it upon request. The spatial reception unit 210 aggregates geometry information generated from spatial information such as BIM, point clouds, and photographs. The spatial reception unit 210 converts the geometry information into mesh format information (vertices, indices, materials, etc.) so that it can be easily used in game engines. Note that the geometry information and mesh format information are included in the real-world spatial information.

[0066] The actor network corresponds to the common unit 211. The common unit 211 models elements or actors present in the physical environment as a network and can provide information upon request. All actors are linked to the IoT platform (information transmission / reception unit 212) and the real actor infrastructure (information transmission / reception unit 212) to ensure compatibility with graph databases or the web, and perform necessary state transformations.

[0067] The IoT platform and coordination infrastructure correspond to the information transmission / reception unit 212. The IoT platform primarily collects, stores, and provides time-series data about people 6 and objects 7 in the real world. When collecting data, the IoT platform can use gateways to perform appropriate data modeling and abstraction, thereby improving machine readability. The IoT platform may utilize a time-series database, or it may utilize HDFS (Hadoop Distributed File System) from the perspective of data storage efficiency or cost. The coordination infrastructure provides functionality for more frequent data exchange with IoT devices. The coordination infrastructure may, for example, perform data exchange using MQTT (Message Queuing Telemetry Transport) and PUB / SUB with Apache Kafka. Unlike IoT platforms, the coordination infrastructure is not intended for long-term data storage.

[0068] The simulation layer includes the game engine and the data analysis infrastructure. This simulation layer is implemented by the game engine device 3.

[0069] The game engine corresponds to the spatial output unit 310, the simulation unit 311, and the information transmission / reception unit 312.

[0070] The data analysis infrastructure corresponds to the spatial output unit 310 and the simulation unit 311. The spatial output unit 310 and the simulation unit 311 have functions to perform various processes and modeling related to statistical analysis, machine learning (ML), AI, etc. in the game engine. For example, these functions perform batch processing based on information from the actor network or IoT platform (information transmission / reception unit 212), and the resulting model is made into an API to link with the game engine.

[0071] A client application is an application that uses data obtained from the abstraction / storage / synchronization layer or the simulation layer. A client application may be developed as a game engine or a web application, or as an application that supports SSCP.

[0072] 6. Operation Flow of Information Processing System 1 An example of preferred information processing performed in the information processing system 1 of this embodiment will be described.

[0073] In this specification, various types of information transmitted and received via communication units 23, 33, 43, 53a, 53b, 53c, and 53d are stored in the memory of the device having the communication unit used, even if not explicitly stated. For example, when information is transmitted and received between server device 2 and game engine device 3, communication units 23 and 33 are used, and the information is stored in memory units 22 and 32.

[0074] 6.1 Overview of Information Processing Next, we will explain the overview of the information processing with reference to Figure 12. Figure 12 is a diagram showing an example of a sequence diagram for explaining the information processing according to the embodiment.

[0075] (Step S1) The spatial reception unit 210 of server device 2 transmits real-world spatial information to game engine device 3. The information transmission / reception unit 312 of game engine device 3 receives real-world spatial information from server device 2. The spatial reception unit 210, acting as a common ground, converts the data format of the real-world spatial information to a format usable by the game engine as needed.

[0076] (Step S2) The IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d) acquire sensing information. The IoT devices transmit the sensing information to the server device 2 via the communication units 53a, 53b, 53c, 53d and the network N. The information transmission / reception unit 212 acquires sensing information from the IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d).

[0077] (Step S3) The commonization unit 211 of the server device 2, acting as a common ground, converts the sensing information into a data format that can be interpreted by the system (game engine) to which the sensing information is output, according to the attributes of the data indicated by the sensing information. The commonization unit 211 converts the data format of the sensing information so that it becomes a common data format when the semantics of the data of the sensing information are the same. With this configuration, if the semantics of the data indicated by the sensing information are the same, the system to which the sensing information is output can utilize the sensing information regardless of the method of acquiring the sensing information.

[0078] The sensing information includes information that allows for the recognition of the human 6's posture. For example, the sensing information includes skeletal information. Skeletal information is information about the skeleton of human 6 as it exists in real space. Skeletal information indicates the position of each part of human 6, such as joints, and their positional relationships, and is used to analyze posture and movement.

[0079] When sensors such as camera 5a and ToF sensors are used, the sensing information may include real-world spatial information. The commonization unit 211 aggregates the geometry generated from sensing information such as point clouds and image data that represent the structure of the real world, and converts it into mesh format data (including elements such as vertices, indices, and materials) so that it can be used in game engines or on the web. With this configuration, if the semantics of the data represented by the sensing information represent the structure of the real world, the agent can utilize the sensing information regardless of the type of sensor.

[0080] (Step S4) The information transmission / reception unit 212 of server device 2 transmits sensing information to game engine device 3. The information transmission / reception unit 312 of game engine device 3 receives sensing information from server device 2.

[0081] (Step S5) The spatial output unit 310 outputs virtual space information corresponding to real space information in real time, based on sensing information such as skeletal information and real space information. The spatial output unit 310 constructs a virtual space using real space information acquired from the server device 2 and characteristic functions of the game engine such as smart objects and navigation meshes. With this configuration, virtual space information can be output using functions such as the rendering engine, physics engine, animation, and lighting effects provided in the game engine.

[0082] (Step S6) The simulation unit 311 of the game engine device 3 generates simulation information from real-world spatial information and sensing information acquired in a time series. In the virtual space field generated by the spatial output unit 310, the simulation unit 311 performs pathfinding or spatial evaluation using a system such as EQS (Environment Query System), calculations for agent operation or interaction, etc., and acquires simulation information. The acquired simulation information may be used for reinforcement learning. The real-world spatial information and sensing information are converted into data usable by the game engine device 3 by the commonization unit 211 of the server device 2.

[0083] The simulation information is information obtained by simulating changes in the real world within a virtual space. The simulation information includes information obtained by simulating the actions of human 6, agent device 4, virtual agent, and object 7 in the real world within a virtual space. With this configuration, simulations in the real world can be performed within the virtual space. With this configuration, the actions of human 6 present in the real world can be predicted.

[0084] The simulation unit 311 may output command information along with the simulation information. The command information is information indicating commands for the agent. The agent may include not only the agent device 4 that exists in the real space, but also virtual agents that exist only in the virtual space. A normal agent synchronizes its body in the physical space and the digital twin space, but a digital agent has a body only in the digital twin space. If a virtual agent is included, the virtual space is described in a manner that includes the virtual agent. A virtual agent may be an AR avatar, a VR character, etc., and may have any shape and size. The virtual agent operates in the virtual space in coordination with the predicted movements of the human 6 based on the simulation information and command information. With this configuration, the virtual agent can be operated in coordination with the human 6 that exists in the real space.

[0085] The spatial output unit 310 and simulation unit 311 of the game engine device 3 control the meta AI, character AI, and spatial AI, and by coordinating these three types of AI, a variety of situations can be created within the game. The meta AI recognizes the overall situation and gives commands to the character AI to make the situation advantageous or to change game objects (objects within the game). The character AI acts as the brain of each agent, functioning to allow each agent to recognize the situation, make judgments, and move their body.

[0086] As a spatial AI, the spatial output unit 310 generates virtual space information in response to changes in the states of the real and virtual spaces, and can adjust the state of the virtual space based on this virtual space information. The spatial output unit 310 outputs virtual space information corresponding to the real space information in real time using a trained model that has been trained to generate virtual space information from real space information. With this configuration, virtual space information can be output with high accuracy using artificial intelligence.

[0087] Furthermore, the simulation unit 311, acting as a spatial AI, recognizes the spatial state and behaves in a way that considers what kind of command information to give to individual agents in order to change the spatial state. That is, it generates simulation information and command information in response to changes in the state of the real space and virtual space, and can adjust the state of the virtual space based on the virtual space information and command information. The simulation unit 311 outputs virtual space information and command information in real time using a trained model that has been trained to generate virtual space information and command information from real space information or virtual space information.

[0088] Meta AI, character AI, and spatial AI may perform reinforcement learning using any of the variables included in real-world spatial information, virtual-world spatial information, simulation information, and instruction information as features.

[0089] (Step S7) The information transmission / reception unit 312 (spatial output unit 310) transmits virtual space information, simulation information, and command information to the agent device 4. The agent device 4 receives the virtual space information and simulation information from the game engine device 3 via the network N and the communication unit 43.

[0090] Here, the temporal resolution of the virtual space information transmitted to the agent device 4 is adjusted as appropriate. That is, the information transmission / reception unit 312 controls the frequency at which it outputs virtual space information to the agent according to the service provided by the agent. For example, if there is an agent device that dances and an agent device that detects human faces, the dancing agent device requires delicate movements, so the information transmission / reception unit 312 sets the spatial resolution of the transmitted virtual space information to a relatively high level (high frequency). On the other hand, the human face detection agent device does not require as delicate movements as the dancing agent device, so the information transmission / reception unit 312 sets the spatial resolution of the transmitted virtual space information to a relatively low level (low frequency). With such a configuration, the frequency of the agent's operations can be controlled according to the service provided by the agent.

[0091] Here, the spatial resolution of the virtual space information transmitted to the agent device 4 is adjusted as appropriate. That is, the information transmission / reception unit 312 controls the resolution of the virtual space indicated by the virtual space information according to the service provided by the agent. For example, if there is an agent device that detects human faces and an agent device that transports luggage, the agent device that detects human faces is required to detect human faces with high resolution, so the information transmission / reception unit 312 sets the spatial resolution of the transmitted virtual space information to a relatively high (high precision). On the other hand, the agent device that transports luggage is not required to operate at a higher resolution than the agent device that detects human faces, so the information transmission / reception unit 312 can set the spatial resolution of the transmitted virtual space information to a relatively low (low precision). With this configuration, the resolution of the virtual space is controlled according to the service provided by the agent, so virtual space information can be efficiently provided to the agent.

[0092] (Step S8) Agent device 4 acts based on virtual space information, simulation information, and command information. For example, agent device 4 operates in a manner that coordinates with human 6, other agent devices, or virtual agents, taking into account the movements of human 6, other agent devices, or virtual agents. Agent device 4 does not simply determine its own actions based on sensing information, but rather determines its own actions based on virtual space information and simulation information. With this configuration, agents can be operated autonomously based on virtual space information, simulation information, and command information. Furthermore, by acquiring virtual space information from game engine device 3, agent device 4 can recognize its own position without using methods such as SLAM (Simultaneous Localization and Mapping) from its own sensors or cameras to recognize its own position. In addition, because agents are operated based on virtual space information, simulation information, and command information, even if there are multiple agents with different specifications, they can be controlled together. In the digital twin of this embodiment, agent device 4, human 6, and virtual agents behave in a manner that coordinates with each other.

[0093] 6.2. Examples Next, we will describe four examples of the information processing outlined above.

[0094] (Example 1) IoT devices acquire temperature map information. This temperature map information includes temperature (location and temperature pairs) corresponding to each location in real space and is an example of sensing information. The IoT device transmits temperature map information to the server device 2 via network N. The information transmission / reception unit 212 of server device 2 receives temperature map information from IoT devices. The commonization unit 211 of server device 2 converts temperature map information into data usable by game engine device 3 using common ground. The information transmission / reception unit 212 of the server device 2 transmits the converted temperature map information to the game engine device 3. The information transmission / reception unit 312 of the game engine device 3 receives the converted temperature map information from the server device 2. The spatial output unit 310 of the game engine device 3 outputs the virtual space as virtual space information in a manner that includes temperature information for each location, based on the converted temperature map information and the real space information. The simulation unit 311 of the game engine device 3 acquires simulation information by simulating the future temperature distribution of a virtual space based on time-series virtual space information. This simulation information includes information obtained by simulating the prediction of temperature changes in the real space within the virtual space. The simulation unit 311 of the game engine device 3 generates instruction information for the actions of each agent based on temperature changes. The information transmission / reception unit 312 of the game engine device 3 transmits virtual space information, simulation information, and command information to multiple agent devices. These agent devices include robots that operate based on the virtual space information, simulation information, and command information. Each robot receives virtual space information, simulation information, and command information from the game engine device 3. Each robot determines its own action based on the temperature conditions indicated by virtual space information, the predicted temperature changes indicated by simulation information, and command information. Each robot then operates based on the action it has decided upon. According to such an embodiment, temperature changes can be predicted based on temperature information corresponding to each location in real space, and the robot can be operated based on the results. Furthermore, according to such an embodiment, if the semantics of the data indicated by the sensing information indicate the temperature corresponding to each location in real space, the agent can utilize the sensing information regardless of the type of sensor.

[0095] (Example 2) IoT devices acquire weight information. This weight information includes the weight of packages present in the real world. More specifically, for example, the weight information includes the weight of multiple packages scheduled for delivery. Weight information is an example of sensing information. The IoT device transmits weight information to the server device 2 via the network N. The information transmission / reception unit 212 of server device 2 receives weight information from IoT devices. The commonization unit 211 of server device 2 converts weight information into data usable by game engine device 3 using common ground. The information transmission / reception unit 212 of server device 2 transmits the converted weight information to game engine device 3. The information transmission / reception unit 312 of the game engine device 3 receives the converted weight information from the server device 2. The spatial output unit 310 of the game engine device 3 outputs the virtual space as virtual space information in a manner that includes information on the weight of luggage present in the real space, based on the converted weight information and the real space information. The simulation unit 311 of the game engine device 3 acquires simulation information by simulating the cargo that each agent will carry in the space in the future, based on the converted weight information. The simulation unit 311 of the game engine device 3 generates command information regarding the cargo to be transported for each agent. The information transmission / reception unit 312 of the game engine device 3 transmits virtual space information, simulation information, and command information to the respective agent devices. These agents include robots whose size and weight of transportable cargo are defined. Each robot receives virtual space information, simulation information, and command information from the game engine device 3. The robots transport one or more loads that they are capable of carrying, based on virtual space information, simulation information, and command information. Each robot chooses a load of a weight it can lift and acts accordingly. For example, if a robot can carry up to 100 kg, it will systematically collect and transport multiple loads, such as a 50 kg load, a 30 kg load, a 20 kg load, and so on. According to such an embodiment, a robot capable of carrying an object can be operated based on information about the weight of the object in real space. According to such an embodiment, if the semantics of the data indicated by the sensing information indicate the weight of an object present in real space, the agent can utilize the sensing information regardless of the type of sensor.

[0096] (Example 3) IoT devices acquire density information. This density information includes the density of people in each section of a real-world space and is an example of sensing information. The IoT device transmits density information to the server device 2 via the network N. The information transmission / reception unit 212 of server device 2 receives density information from IoT devices. The commonization unit 211 of server device 2 converts density information into data usable by game engine device 3 using common ground. The information transmission / reception unit 212 of the server device 2 transmits the converted density information to the game engine device 3. The information transmission / reception unit 312 of the game engine device 3 receives the converted density information from the server device 2. The spatial output unit 310 of the game engine device 3 outputs the virtual space as virtual space information in a manner that includes information on the density of people in each section within the real space, based on the converted density information and the real space information. The simulation unit 311 of the game engine device 3 obtains simulation information by simulating the flow of people from the temporal change in the density of people in each section of the space, based on the converted density information. The simulation information includes information that simulates the prediction of the density of people in the real space in a virtual space. The simulation unit 311 of the game engine device 3 generates command information indicating that it should move to a location with a high density of people and trigger an alarm. The game engine device 3 transmits virtual space information, simulation information, and command information to the agent device. The agent includes a drone capable of generating alarms. The drone receives virtual space information, simulation information, and command information from the game engine device 3. The drone moves to areas with high human density based on virtual space information, simulation information, and command information, and triggers an alarm. According to this embodiment, changes in human density can be predicted based on information about the human density corresponding to each section in real space, and an alarm can be generated on the drone based on the results. According to this embodiment, if the semantics of the data indicated by the sensing information indicate the human density for each section in real space, the agent can utilize the sensing information regardless of the type of sensor.

[0097] (Example 4) An IoT device that functions as a transported package transmits package classification information and package location information to a server device 2 via network N. Package classification information includes information indicating the package's classification and is an example of sensing information. Similarly, package location information includes information indicating the package's location and is also an example of sensing information. The information transmission / reception unit 212 of the server device 2 receives package classification information and package location information from IoT devices via the network N. The commonization unit 211 converts the luggage classification information and luggage location information into data usable by the game engine device 3 using Commonground. The information transmission / reception unit 212 of the server device 2 transmits the converted package classification information and converted package location information to the game engine device 3. The information transmission / reception unit 312 of the game engine device 3 receives the converted package classification information and converted package location information from the server device 2. The spatial output unit 310 of the game engine device 3 outputs the virtual space as virtual space information in a manner that includes information on the classification of the luggage and information on the location of the luggage in the real space, based on the converted luggage classification information, the converted luggage location information, and the real space information. The simulation unit 311 of the game engine device 3 obtains simulation information by simulating the transportation of luggage by a robot from luggage information for each section in real space, based on the converted luggage classification information and converted luggage location information. The sections are predefined in real space. The simulation information includes information simulated in virtual space that predicts the transportation of luggage for each section in real space, and information that simulates the remaining luggage capacity at the luggage collection point based on luggage classification information and luggage location information acquired in a time series. The luggage collection point is an example of a predetermined location. The game engine device 3 transmits virtual space information, simulation information, and command information to the agent device. This agent includes a robot capable of carrying cargo. The robot receives virtual space information, simulation information, and command information from the game engine device 3. The robot transports the corresponding packages based on the capacity of the packages accumulated in the compartment. According to this embodiment, based on information about the classification and location of luggage in real space, the remaining capacity of luggage at a luggage collection point can be simulated. According to this embodiment, if the semantics of the data indicated by the sensing information indicate the classification and location of luggage, the system to which the sensing information is output can utilize the sensing information, regardless of the type of sensor.

[0098] 7. Experimental Examples Below, we will explain experimental examples using this disclosure, with reference to Figures 13 to 20.

[0099] Figures 13 and 14 show data acquired from IoT equipment installed in the experimental site. The experimental site shown in Figure 13 is prepared to acquire real-world spatial information from BIM, point cloud data, LiDAR, camera sensor data, etc. Figure 13 shows that the spatial output unit 310 of the game engine device 3 detected a human based on real-world spatial information. Figure 14 shows how the spatial output unit 310 of the game engine device 3 tracked a human based on time-series real-world spatial information.

[0100] Figure 15 shows data that reproduces the flow of people within the experimental area. In the example in Figure 15, the spatial output unit 310 of the game engine device 3 receives location information of people as sensing information from seven ToF sensors installed in the experimental area and integrates it with real-world spatial information (BIM in this example) in real time to provide a virtual space that accurately reproduces the real-world space, including the flow of people. Object detection and mapping of the virtual space were achieved using LiDAR with ToF sensors. As a result, the spatial output unit 310 of the game engine device 3 was able to reduce the rendering delay time of the entire virtual space to 0.6 seconds or less. In the example in Figure 15, the spatial AI, specifically the simulation unit 311 of the game engine device 3, analyzes the distance between people in the virtual space in real time, immediately identifies crowded situations, and provides command information to the display or speaker. Based on this command information, the display or speaker generates a warning in a manner that is recognizable to the user.

[0101] Figure 16 shows an example of a wheelchair being controlled while recognizing its surroundings in an experimental setting. In the example in Figure 16, the wheelchair used in the experiment is an agent that does not have sensors to recognize the surrounding environment. The wheelchair acquired virtual space information and command information, and based on this virtual space information and command information, recognized the 3D structure of the space, real-time human flow and obstacles, and moved safely.

[0102] Figure 17 shows real-world data from the experimental site where a wheelchair and robot were controlled while recognizing their surroundings. Figure 18 also shows real-world data from the experimental site where a wheelchair and robot were controlled while recognizing their surroundings. In the examples of Figures 17 and 18, WHILL's personal mobility device "WHILL model CR" and Temi USA Inc.'s personal robot "Temi" are controlled as agents. In the examples of Figures 17 and 18, a spatial AI automatically guides a person in a wheelchair and a person who has remotely logged into the robot to a reserved seat in an unmanned restaurant. In the examples of Figures 17 and 18, when a person or obstacle enters the wheelchair's path, the simulation unit 311 sends command information to the wheelchair indicating that it should slow down or stop. Also, if the situation does not change within a few seconds, the simulation unit 311 sends command information to the wheelchair indicating that it should change its course. The delay required for obstacle detection was 0.3 seconds, and the control delay was within 0.1 seconds.

[0103] Figure 19 shows data in the real world where the agent was controlled based on human movement. Figure 20 shows data in the virtual world where the agent was controlled based on human movement. The spatial output unit 310 determined that the human waved their hand based on skeletal information. The simulation unit 311 of the game engine device 3 sent command information to the personal mobility device indicating an instruction to move the personal mobility device, which acts as an agent, to the location of that human.

[0104] As described above, this disclosure provides a technology that can output a virtual space based on real-space information indicating the structure of the real space and information sensed from the real space, or a technology that enables the system receiving the sensing information to utilize the sensing information, regardless of the method of acquiring the sensing information. The former allows for the representation of a virtual space by adding elements of sensing information to the structure of the real space, thus providing a technology that can contribute to the development of industries in various fields such as medicine, chemistry, engineering, architecture, agriculture, IT, finance, entertainment, education, and tourism. The latter allows the system receiving the sensing information to utilize the sensing information, regardless of the method of acquiring the sensing information, thus leading to the omission of design work to ensure compatibility or the adoption of inexpensive sensors, thereby mitigating difficulties in various research, development, and business activities.

[0105] [others] Regarding the information processing system 1 according to the embodiment described above, the computer may be a program that causes the computer to function as the processor 31 of the information processing system 1. Alternatively, it may be an information processing method executed by the information processing system 1. This information processing method includes processes executed by the processor of the information processing system 1. With such a configuration, the agent can be operated autonomously.

[0106] In this embodiment, the server device 2 and the game engine device 3 were described as separate devices, but in a modified example, the server device 2 and the game engine device 3 may be the same device.

[0107] Server device 2 may be on-premises or in a cloud environment. In the case of a cloud-based server device 2, for example, it may provide the above functions and processing in the form of SaaS (Software as a Service) or cloud computing. The same applies to game engine device 3.

[0108] In the above embodiment, the server device 2 performed various storage and control functions, but multiple external devices may be used instead of the server device 2. That is, various information and programs may be stored in a distributed manner across multiple external devices using blockchain technology or the like. The same applies to the game engine device 3.

[0109] Furthermore, the following embodiments may also be provided.

[0110] (1) An information processing system comprising at least one processor that performs each of the following steps: in a spatial reception step, it receives real space information, the real space information being information indicating the structure of real space; in a reception step, it receives sensing information, the sensing information being information obtained by sensing the real space; and in a spatial output step, it outputs virtual space information in real time, the virtual space information being information indicating a virtual space in which the sensing results indicated by the sensing information are reflected in the structure of real space indicated by the real space information.

[0111] With this configuration, a virtual space can be output based on information indicating the structure of the real space and information sensed from the real space.

[0112] (2) An information processing system as described in (1) above, wherein one or more processors further perform a simulation step, in which the simulation step generates simulation information from the sensing information acquired in a time series, and the simulation information is information obtained by simulating the prediction of changes in the real space in a virtual space.

[0113] With this configuration, simulations of real-world space can be performed in a virtual space.

[0114] (3) An information processing system as described in (2) above, wherein in the spatial output step, the simulated information is transmitted to an agent, and the agent acts based on the simulated information.

[0115] With this configuration, the agent can operate autonomously based on simulated information.

[0116] (4) An information processing system as described in (3) above, wherein the sensing information includes skeletal information, the skeletal information is information relating to the skeleton of a human being present in the real space, and the simulation information includes information obtained by simulating the movement of the human being in the real space in a virtual space.

[0117] With this configuration, it is possible to predict the movements of humans in real space.

[0118] (5) An information processing system according to any one of the above (4), wherein the agent includes a virtual agent that exists only in the virtual space, and the virtual agent operates in coordination with the movements of the human based on the simulated information.

[0119] With this configuration, virtual agents can be operated in cooperation with humans present in the real world.

[0120] (6) An information processing system according to any one of (3) to (5) above, wherein the sensing information includes temperature map information indicating the temperature corresponding to each location in the real space, the simulation information includes information obtained by simulating in a virtual space the prediction of the temperature change in the real space based on the temperature map information acquired in a time series, and the agent includes a robot that operates based on the simulation information.

[0121] With this configuration, it is possible to predict temperature changes based on temperature information corresponding to each location in real space, and then operate the robot based on those predictions.

[0122] (7) An information processing system according to any one of (3) to (6) above, wherein the sensing information includes weight information indicating the weight of luggage present in the real space, the agent includes a robot whose transportable luggage weight is predetermined, and the robot transports one or more transportable luggage based on the weight information.

[0123] With this configuration, it is possible to operate a robot capable of transporting an object based on information about the weight of that object in real space.

[0124] (8) An information processing system according to any one of (3) to (7) above, wherein the sensing information includes density information indicating the density of people in each section within the real space, the simulation information includes information obtained in a time series, which is simulated in a virtual space to predict the density of people in the real space, the agent includes a drone capable of generating an alarm, and the drone moves to a location where the density of people is estimated to be high based on the simulation information and generates an alarm.

[0125] With this configuration, changes in human density can be predicted based on information about the density of people in each section of the real world, and based on the results, an alarm can be triggered on the drone.

[0126] (9) An information processing system according to any one of (3) to (8) above, wherein the sensing information includes luggage classification information and luggage location information, the luggage classification information includes information indicating the classification of luggage, the luggage location information includes information indicating the location of luggage, the agent transports the luggage to a predetermined location based on the luggage classification information and the luggage location information, and the simulation information includes information simulating the remaining luggage capacity at the predetermined location based on the luggage classification information and the luggage location information acquired in a time series.

[0127] With this configuration, it is possible to simulate the remaining capacity of luggage at a luggage collection point based on information about the classification and location of luggage in real space.

[0128] (10) An information processing system according to any one of (1) to (9) above, wherein in the spatial output step, the virtual spatial information corresponding to the real spatial information is output in real time using a trained model that has been trained to generate virtual spatial information from real spatial information.

[0129] With this configuration, it is possible to output virtual space information with high accuracy using artificial intelligence.

[0130] (11) An information processing system according to any one of (1) to (10) above, wherein in the spatial output step, a game engine is used to output the virtual space information.

[0131] With this configuration, virtual space information can be output using the rendering engine, physics engine, animation, lighting effects, and other functions provided by the game engine.

[0132] (12) A program that causes a computer to function as the processor of any one of the information processing systems described in (1) to (11) above.

[0133] With this configuration, a virtual space can be output based on information indicating the structure of the real space and information sensed from the real space.

[0134] (13) An information processing method to be performed by an information processing system, comprising each process performed by the processor of the information processing system described in any one of (1) to (11) above.

[0135] With this configuration, a virtual space can be output based on information indicating the structure of the real space and information sensed from the real space. Of course, this is not always the case.

[0136] It may also be provided in the following forms. (1) An information processing system comprising at least one processor that performs the following steps, wherein in the receiving step, sensing information is information obtained by sensing the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by the system to which the sensing information is output, according to the data attributes indicated by the sensing information.

[0137] With this configuration, the system receiving the sensing information can utilize the sensing information regardless of the method used to acquire it.

[0138] (2) An information processing system as described in (1) above, wherein in the standardization step, the data format of the sensing information is converted to a common data format when the semantics of the sensing information data are the same.

[0139] With this configuration, if the semantics of the data indicated by the sensing information are the same, the system to which the sensing information is output can utilize the sensing information, regardless of the method used to acquire the sensing information.

[0140] (3) An information processing system as described in (1) or (2) above, wherein one or more processors further perform a spatial output step, in which the spatial output step outputs virtual space information to a system that is the output destination of the sensing information, and the virtual space information is information that indicates a virtual space in which the results of the sensing are reflected in the real space.

[0141] With this configuration, agents can be made to operate autonomously based on virtual space information.

[0142] (4) An information processing system as described in (3) above, wherein in the spatial output step, the frequency at which the virtual space information is output to the agent is controlled according to the service provided by the agent.

[0143] With this configuration, the frequency of the agent's operations can be controlled depending on the services the agent provides.

[0144] (5) An information processing system as described in (3) or (4) above, wherein in the spatial output step, the information processing system controls the resolution of the virtual space indicated by the virtual space information according to the service provided by the agent.

[0145] With this configuration, the resolution of the virtual space can be controlled according to the services provided by the agent, allowing for efficient provision of virtual space information to the agent.

[0146] (6) An information processing system according to any one of (1) to (5) above, wherein the sensing information includes data indicating the structure of the real space, and in the commonization step, the sensing information is converted into mesh-format data indicating the structure of the real space.

[0147] With this configuration, if the semantics of the data indicated by the sensing information represent the structure of real space, the system that receives the sensing information can utilize the sensing information regardless of the type of sensor.

[0148] (7) An information processing system according to any one of (1) to (6) above, wherein the sensing information includes information indicating the temperature corresponding to each location in the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by the system to which the sensing information is output, as information indicating the temperature corresponding to each location in the real space.

[0149] With this configuration, if the semantics of the data indicated by the sensing information represent the temperature corresponding to each location in real space, the system receiving the sensing information can utilize the sensing information regardless of the type of sensor.

[0150] (8) An information processing system according to any one of (1) to (7) above, wherein the sensing information includes information indicating the weight of luggage present in the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by the system to which the sensing information is output, as information indicating the weight of luggage present in the real space.

[0151] With this configuration, if the semantics of the data indicated by the sensing information represent the weight of an object present in real space, the system receiving the sensing information can utilize the sensing information regardless of the type of sensor.

[0152] (9) An information processing system according to any one of (1) to (8) above, wherein the sensing information includes information indicating the density of people in each section within the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by the system to which the sensing information is output, as information indicating the density of people in each section within the real space.

[0153] With this configuration, if the semantics of the data indicated by the sensing information represent the density of people in each section of real space, the system receiving the sensing information can utilize the sensing information regardless of the type of sensor.

[0154] (10) An information processing system according to any one of (1) to (9) above, wherein the sensing information includes information indicating the classification of the luggage and information indicating the location of the luggage, and in the commonization step, the sensing information is converted into a data format that can be interpreted by the system to which the sensing information is output, as the information indicating the classification of the luggage and the information indicating the location of the luggage.

[0155] With this configuration, if the semantics of the data indicated by the sensing information represent the classification and location of the luggage, the system receiving the sensing information can utilize the sensing information regardless of the type of sensor.

[0156] (11) An information processing system described in any one of (1) to (10) above, wherein the output destination system is a system that uses a game engine as software.

[0157] With this configuration, the system using the game engine can utilize the sensing information, regardless of the method used to acquire it.

[0158] (12) A program that causes a computer to function as the processor of any one of the information processing systems described in (1) to (11) above.

[0159] With this configuration, the system receiving the sensing information can utilize the sensing information regardless of the type of sensor.

[0160] (13) An information processing method to be performed by an information processing system, comprising each process performed by the processor of the information processing system described in any one of (1) to (11) above.

[0161] With this configuration, the system receiving the sensing information can utilize the sensing information regardless of the type of sensor.

[0162] Finally, while various embodiments of the present invention have been described, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of symbols]

[0163] 1: Information Processing System 2: Server device 21: Processor 210: Spatial Reception Department 211:Commonization department 212: Information transmission and reception unit 22: Storage section 23: Communications Department 3: Game engine device 31: Processor 310: Spatial output section 311: Simulation Unit 312: Information transmission and reception unit 32: Storage section 33: Communications Department 4: Agent device 41: Processor 42: Storage section 43: Communications Department 44: Sensing Department 45: Drive unit 5a: Camera 51a: Processor 52a: Storage section 53a: Communications Department 54a: Imaging Unit 5b: Sensor 51b: Processor 52b: Storage section 53b: Communications Department 54b: Sensing Unit 5c: Lighting 51c: Processor 52c: Storage section 53c:Communication Department 54c: Drive unit 5d: Smartphone 51d: Processor 52d: Storage section 53d: Communications Department 54d: Input section 55d: Output section 6: Human 7: Object N: Network

Claims

1. An information processing system, The system comprises at least one processor that performs each of the following steps: In the spatial reception step, real-world spatial information is received. The aforementioned real-world spatial information is information that shows the structure of real space, In the receiving step, sensing information is received. The sensing information includes skeletal information, where the skeletal information is information relating to the skeleton of a human being present in the real space. In the simulation step, simulation information is generated by simulating the prediction of the human movement in the real space in a virtual space from the sensing information acquired in a time series. In the spatial output step, virtual space information is output in real time. The virtual space information is information that shows a virtual space in which the sensing results shown in the sensing information are reflected in the structure of the real space shown in the real space information. The virtual agent, which exists only in the aforementioned virtual space, operates in coordination with the movements of the human based on the simulation information. Information processing system.

2. In the information processing system described in claim 1, In the spatial output step, the simulation information is sent to the agent. The agent acts based on the simulated information. Information processing system.

3. In the information processing system described in claim 2, The sensing information includes temperature map information indicating the temperature corresponding to each location in the real space, The simulation information includes information obtained by simulating in a virtual space the prediction of temperature changes in the real space based on temperature map information acquired in a time series. The agent includes a robot that operates based on the simulation information, Information processing system.

4. In the information processing system described in claim 2, The sensing information includes weight information indicating the weight of the luggage present in the real space. The agent includes a robot whose load capacity is predetermined, The robot transports one or more loads that can be carried based on the weight information. Information processing system.

5. In the information processing system described in claim 2, The sensing information includes density information indicating the density of people in each section within the real space. The simulation information includes information obtained by simulating in a virtual space the prediction of the density of humans in the real space based on density information acquired in a time series. The agent includes a drone capable of generating an alarm, The drone moves to a location where the human density is estimated to be high based on the simulated information and generates an alarm. Information processing system.

6. In the information processing system described in claim 2, The sensing information includes luggage classification information and luggage location information. The aforementioned luggage classification information includes information indicating the classification of the luggage, The aforementioned luggage location information includes information indicating the location of the luggage, The agent transports the luggage to a predetermined location based on the luggage classification information and the luggage location information. The simulation information includes information that simulates the remaining allowable amount of luggage at the predetermined location, based on the luggage classification information and luggage location information acquired in a time series. Information processing system.

7. In the information processing system described in claim 1, In the spatial output step, a trained model that has been trained to generate virtual spatial information from real spatial information is used to output the virtual spatial information corresponding to the real spatial information in real time. Information processing system.

8. In the information processing system described in claim 1, In the spatial output step, the game engine is used to output the virtual space information. Information processing system.

9. It is a program, The computer is made to function as the processor of the information processing system according to any one of claims 1 to 8. program.

10. An information processing method performed by an information processing system, The information processing system according to any one of claims 1 to 8 comprises each process performed by the processor, Information processing methods.

Images