Information processing system, program, and information processing method

Abstract

To provide a technique capable of outputting a virtual space on the basis of information obtained by sensing real space information indicating a structure of a real space and the real space, or a technique capable of allowing a system as an output destination of the sensing information to activate the sensing information, without depending on the kind of a method for acquiring the sensing information.SOLUTION: An information processing system includes at least one or more processors executing the following steps. A space reception step receives real space information. The real space information is information indicating the structure of the real space. The reception step receives sensing information. The sensing information is information obtained by sensing the real space. A space output step outputs virtual space information at real time. The virtual space information is information indicating a virtual space obtained by reflecting the result of sensing indicated by the sensing information, on the structure of the real space indicated by the real space information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a program, and an information processing method. [Background technology]

[0002] Conventionally, a known technology for a digital twin analysis device includes a data receiving unit that receives input of data to be analyzed, an evaluation index storage unit that stores evaluation indexes for evaluating the accuracy of the data received by the data receiving unit, a data evaluation unit that evaluates the accuracy of the received data based on the evaluation indexes stored in the evaluation index storage unit, a data analysis unit that performs analysis using data selected based on the accuracy evaluated by the data evaluation unit, and an output unit that outputs the results of analysis by the data analysis means (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-183887 A Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for technology to output a virtual space based on real space information that indicates the structure of the real space and information sensed from the real space, or technology that enables the system to which the sensing information is output to utilize the sensing information regardless of the type of method used to acquire the sensing information.

[0005] In consideration of the above circumstances, the present invention provides technology that can output a virtual space based on real space information that indicates the structure of the real space and information sensed from the real space, or technology that enables the system to which the sensing information is output to utilize the sensing information regardless of the type of method used to acquire the sensing information. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an information processing system. The information processing system includes at least one processor that executes the following steps: In the space receiving step, real space information is received. The real space information is information indicating the structure of the real space. In the receiving step, sensing information is received. The sensing information is information obtained by sensing the real space. In the space output step, virtual space information is output in real time. The virtual space information is information indicating a virtual space in which the sensing result indicated by the sensing information is reflected in the structure of the real space indicated by the real space information.

[0007] According to the present disclosure, a virtual space can be output based on real space information indicating the structure of the real space and information sensed from the real space, or the system to which the sensing information is output can utilize the sensing information regardless of the type of method used to acquire the sensing information. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of a system configuration of an information processing system 1. [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of a server device 2. FIG. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of a game engine device 3. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an agent device 4. [Diagram 5] FIG. 2 is a diagram illustrating an example of a hardware configuration of a camera 5a. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a sensor 5b. [Figure 7] FIG. 13 is a diagram illustrating an example of a hardware configuration of a lighting device 5c. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of a smartphone 5d. [Figure 9] 2 is an example of a block diagram showing functions realized by a processor 21 of the server device 2. FIG. [Figure 10] 2 is an example of a block diagram showing functions realized by a processor 31 of the game engine device 3. FIG. [Figure 11] FIG. 1 illustrates an example of an SSCP architecture. [Figure 12] FIG. 1 is a diagram showing an example of a sequence diagram for explaining information processing according to an embodiment. [Figure 13] The data obtained from the IoT equipment installed in the test site is shown below. [Figure 14] The data obtained from the IoT equipment installed in the test site is shown below. [Figure 15] The data shows the flow of people in the experimental site. [Figure 16] We present an example in which a wheelchair was controlled while recognizing its surroundings in an experimental field. [Figure 17] This shows data on the real space in which a wheelchair and a robot were controlled while recognizing the surrounding situation in the experimental field. [Figure 18] This shows data on the real space in which a wheelchair and a robot were controlled while recognizing the surrounding situation in the experimental field. [Figure 19] FIG. 1 illustrates data of a real space in which an agent is controlled based on human movements. [Figure 20] FIG. 13 is a diagram showing data of a virtual space in which an agent is controlled based on human movements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [Embodiment] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. Various characteristic features shown in the following embodiments can be combined with each other.

[0010] 1. Definitions First, the definitions of each term will be explained.

[0011] The program for realizing the software appearing in this embodiment may be provided as a non-transitory computer-readable recording medium, or may be provided so as to be downloadable from an external server, or may be provided so that the program is launched on an external computer and its functions are realized on a client device (so-called cloud computing).

[0012] In addition, in this embodiment, the term "unit" may include, for example, a combination of hardware resources implemented by a circuit in the broad sense and software information processing that can be specifically realized by these hardware resources. In addition, in this embodiment, various information is handled, and this information is represented, for example, by physical values ​​of signal values ​​representing voltage and current, high and low signal values ​​as a binary bit collection consisting of 0 or 1, or quantum superposition (so-called quantum bits), and communication and calculation can be performed on the circuit in the broad sense.

[0013] In addition, a circuit in the broad sense is a circuit realized by at least appropriately combining a circuit, circuitry, a processor, a memory, etc. In other words, it includes an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)), etc.

[0014] This disclosure discloses an architecture called "SSCP." In SSCP (Spatial Simulation-based Cyber-Physical Model), humans, agents, real space, etc. are controlled through a virtual space. An example of the SSCP architecture will be described later with reference to FIG. 12.

[0015] A "virtual space" is an area defined by information processing. A virtual space has the properties of data and does not have a physical form. In this embodiment, the virtual space is assumed to be a space defined in three dimensions, but may include a space defined in one or two dimensions. The virtual space may include a centrally managed space, a space managed as a shared property (a so-called metaverse), and the like. In this embodiment, the virtual space is represented by a digital twin.

[0016] A "digital twin" is a technology that expresses a virtual space synchronized with the real space in time based on real space information collected in real time and past real space information. The virtual space expressed by the digital twin is constructed on the premise that the spatial data matches the real physical space and necessary changes in the space are adjusted through sensors. The world of the digital twin may be constructed manually by a developer using BIM or the like, or may be constructed from images acquired by a camera or data acquired by LiDAR (Light Detection And Ranging) if constructed with a purpose in mind. In addition, it can be constructed so that the object itself provides information through IoT devices such as the camera 5a and the sensor 5b described later. In this embodiment, the digital twin has real-time properties, multi-agent orientation, multi-site symmetry, and spatiotemporal synchronization.

[0017] "Real-time" refers to the rendering of a virtual space as a dynamic space with a time difference with the current real space as close to zero as possible, and at a high frequency (for example, in units of 1 / 100 seconds). In other words, by having real-time properties, the virtual space is rendered in real time.

[0018] "Multi-agent oriented" refers to the reproduction of a virtual space by a digital twin, assuming the presence of multiple participants in the virtual space, which are multiple agents. While scanning of a small number of people is used in general games and motion capture, multi-agent oriented does not assume a system for a small number of people.

[0019] "Multi-site symmetry" means that the virtual space reproduced by the digital twin is not described in a different manner depending on the viewpoint of the participants. A virtual space with multi-site symmetry is described without depending on the location of the participants and the viewpoint of the participants. In other words, a virtual space with multi-site symmetry is not expected to be described in one direction from one location.

[0020] "Time-space synchrony" means that the virtual space reproduced by the digital twin is not described differently by each participant. In a virtual space with time-space synchrony, multiple participants share the same virtual space.

[0021] "Real space" is a space where various objects exist. Real space is a concept that contrasts with virtual space, where no objects exist.

[0022] A "common ground" is a platform that allows multiple agents to share information with each other. The common ground provides a common understanding to minimize misunderstandings between participants for information exchange, joint work, cooperative actions, and the like. In the common ground, data that has the same semantics but different data formats can be commonized and treated as data of the same format. Furthermore, in the common ground, data with similar semantics may also be commonized in a data format and treated as data of the same format. In this embodiment, the spatial receiving unit 210 and the commonization unit 211 function as a common ground.

[0023] "Semantics" refers to the meaning, intent, or content of data. For example, even if two pieces of data have the same semantics, the two pieces of data may be described in different formats or structures.

[0024] 2. System configuration of information processing system 1 Next, the system configuration of the information processing system 1 of this embodiment will be described with reference to FIG.

[0025] FIG. 1 is a diagram showing an example of a system configuration of an information processing system 1. As shown in FIG. 1, the information processing system 1 includes a server device 2, a game engine device 3, an agent device 4, IoT devices (camera 5a, sensor 5b, light 5c, smartphone 5d), a person 6, an object 7, and a network N. The server device 2, the game engine device 3, the agent device 4, and the IoT devices (camera 5a, sensor 5b, light 5c, smartphone 5d) are configured to be able to communicate with each other via the network N. As a result, the server device 2, the game engine device 3, the agent device 4, and the IoT devices (camera 5a, sensor 5b, light 5c, smartphone 5d) can transmit or receive various information to each other. Note that there may be a plurality of server devices 2, game engine devices 3, agent devices 4, IoT devices (camera 5a, sensor 5b, light 5c, smartphone 5d), people 6, and objects 7. Here, the system exemplified by the information processing system 1 is composed of one or more devices or components. Therefore, even the server device 2 alone, the game engine device 3 alone, the agent device 4 alone, or an IoT device (camera 5a, sensor 5b, lighting 5c, smartphone 5d) alone are included in the system exemplified by the information processing system 1.

[0026] 3. Hardware Configuration Next, the hardware configurations of the server device 2, the game engine device 3, the agent device 4, and the IoT devices (camera 5a, sensor 5b, lighting 5c, and smartphone 5d) of this embodiment will be described with reference to FIGS.

[0027] 3.1. Hardware configuration of server device 2 Fig. 2 is a diagram showing an example of a hardware configuration of the server device 2. As shown in Fig. 2, the server device 2 includes a processor 21, a storage unit 22, and a communication unit 23, and these components are electrically connected via a communication bus inside the server device 2. The server device 2 executes the process according to the embodiment.

[0028] The processor 21 processes and controls the overall operations related to the server device 2. The processor 21 is, for example, a central processing unit (CPU). The processor 21 reads out a predetermined program stored in the storage unit 22 and executes processing based on the program, thereby realizing various functions related to the server device 2, for example, the processing shown in FIG. 12 described later. The processor 21 is not limited to being single, and may be implemented with multiple processors 21 for each function. Also, a combination of these may be used.

[0029] The storage unit 22 stores various information defined by the above description. This can be implemented, for example, as a storage device such as a solid state drive (SSD) that stores various programs and the like related to the server device 2 executed by the processor 21, or as a memory such as a random access memory (RAM) that stores temporarily required information (arguments, arrays, etc.) related to the calculation of the programs. The storage unit 22 stores various programs and variables related to the server device 2 executed by the processor 21, and data and the like used when the processor 21 executes processing based on the programs. The storage unit 22 is an example of a storage medium.

[0030] The communication unit 23 is preferably a wired communication means such as USB, IEEE1394, Thunderbolt (registered trademark), wired LAN network communication, etc., but may also include wireless LAN network communication, mobile communication such as LTE / 3G / 4G / 5G, BLUETOOTH (registered trademark) communication, etc. as necessary. In other words, it is more preferable to implement it as a collection of multiple communication means. In other words, the server device 2 may communicate various information from the outside via the communication unit 23.

[0031] 3.2. Hardware Configuration of Game Engine Device 3 Next, the game engine device 3 will be described with reference to FIG. 3. The game engine device 3 is hardware equipped with a game engine, which is software. The game engine is used as a development environment for creating games. The game engine has a feature that character data, terrain data, etc. are arranged in a virtual space, the virtual space is drawn on a screen, and the virtual space can be operated by data specific to the game. Any software such as Unity3D, Unreal Engine, GameMaker, Godot, Cocos2d-x, PhyreEngine, etc. may be used as the game engine. Furthermore, by using the game engine, the information processing of this embodiment can be executed in a real-time state.

[0032] Fig. 3 is a diagram showing an example of a hardware configuration of the game engine device 3. As shown in Fig. 3, the game engine device 3 has a processor 31, a storage unit 32, and a communication unit 33, and these components are electrically connected via a communication bus inside the game engine device 3. The game engine device 3 executes the processing according to the embodiment. For the processor 31, storage unit 32, and communication unit 33 of the game engine device 3, please refer to the processor 21, storage unit 22, and communication unit 23 of the server device 2.

[0033] 3.3.Hardware Configuration of Agent Device 4 Next, the agent device 4 will be described with reference to FIG. 4. The agent device 4 is an autonomous agent other than humans 6 and animals, and is a general term for autonomously moving objects, robots, etc. The agent device 4 is a machine that can operate independently or as part of a vehicle management system. The agent device 4 is developed based on a robot system such as ROS (Robot Operating System) or Autoware. The agent device 4 can provide its own position coordinates and sensing information acquired by itself to the server device 2.

[0034] Fig. 4 is a diagram showing an example of a hardware configuration of the agent device 4. As shown in Fig. 4, the agent device 4 has a processor 41, a storage unit 42, a communication unit 43, a sensing unit 44, and a drive unit 45, and these components are electrically connected via a communication bus inside the agent device 4. The agent device 4 executes the process according to the embodiment. For the processor 41, the storage unit 42, and the communication unit 43 of the agent device 4, please refer to the processor 21, the storage unit 22, and the communication unit 23 of the server device 2.

[0035] The sensing portion 44 operates based on certain physical or chemical principles to convert a detected input into an appropriate output signal, which is often manifested as a change in an electrical property such as resistance, capacitance, voltage, or current.

[0036] The actuator 45 is an actuator that converts a control signal into physical action or movement. The actuator 45 typically receives an electrical, thermal, or pressure signal and converts it into mechanical action.

[0037] 3.4. IoT device hardware configuration Next, IoT devices will be described with reference to Figs. 5 to 8. The IoT devices are devices that are connected to a network N and can communicate with a server device 2 or a game engine device 3. In the example of the information processing system 1 in Fig. 1, the IoT devices include a camera 5a, a sensor 5b, a light 5c, a smartphone 5d, and the like. Since a wide variety of IoT devices can exist, the hardware configuration of the IoT devices shown below is merely an example, and there may be variations corresponding to the type of device.

[0038] The camera 5a captures real space to obtain image data. The image data is used as real space information or sensing information. FIG. 5 is a diagram showing an example of a hardware configuration of the camera 5a. As shown in FIG. 5, the camera 5a has a processor 51a, a storage unit 52a, a communication unit 53a, and an imaging unit 54a, and these components are electrically connected via a communication bus inside the camera 5a. The camera 5a executes the process according to the embodiment. For the processor 51a, the storage unit 52a, and the communication unit 53a of the camera 5a, please refer to the processor 21, the storage unit 22, and the communication unit 23 of the server device 2.

[0039] The imaging unit 54a is equipped with an optical lens inside, which collects light from a subject located in the imaging direction of the camera 5a. The collected light is guided to an imaging element disposed inside the body of the camera 5a. The imaging element is configured to convert the collected light into an electronic signal, which is processed and generated as image data.

[0040] The sensor 5b acquires sensing information by sensing the real space. FIG. 6 is a diagram showing an example of the hardware configuration of the sensor 5b. As shown in FIG. 6, the sensor 5b has a processor 51b, a storage unit 52b, a communication unit 53b, and a sensing unit 54b, and these components are electrically connected inside the sensor 5b via a communication bus. The sensor 5b executes the process according to the embodiment. For the processor 51b, the storage unit 52b, and the communication unit 53b of the sensor 5b, please refer to the processor 21, the storage unit 22, and the communication unit 23 of the server device 2. As an IoT device having a hardware configuration similar to that of the sensor 5b, sensors such as LiDAR, ToF sensor, thermo sensor, weight sensor, and microphone are used. In addition to the sensor 5b itself capable of sensing from the real space, the sensor as an IoT device may include a device that collects data acquired by the sensor 5b. For the sensing unit 54b of the sensor 5b, please refer to the sensing unit 44 of the agent device 4.

[0041] The lighting 5c ​​is an example of an IoT device that operates to illuminate a real space. FIG. 7 is a diagram showing an example of a hardware configuration of the lighting 5c. As shown in FIG. 7, the lighting 5c ​​has a processor 51c, a storage unit 52c, a communication unit 53c, and a driving unit 54c, and these components are electrically connected via a communication bus inside the lighting 5c. The lighting 5c ​​executes the process according to the embodiment. For the processor 51c, the storage unit 52c, and the communication unit 53c of the lighting 5c, please refer to the processor 21, the storage unit 22, and the communication unit 23 of the server device 2. As IoT devices having a hardware configuration similar to that of the lighting 5c, devices such as an automatic door, an elevator, a water heater, and an air conditioner are used. For the drive unit 54c of the lighting 5c, please refer to the drive unit 45 of the agent device 4.

[0042] The smartphone 5d is an example of an IoT device having a display in real space. FIG. 8 is a diagram showing an example of a hardware configuration of the smartphone 5d. As shown in FIG. 7, the smartphone 5d has a processor 51d, a storage unit 52d, a communication unit 53d, an input unit 54d, and an output unit 55d, and these components are electrically connected via a communication bus inside the smartphone 5d. The smartphone 5d executes the process according to the embodiment. For the processor 51d, the storage unit 52d, and the communication unit 53d of the smartphone 5d, please refer to the processor 21, the storage unit 22, and the communication unit 23 of the server device 2. As an IoT device having a hardware configuration similar to that of the smartphone 5d, a device such as a speaker, a screen, a playback device, or a head-mounted display may be used.

[0043] The input unit 54d may be included in the housing of the smartphone 5d or may be externally attached. For example, the input unit 54d may be implemented as a touch panel integrated with the output unit 55d. If it is a touch panel, the user can input a tap operation, a swipe operation, or the like. Of course, a switch button, a mouse, a QWERTY keyboard, or the like may be adopted instead of the touch panel. That is, the input unit 54d accepts an input based on an operation performed by the user. The input is transferred as a command signal to the processor 51d via a communication bus, and the processor 51d can execute a predetermined control or calculation as necessary.

[0044] The output unit 55d can function as a display unit of the smartphone 5d. The output unit 55d may be included in the housing of the smartphone 5d, or may be externally attached. The output unit 55d displays a screen of a graphical user interface (GUI) that can be operated by a user. This is preferably performed by using display devices such as a CRT display, a liquid crystal display, an organic EL display, and a plasma display according to the type of the smartphone 5d.

[0045] The human 6 represents an individual present in the environment. The human 6 operates to report its own state, position, etc. to the server device 2 using the smartphone 5d. In addition, the state, position, and skeletal information of the human 6 are reported to the server device 2 using various methods such as RFID and motion capture.

[0046] The object 7 includes any real object such as a desk, a chair, a car, a plant, a building, luggage, a bench, a traffic light, etc. The range of the object 7 may include a wall, a floor, a river, an ocean, etc. The range of the object 7 may also include the above-mentioned IoT device, a human 6, etc.

[0047] 4. Functional configuration In this section, the functional configuration of this embodiment will be described with reference to Fig. 9 to Fig. 10. The information processing system 1 includes at least one processor 21, 31 that executes each of the functional units shown in Figs.

[0048] 4.1. Functional configuration of server device 2 9 is an example of a block diagram showing functions realized by the processor 21 of the server device 2. The processor 21 of the server device 2, which is an example of the information processing system 1, has functions as a spatial reception unit 210, a common unit 211, and an information transmission / reception unit 212. As described above, information processing by software stored in the storage unit 22 is specifically realized by the processor 21, which is an example of hardware, and can be executed as each functional unit included in the processor 21.

[0049] The space receiving unit 210 is configured to receive, receive, or acquire real space information from other devices via the network N and the communication unit 23. The space receiving unit 210 is also configured to transmit or output the real space information to other devices via the communication unit 23 and the network N. Furthermore, the space receiving unit 210 as a common ground converts the real space information into data in a mesh format (such as vertices, indexes, and materials) so that it can be used in a game engine. The space receiving unit 210 executes a space receiving step.

[0050] The real space information is information that indicates the state of the real space. The real space information includes information that expresses spaces such as a site, a building, and a room. The real space information may be information based on information modeled using methods such as BIM (Building Information Modeling) and 3DCAD (3D Computer Aided Design). When the real space information is based on BIM, 3DCAD, etc., the real space information needs to model the connections between the spaces, the information contained in each space, the configuration of each space, etc. The real space information may be information based on information acquired from a platform such as PLATEAU, GIS (Geographic Information System), and a three-dimensional spatial information platform (Spatial ID). The real space information may be information based on point cloud data. When the real space information is based on point cloud data, the real space information may be converted or compressed as surface information as a surface model, thereby reducing data and allocating attribute data.

[0051] The real space information includes information on various objects 7 such as walls, floors, equipment, desks, chairs, robots, etc. The real space information has information on changes in position, status, shape, etc. of various objects 7 in the real space. The objects 7 may be modeled in an abstracted form as "points" and all managed through a Building. Automation and Control System (BACS) that uses a standard protocol such as the Building Automation and Control Networking Protocol (BACnet). These objects may also be processed using a protocol such as Message Queuing Telemetry Transport (MQTT).

[0052] The commonization unit 211 converts the sensing information into a data format that can be interpreted by a system to which the sensing information is output, according to the attribute of the data indicated by the sensing information. The commonization unit 211 functions as a common ground. The commonization unit 211 executes a commonization step.

[0053] The information transmitting / receiving unit 212 is configured to accept, receive, or acquire sensing information from another device via the network N and the communication unit 23. The information transmitting / receiving unit 212 is also configured to transmit or output the sensing information to another device via the network N and the communication unit 23. The information transmitting / receiving unit 212 executes an information transmitting / receiving step or a receiving step.

[0054] The sensing information is information obtained by sensing or observing a real space with a device such as a sensor or a camera. The sensing information may include information indicating temperature, humidity, pressure, vibration, acceleration, position, direction, speed, gas concentration, pH, light intensity, light color, distance, image, sound, frequency, magnetic field, electric field, stress, strain, weight, radiation, etc. The sensing information may include information defined by combining two or more pieces of information listed above (for example, temperature map information by combining position and temperature).

[0055] The spatial reception unit 210, the common unit 211, and the information transmission / reception unit 212 will be described in detail later.

[0056] 4.2. Functional configuration of the game engine device 3 10 is an example of a block diagram showing functions realized by the processor 31 of the game engine device 3. The processor 31 of the game engine device 3, which is an example of the information processing system 1, includes a spatial output unit 310, a simulation unit 311, and an information transmission / reception unit 312. As described above, information processing by software stored in the storage unit 32 is specifically realized by the processor 31, which is an example of hardware, and can be executed as each functional unit included in the processor 31.

[0057] The space output unit 310 outputs virtual space information corresponding to the real space information in real time. The space output unit 310 executes a space output step.

[0058] The virtual space information is information indicating a virtual space expressed by a digital twin. That is, the virtual space information is information indicating a virtual space corresponding to a real space. In this embodiment, the virtual space information is information indicating a virtual space in which the result of sensing indicated by the sensing information is reflected in the structure of the real space indicated by the real space information. In addition, since the virtual space information is output in real time, it is data indicating a virtual space that changes over time. In the embodiment, for example, the virtual space information includes information indicating what posture the human 6 is in, where the agent device 4, the human 6, and the object 7 are located, what positional relationship they are in, and how much acceleration, speed, and weight they have. In addition, in the embodiment, for example, the virtual space information may include information indicating the distribution in space of temperature, humidity, pressure, vibration, gas concentration, pH, light intensity, light color, sound, frequency, magnetic field, electric field, radiation, and the like.

[0059] The simulator 311 generates simulation information from real space information acquired in time series. The simulation information is information obtained by simulating a prediction of a change in the real space in a virtual space. The simulator 311 executes a simulation step.

[0060] The information transmitting / receiving unit 312 is configured to accept, receive, or acquire various pieces of information from other devices via the network N and the communication unit 33. The information transmitting / receiving unit 312 is also configured to transmit or output various pieces of information to other devices via the communication unit 33 and the network N. The information transmitting / receiving unit 312 executes an information transmitting / receiving step or an information receiving step.

[0061] The spatial output unit 310, the simulation unit 311, and the information transmission / reception unit 312 will be described in detail later.

[0062] 5. SSCP Architecture Next, the relationship between the above-mentioned system, hardware, and software and the SSCP architecture will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of an SSCP architecture. The SSCP in Fig. 11 includes at least a physical environment layer, an abstraction / storage / synchronization layer, a simulation layer, and a client application.

[0063] The physical environment layer includes real space information such as BIM and point cloud data, sensors (cameras 5a, sensors 5b), actuators (lighting 5c), robots (agent devices 4), humans 6, and audiovisuals (smartphones 5d).

[0064] The abstraction / storage / synchronization layer includes a model server, an actor network, an IoT platform, and a coordination infrastructure, and is realized by a server device 2.

[0065] The model server corresponds to the space receiving unit 210. The space receiving unit 210 stores information on geometry that exists in the target space, such as buildings, facilities, equipment, and robots, and provides it upon request. The space receiving unit 210 aggregates geometry information generated from space information such as BIM, point clouds, and photographs. The space receiving unit 210 converts the geometry information into information in a mesh format (such as vertices, indexes, and materials) so that it can be easily used in a game engine. Note that the geometry information and mesh format information are included in the real space information.

[0066] The actor network corresponds to the commonization unit 211. The commonization unit 211 models elements or actors existing in the physical environment as a network and can provide information upon request. All actors are linked to the IoT platform (information transmission / reception unit 212) and the real actor preparation infrastructure (information transmission / reception unit 212) to be compatible with a graph database or the web, and perform the necessary state transformations.

[0067] The IoT platform and the coordination infrastructure correspond to the information transmitting / receiving unit 212 . The IoT platform mainly collects, stores, and provides data related to humans 6 and objects 7 in the real space as time-series data. When collecting data, the IoT platform can use a gateway to perform appropriate data modeling and abstraction to improve machine readability. As the IoT platform, a time-series database may be used, or HDFS (Hadoop Distributed File System) may be used from the perspective of data storage efficiency or cost. The coordination infrastructure is a function for more frequent data exchange with IoT devices. In the coordination infrastructure, for example, data exchange by PUB / SUB using MQTT (Message Queuing Telemetry Transport) and Apache Kafka may be performed. Unlike the IoT platform, the coordination infrastructure does not assume long-term storage of data.

[0068] The simulation layer includes a game engine and a data analysis infrastructure, and is realized by a game engine device 3.

[0069] The game engine corresponds to a spatial output unit 310 , a simulation unit 311 , and an information transmission / reception unit 312 .

[0070] The data analysis infrastructure corresponds to the space output unit 310 and the simulator unit 311. The space output unit 310 and the simulator unit 311 have a function of performing various processes and modeling related to statistical analysis, machine learning (ML), AI, etc. in the game engine. For example, this function performs batch processing based on information from the actor network or the IoT platform (information transmission / reception unit 212), and converts the obtained model into an API to link with the game engine.

[0071] A client application is an application that uses data obtained from the abstraction / storage / synchronization layer or the simulation layer. A client application may be developed as a game engine or a web application, or as an application that supports the SSCP.

[0072] 6. Operational flow of information processing system 1 An example of a preferable information process executed by the information processing system 1 of this embodiment will be described.

[0073] In this specification, various information transmitted and received via the communication units 23, 33, 43, 53a, 53b, 53c, and 53d is stored in the memory unit of the device having the communication unit used, even if not explicitly stated. For example, when information is transmitted and received between the server device 2 and the game engine device 3, the communication units 23 and 33 are used, and therefore the information is stored in the memory units 22 and 32.

[0074] 6.1 Overview of Information Processing Next, an overview of the information processing will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of a sequence diagram for explaining the information processing according to the embodiment.

[0075] (Step S1) The space receiving unit 210 of the server device 2 transmits the real space information to the game engine device 3. The information transmitting / receiving unit 312 of the game engine device 3 receives the real space information from the server device 2. The space receiving unit 210 as the common ground converts the data format of the real space information into a format that can be used by the game engine, as necessary.

[0076] (Step S2) The IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d) acquire sensing information. The IoT devices transmit the sensing information to the server device 2 via the communication units 53a, 53b, 53c, and 53d and the network N. The information transmitting / receiving unit 212 acquires the sensing information from the IoT devices (camera 5a, sensor 5b, lighting 5c, smartphone 5d).

[0077] (Step S3) The common unit 211 of the server device 2 serving as a common ground converts the sensing information into a data format that can be interpreted by a system (game engine) to which the sensing information is output, according to the attributes of the data indicated by the sensing information. When the semantics of the data of the sensing information are the same, the common unit 211 converts the data format of the sensing information so that the data format is a common data format. With this configuration, when the semantics of the data indicated by the sensing information are the same, the system to which the sensing information is output can utilize the sensing information, regardless of the type of method for acquiring the sensing information.

[0078] The sensing information includes information that allows the posture of the human 6 to be recognized. For example, the sensing information includes skeletal information. The skeletal information is information about the skeleton of the human 6 that exists in real space. The skeletal information is information that indicates the positions of each part of the human 6, such as the joints, and the positional relationships between them, and is used to analyze posture and movement.

[0079] When a camera 5a, a ToF sensor, or the like is used as a sensor, the sensing information may include real space information. The commonization unit 211 aggregates geometry generated from sensing information such as point clouds and image data indicating the structure of real space, and converts the data into mesh format (including elements such as vertices, indexes, and materials) so that it can be used in a game engine or on the web. With this configuration, when the semantics of the data indicated by the sensing information indicates the structure of real space, the agent can utilize the sensing information regardless of the type of sensor.

[0080] (Step S4) The information transmitting / receiving unit 212 of the server device 2 transmits the sensing information to the game engine device 3. The information transmitting / receiving unit 312 of the game engine device 3 receives the sensing information from the server device 2.

[0081] (Step S5) The space output unit 310 outputs virtual space information corresponding to the real space information in real time based on the real space information indicated by the sensing information such as skeletal information. The space output unit 310 constructs a virtual space using the real space information acquired from the server device 2 and characteristic functions of the game engine such as smart objects and navigation mesh. With this configuration, it is possible to output virtual space information by utilizing functions of the game engine such as a rendering engine, a physics engine, animation, and lighting effects.

[0082] (Step S6) The simulator 311 of the game engine device 3 generates simulation information from the real space information and sensing information acquired in time series. The simulator 311 executes route search or space evaluation using a system such as EQS (Environment Query System), calculation for agent action or interaction, etc. in the virtual space field generated by the space output unit 310, and acquires simulation information. The acquired simulation information may be used for reinforcement learning. The real space information and sensing information are converted by the common unit 211 of the server device 2 into data usable by the game engine device 3.

[0083] The simulation information is information obtained by simulating in a virtual space a predicted change in the real space. The simulation information includes information obtained by simulating in the virtual space the movement of the human 6, the movement of the agent device 4, the movement of the virtual agent, and the movement of the object 7 in the real space. With this configuration, a simulation in the real space can be executed in the virtual space. With this configuration, the movement of the human 6 existing in the real space can be predicted.

[0084] The simulator 311 may output command information together with the simulation information. The command information is information indicating a command to the agent. The agent may include a virtual agent that exists only in the virtual space in addition to the agent device 4 that exists in the real space. A normal agent has its body synchronized in the physical space and the digital twin space, but a digital agent has a body only in the digital twin space. When including a virtual agent, the virtual space is described in a manner including the virtual agent. The virtual agent corresponds to an AR avatar, a VR character, etc., and has any shape and size. The virtual agent operates in the virtual space in cooperation with the predicted movement of the human 6 based on the simulation information and the command information. According to such a configuration, the virtual agent can be operated in cooperation with the human 6 that exists in the real space.

[0085] The space output unit 310 and the simulator unit 311 of the game engine device 3 control the meta AI, the character AI, and the space AI, and can create a variety of situations within the game by coordinating these three types of AI. The meta AI recognizes the overall situation and gives commands to the character AI to make the situation advantageous or to change game objects within the game. The character AI functions as the brain of each agent, allowing each agent to recognize the situation, make decisions, and move their bodies.

[0086] The space output unit 310, which serves as a space AI, can generate virtual space information in response to changes in the states of the real space and the virtual space, and adjust the state of the virtual space based on the virtual space information. The space output unit 310 uses a trained model that has been trained to generate virtual space information from real space information, and outputs virtual space information corresponding to the real space information in real time. With this configuration, it is possible to output virtual space information with high accuracy by utilizing artificial intelligence.

[0087] The simulation unit 311 as a spatial AI also recognizes the spatial state and behaves to consider what kind of command information to give to each agent in order to change the spatial situation. That is, it generates simulation information and command information in response to changes in the states of the real space and the virtual space, and can adjust the state of the virtual space based on the virtual space information and the command information. The simulation unit 311 outputs the virtual space information and command information in real time using a trained model trained to generate virtual space information and command information from real space information or virtual space information.

[0088] The meta AI, character AI, and spatial AI may undergo reinforcement learning using any of the variables contained in the real space information, virtual space information, simulation information, and command information as features.

[0089] (Step S7) The information transmitting / receiving unit 312 (space output unit 310) transmits virtual space information, simulation information, and command information to the agent device 4. The agent device 4 receives the virtual space information and simulation information from the game engine device 3 via the network N and the communication unit 43.

[0090] Here, the time resolution of the virtual space information transmitted to the agent device 4 is adjusted as appropriate. That is, the information transmitting / receiving unit 312 controls the frequency of outputting the virtual space information to the agent according to the service provided by the agent. For example, when there are an agent device that dances and an agent device that determines a person's face, the agent device that dances is required to perform delicate actions, so the information transmitting / receiving unit 312 transmits the virtual space information with a relatively high spatial resolution (high frequency). On the other hand, the agent device that determines a person's face is not required to perform delicate actions compared to the agent device that dances, so the information transmitting / receiving unit 312 transmits the virtual space information with a relatively low spatial resolution (low frequency). With this configuration, the frequency of the agent's actions can be controlled according to the service provided by the agent.

[0091] Here, the spatial resolution of the virtual space information transmitted to the agent device 4 is adjusted as appropriate. That is, the information transmitting / receiving unit 312 controls the resolution of the virtual space indicated by the virtual space information according to the service provided by the agent. For example, when an agent device that determines a person's face and an agent device that transports luggage exist, the agent device that determines a person's face is required to determine the person's face with high resolution, so the information transmitting / receiving unit 312 sets the spatial resolution of the virtual space information to be transmitted relatively high (high accuracy). On the other hand, the agent device that transports luggage is not required to operate with a high resolution compared to the agent device that determines a person's face, so the information transmitting / receiving unit 312 can set the spatial resolution of the virtual space information to be transmitted relatively low (low accuracy). With this configuration, the resolution of the virtual space is controlled according to the service provided by the agent, so that the virtual space information can be efficiently provided to the agent.

[0092] (Step S8) The agent device 4 acts based on the virtual space information, the simulation information, and the command information. For example, the agent device 4 acts as if it is cooperating with the human 6, other agent devices, or virtual agents by referring to the movements of the human 6, other agent devices, or virtual agents. The agent device 4 does not simply determine its own movement based on the sensing information, but determines its own movement based on the virtual space information and the simulation information. With this configuration, the agent device 4 can autonomously operate based on the virtual space information, the simulation information, and the command information. In addition, the agent device 4 can recognize its own position by acquiring the virtual space information from the game engine device 3, without using a method such as SLAM (Simultaneous Localization and Mapping) from its own sensor or camera to recognize its own position. In addition, since the agent is operated based on the virtual space information, the simulation information, and the command information, even if there are multiple agents with different specifications, they can be controlled collectively. In the digital twin of this embodiment, the agent device 4, the human 6, and the virtual agent behave as if they are operating in cooperation with each other.

[0093] 6.2.Example Next, four examples will be described as details of the information processing outlined above.

[0094] Example 1 The IoT device acquires temperature map information. The temperature map information includes information on temperatures (pairs of positions and temperatures) corresponding to positions in the real space, and is an example of sensing information. The IoT device transmits the temperature map information to the server device 2 via the network N. The information transmitting / receiving unit 212 of the server device 2 receives the temperature map information from the IoT device. The common unit 211 of the server device 2 converts the temperature map information into data that can be used by the game engine device 3 via a common ground. The information transmitting / receiving unit 212 of the server device 2 transmits the converted temperature map information to the game engine device 3. The information transmitting / receiving unit 312 of the game engine device 3 receives the converted temperature map information from the server device 2 . The space output section 310 of the game engine device 3 outputs the virtual space as virtual space information in a form including information on the temperature for each position, based on the converted temperature map information and the real space information. The simulator 311 of the game engine device 3 acquires simulation information by simulating future temperature distribution in the space based on time-series virtual space information. The simulation information includes information obtained by simulating predictions of temperature changes in the real space in the virtual space. The simulator 311 of the game engine device 3 generates, as command information, instructions for the actions of each agent based on the temperature change. The information transmitting / receiving unit 312 of the game engine device 3 transmits virtual space information, simulation information, and command information to a plurality of agent devices. The agent devices include robots that operate based on the virtual space information, simulation information, and command information. Each robot receives virtual space information, simulation information and command information from the game engine device 3. Each robot determines its own behavior based on the temperature situation indicated by the virtual space information, the predicted temperature change indicated by the simulation information, and the command information. Each robot operates based on its own determined behavior. According to such an embodiment, a change in temperature can be predicted based on information on the temperature corresponding to each position in the real space, and the robot can be operated based on the result. Also, according to such an embodiment, when the semantics of the data indicated by the sensing information indicates the temperature corresponding to each position in the real space, the agent can utilize the sensing information regardless of the type of sensor.

[0095] Example 2 The IoT device acquires weight information. The weight information includes information on the weight of packages present in the real space. More specifically, for example, the weight information includes information on a plurality of packages to be delivered and the weight of each package. The weight information is an example of sensing information. The IoT device transmits the weight information to the server device 2 via the network N. The information transmitting / receiving unit 212 of the server device 2 receives the weight information from the IoT device. The common unit 211 of the server device 2 converts the weight information into data that can be used by the game engine device 3 via a common ground. The information transmitting / receiving unit 212 of the server device 2 transmits the converted weight information to the game engine device 3 . The information transmitting / receiving unit 312 of the game engine device 3 receives the converted weight information from the server device 2 . The space output unit 310 of the game engine device 3 outputs the virtual space as virtual space information in a form including information on the weight of luggage present in the real space, based on the converted weight information and the real space information. The simulator 311 of the game engine device 3 acquires simulation information by simulating the baggage that each agent in the space will carry in the future, based on the converted weight information. The simulator 311 of the game engine device 3 generates command information related to the luggage to be carried by each agent. The information transmitting / receiving unit 312 of the game engine device 3 transmits virtual space information, simulation information, and command information to each of the agent devices. The agents include robots each having a specified size and weight of luggage that they can carry. Each robot receives virtual space information, simulation information and command information from the game engine device 3. The robot carries one or more loads that it can carry based on virtual space information, simulation information, and command information. Each robot selects loads that it can lift and acts accordingly. If a robot can carry up to 100 kg, the robot will systematically collect and carry multiple loads, such as loads of 50 kg, 30 kg, and 20 kg. According to such an embodiment, a robot capable of carrying a load can be operated based on information on the weight of the load in the real space. According to such an embodiment, when the semantics of the data indicated by the sensing information indicates the weight of the load present in the real space, the agent can utilize the sensing information regardless of the type of sensor.

[0096] Example 3 The IoT device acquires density information, which includes information on the density of people in each section of a real space and is an example of sensing information. The IoT device transmits the density information to the server device 2 via the network N. The information transmitting / receiving unit 212 of the server device 2 receives the density information from the IoT device. The commonization unit 211 of the server device 2 converts the density information into data that can be used by the game engine device 3 via a common ground. The information transmitting / receiving unit 212 of the server device 2 transmits the converted density information to the game engine device 3. The information transmitting / receiving unit 312 of the game engine device 3 receives the converted density information from the server device 2 . The space output unit 310 of the game engine device 3 outputs the virtual space as virtual space information in a form including information on human density for each section in the real space, based on the converted density information and the real space information. The simulation unit 311 of the game engine device 3 acquires simulation information by simulating the flow of people from the change over time in the human density for each section in the space based on the converted density information. The simulation information includes information obtained by simulating in the virtual space the predicted human density in the real space. The simulator 311 of the game engine device 3 generates command information instructing to move to a location where the density of people is high and to issue an alarm. The game engine device 3 transmits virtual space information, simulation information, and command information to the agent device. The agent includes a drone capable of generating an alarm. The drone receives virtual space information, simulation information, and command information from the game engine device 3. Based on virtual space information, simulation information and command information, the drone moves to areas with high human density and issues an alarm. According to such an embodiment, a change in the density of people can be predicted based on information on the density of people corresponding to each section in the real space, and an alarm can be generated in the drone based on the result. According to such an embodiment, when the semantics of the data indicated by the sensing information indicates the density of people for each section in the real space, the agent can utilize the sensing information regardless of the type of sensor.

[0097] Example 4 An IoT device having an aspect of a transported baggage transmits baggage classification information and baggage location information to the server device 2 via the network N. The baggage classification information includes information indicating the classification of the baggage and is an example of sensing information. Also, the baggage location information includes information indicating the location of the baggage and is an example of sensing information. The information transmitting / receiving unit 212 of the server device 2 receives, via the network N, the baggage classification information and the baggage position information from the IoT device. The common unit 211 converts the baggage classification information and baggage position information into data that can be used by the game engine device 3 via a common ground. The information transmitting / receiving unit 212 of the server device 2 transmits the converted baggage classification information and the converted baggage position information to the game engine device 3. The information transmitting / receiving unit 312 of the game engine device 3 receives the converted baggage classification information and the converted baggage position information from the server device 2. The spatial output unit 310 of the game engine device 3 outputs the virtual space as virtual space information based on the converted luggage classification information, the converted luggage position information, and real space information, in a form including information on the luggage classification in the real space and information on the position of the luggage. The simulation unit 311 of the game engine device 3 acquires simulation information by simulating the transportation of luggage by a robot from luggage information for each section in real space based on the converted luggage classification information and the converted luggage position information. The sections are defined in advance in the real space. The simulation information includes information obtained by simulating in virtual space a prediction of luggage transportation for each section in the real space, and information obtained by simulating the remaining luggage capacity at a location where the luggage is accumulated based on the luggage classification information and luggage position information acquired in time series. The location where the luggage is accumulated is an example of a predetermined location. The game engine device 3 transmits virtual space information, simulation information, and command information to the agent device. The agent includes a robot capable of carrying luggage. The robot receives virtual space information, simulation information and command information from the game engine device 3. The robot transports the corresponding load from the load capacity accumulated in the compartment. According to such an embodiment, it is possible to simulate the remaining baggage capacity at a location where the baggage is collected, based on baggage classification information and location information in the real space. According to such an embodiment, when the semantics of the data indicated by the sensing information indicate the baggage classification and baggage location, the sensing information can be utilized by a system to which the sensing information is output, regardless of the type of sensor.

[0098] 7. Experimental Example An experimental example using the present disclosure will be described below with reference to FIGS.

[0099] 13 and 14 show data acquired from IoT equipment installed in a test site. The test site shown in FIG. 13 is prepared so that real space information can be acquired from BIM, point cloud data, LiDAR, camera sensor data, etc. FIG. 13 shows that the space output unit 310 of the game engine device 3 detects a human based on the real space information. FIG. 14 shows that the space output unit 310 of the game engine device 3 tracks a human based on time-series real space information.

[0100] FIG. 15 shows data that reproduces the flow of people in the experimental field. In the example of FIG. 15, the spatial output unit 310 of the game engine device 3 receives position information of people as sensing information from seven ToF sensors installed in the experimental field, and integrates it with real space information (BIM in this experimental example) in real time to provide a virtual space that accurately reproduces the real space including the flow of people. Object detection and mapping of the virtual space were realized by LiDAR using a ToF sensor. As a result, the spatial output unit 310 of the game engine device 3 was able to suppress the delay time of rendering the entire virtual space to 0.6 seconds or less. In the example of FIG. 15, the spatial AI, the simulator unit 311 of the game engine device 3 analyzes the distance between people in the virtual space in real time, immediately identifies a crowded situation, and gives command information to the display or speaker. The display or speaker generates a warning in a manner that can be identified by the user based on the command information.

[0101] Fig. 16 shows an example in which a wheelchair was controlled while recognizing the surrounding situation in an experimental field. In the example of Fig. 16, the wheelchair used in the experiment is an agent that does not have a sensor mounted thereon for recognizing the surrounding environment. The wheelchair acquired virtual space information and command information, and recognized the three-dimensional structure of the space, the real-time flow of people, and obstacles based on the virtual space information and the command information, and moved safely.

[0102] FIG. 17 shows data of the real space in which the wheelchair and the robot are controlled while recognizing the surrounding situation in the experimental field. FIG. 18 shows data of the real space in which the wheelchair and the robot are controlled while recognizing the surrounding situation in the experimental field. In the examples of FIG. 17 and FIG. 18, the personal mobility "WHILL model CR" of WHILL Inc. and the personal robot "Temi" of Temi USA Inc. are controlled as agents. In the examples of FIG. 17 and FIG. 18, a space AI is shown automatically guiding a person in a wheelchair and a person who remotely logged in to the robot to a reserved seat in an unmanned restaurant. In the examples of FIG. 17 and FIG. 18, when a person or an obstacle enters the path of the wheelchair, the simulator 311 transmits command information to the wheelchair indicating that the wheelchair should slow down or stop. Also, if the situation does not change within a few seconds, the simulator 311 transmits command information to the wheelchair indicating that the wheelchair should change course. The delay required for obstacle detection was 0.3 seconds, and the control delay was within 0.1 seconds.

[0103] Fig. 19 is a diagram showing data of a real space in which an agent is controlled based on the movement of a human. Fig. 20 is a diagram showing data of a virtual space in which an agent is controlled based on the movement of a human. The space output unit 310 determined that the human waved his / her hand based on the skeletal information. The simulation unit 311 of the game engine device 3 transmitted command information to the personal mobility, which indicates an instruction to move the personal mobility as an agent to the location of the human.

[0104] As described above, according to the present disclosure, a technology capable of outputting a virtual space based on real space information indicating the structure of the real space and information obtained by sensing the real space, or a technology capable of allowing a system to which the sensing information is output to utilize the sensing information regardless of the type of method for acquiring the sensing information, can be provided. According to the former, a virtual space can be expressed by further adding an element of sensing information to the structure of the real space, and therefore a technology capable of contributing to the development of industries in various scenes such as medicine, chemistry, engineering, architecture, agriculture, IT, finance, entertainment, education, and tourism can be provided. According to the latter, a system to which the sensing information is output can utilize the sensing information regardless of the type of method for acquiring the sensing information, which leads to the omission of a design for ensuring compatibility or the adoption of an inexpensive sensor, and thus the difficulty in various research, development, business activities, and the like can be reduced.

[0105] [others] Regarding the information processing system 1 according to the above embodiment, the present invention may be a program that causes a computer to function as the processor 31 of the information processing system 1. The present invention may also be an information processing method executed by the information processing system 1. The information processing method includes each process executed by the processor of the information processing system 1. With such a configuration, the agent can be operated autonomously.

[0106] In the embodiment, the server device 2 and the game engine device 3 are described as separate devices, but in a modified example, the server device 2 and the game engine device 3 may be the same device.

[0107] The server device 2 may be an on-premise type or a cloud type. The server device 2 in the cloud type may provide the above functions and processes in the form of, for example, SaaS (Software as a Service) or cloud computing. The same applies to the game engine device 3.

[0108] In the above embodiment, the server device 2 performs various storage and control, but multiple external devices may be used instead of the server device 2. That is, various information and programs may be distributed and stored in multiple external devices using block chain technology or the like. The same applies to the game engine device 3.

[0109] Furthermore, the present invention may be provided in the following manner.

[0110] (1) An information processing system comprising at least one processor that executes each of the following steps: a spatial receiving step receives real space information, the real space information being information indicating a structure of the real space; a receiving step receives sensing information, the sensing information being information obtained by sensing the real space; and a spatial output step outputs virtual space information in real time, the virtual space information being information indicating a virtual space in which the sensing results indicated by the sensing information are reflected in the structure of the real space indicated by the real space information.

[0111] According to this configuration, it is possible to output a virtual space based on information indicating the structure of the real space and information obtained by sensing the real space.

[0112] (2) In the information processing system described in (1) above, the one or more processors further execute a simulating step, in which simulated information is generated from the sensing information acquired in a time series, and the simulated information is information obtained by simulating in a virtual space a prediction of changes in the real space.

[0113] With this configuration, a simulation in real space can be executed in virtual space.

[0114] (3) In the information processing system described in (2) above, in the space output step, the simulated information is transmitted to an agent, and the agent acts based on the simulated information.

[0115] According to this configuration, the agent can be made to operate autonomously based on the simulation information.

[0116] (4) In the information processing system described in (3) above, the sensing information includes skeletal information, which is information regarding the skeleton of a human being present in the real space, and the simulated information includes information obtained by simulating in a virtual space a prediction of the movement of the human being in the real space.

[0117] With this configuration, it is possible to predict the movements of people existing in real space.

[0118] (5) An information processing system according to any one of (4) above, wherein the agent includes a virtual agent that exists only in the virtual space, and the virtual agent operates in coordination with the movements of the human based on the simulated information.

[0119] With this configuration, it is possible to operate the virtual agent in cooperation with a human being present in the real space.

[0120] (6) In the information processing system described in any one of (3) to (5) above, the sensing information includes temperature map information indicating temperatures corresponding to each position in the real space, the simulated information includes information obtained by simulating in a virtual space a prediction of a change in temperature in the real space based on temperature map information acquired in time series, and the agent includes a robot that operates based on the simulated information.

[0121] According to this configuration, a change in temperature can be predicted based on information about the temperature corresponding to each position in real space, and the robot can be operated based on the result of the prediction.

[0122] (7) In the information processing system described in any one of (3) to (6) above, the sensing information includes weight information indicating a weight of luggage present in the real space, the agent includes a robot having a predetermined weight of luggage that it can carry, and the robot carries one or more luggage that it can carry based on the weight information.

[0123] According to this configuration, it is possible to operate a robot capable of transporting luggage based on information on the weight of the luggage in real space.

[0124] (8) In the information processing system described in any one of (3) to (7) above, the sensing information includes density information indicating the density of people in each section in the real space, the simulated information includes information obtained by simulating in a virtual space a prediction of the density of people in the real space based on density information acquired in a time series, the agent includes a drone capable of generating an alarm, and the drone moves to a location where the density of people is estimated to be high based on the simulated information and generates an alarm.

[0125] With this configuration, changes in human density can be predicted based on information on the human density corresponding to each section in real space, and an alarm can be issued to the drone based on the results.

[0126] (9) In the information processing system described in any one of (3) to (8) above, the sensing information includes baggage classification information and baggage location information, the baggage classification information includes information indicating the classification of the baggage, the baggage location information includes information indicating the location of the baggage, the agent transports the baggage to a predetermined location based on the baggage classification information and the baggage location information, and the simulated information includes information simulating the remaining baggage capacity at the predetermined location based on the baggage classification information and the baggage location information acquired in chronological order.

[0127] With this configuration, it is possible to simulate the remaining luggage capacity at a luggage collection location based on luggage classification information and location information in the real space.

[0128] (10) In the information processing system described in any one of (1) to (9) above, in the space output step, the virtual space information corresponding to the real space information is output in real time using a trained model trained to generate virtual space information from real space information.

[0129] According to this configuration, virtual space information can be output with high accuracy by utilizing artificial intelligence.

[0130] (11) In the information processing system according to any one of (1) to (10) above, in the space output step, the virtual space information is output using a game engine.

[0131] According to such a configuration, it is possible to output virtual space information by utilizing the functions of the game engine, such as the rendering engine, the physics engine, animation, and lighting effects.

[0132] (12) A program that causes a computer to function as the processor of the information processing system according to any one of (1) to (11) above.

[0133] According to this configuration, it is possible to output a virtual space based on information indicating the structure of the real space and information obtained by sensing the real space.

[0134] (13) An information processing method executed by an information processing system, the information processing method comprising each process executed by the processor of the information processing system according to any one of (1) to (11) above.

[0135] According to this configuration, it is possible to output a virtual space based on information indicating the structure of the real space and information obtained by sensing the real space. Of course, this is not the case.

[0136] It may also be provided in the following aspects: (1) An information processing system comprising at least one processor that executes each of the following steps: in a receiving step, sensing information is received, the sensing information being information obtained by sensing a real space; and in a commonizing step, the sensing information is converted into a data format that can be interpreted by a system to which the sensing information is output, in accordance with attributes of data indicated by the sensing information.

[0137] According to this configuration, the system to which the sensing information is output can utilize the sensing information, regardless of the type of method used to acquire the sensing information.

[0138] (2) In the information processing system described in (1) above, in the commonization step, when the semantics of the data of the sensing information are identical, the data format of the sensing information is converted so that it becomes a common data format.

[0139] According to such a configuration, when the semantics of the data indicated by the sensing information is the same, the system to which the sensing information is output can utilize the sensing information regardless of the type of method used to acquire the sensing information.

[0140] (3) In the information processing system described in (1) or (2) above, the one or more processors further execute a space output step, in which virtual space information is output to a system to which the sensing information is output, and the virtual space information is information indicating a virtual space in which the results of the sensing are reflected in the real space.

[0141] With this configuration, the agent can be made to act autonomously based on the virtual space information.

[0142] (4) In the information processing system described above in (3), in the space output step, a frequency of outputting the virtual space information to the agent is controlled in accordance with a service provided by the agent.

[0143] With this configuration, the frequency of the agent's actions can be controlled according to the services the agent provides.

[0144] (5) In the information processing system described in (3) or (4) above, in the space output step, a resolution of the virtual space indicated by the virtual space information is controlled in accordance with a service provided by the agent.

[0145] According to such a configuration, the resolution of the virtual space is controlled in accordance with the service provided by the agent, so that virtual space information can be efficiently provided to the agent.

[0146] (6) In the information processing system described in any one of (1) to (5) above, the sensing information includes data indicating a structure of real space, and in the commonization step, the sensing information is converted into mesh-format data indicating the structure of the real space.

[0147] According to this configuration, when the semantics of the data indicated by the sensing information indicates a structure of real space, the system to which the sensing information is output can utilize the sensing information regardless of the type of sensor.

[0148] (7) In the information processing system described in any one of (1) to (6) above, the sensing information includes information indicating a temperature corresponding to each position in the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by a system to which the sensing information is output, as information indicating a temperature corresponding to each position in the real space.

[0149] According to this configuration, when the semantics of the data indicated by the sensing information indicate temperatures corresponding to each position in real space, the system to which the sensing information is output can utilize the sensing information regardless of the type of sensor.

[0150] (8) An information processing system according to any one of (1) to (7) above, wherein the sensing information includes information indicating a weight of luggage present in the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by a system to which the sensing information is output, as information indicating the weight of luggage present in the real space.

[0151] With this configuration, when the semantics of the data indicated by the sensing information indicates the weight of luggage present in real space, the system to which the sensing information is output can utilize the sensing information regardless of the type of sensor.

[0152] (9) An information processing system according to any one of (1) to (8) above, wherein the sensing information includes information indicating the density of people in each section in the real space, and in the commonization step, the sensing information is converted into a data format that can be interpreted by a system to which the sensing information is output, as information indicating the density of people in each section in the real space.

[0153] With this configuration, if the semantics of the data indicated by the sensing information indicates the density of people in each section in real space, the system to which the sensing information is output can utilize the sensing information regardless of the type of sensor.

[0154] (10) In an information processing system described in any one of (1) to (9) above, the sensing information includes information indicating a classification of the luggage and information indicating a location of the luggage, and in the commonization step, the sensing information is converted into a data format that can be interpreted by a system to which the sensing information is output, as the information indicating the classification of the luggage and the information indicating the location of the luggage.

[0155] According to this configuration, when the semantics of the data indicated by the sensing information indicates the classification and location of the luggage, the system to which the sensing information is output can utilize the sensing information regardless of the type of sensor.

[0156] (11) The information processing system according to any one of (1) to (10) above, wherein the output destination system is a system that uses a game engine as software.

[0157] According to this configuration, the sensing information can be utilized by a system that uses a game engine, regardless of the type of method used to acquire the sensing information.

[0158] (12) A program that causes a computer to function as the processor of the information processing system according to any one of (1) to (11) above.

[0159] According to this configuration, the sensing information can be utilized by a system to which the sensing information is output, regardless of the type of sensor.

[0160] (13) An information processing method executed by an information processing system, the information processing method comprising each process executed by the processor of the information processing system according to any one of (1) to (11) above.

[0161] According to this configuration, the sensing information can be utilized by a system to which the sensing information is output, regardless of the type of sensor.

[0162] Finally, although various embodiments of the present invention have been described, these are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. The embodiments and their modifications are included within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0163] 1: Information processing system 2: Server device 21: Processor 210: Spatial reception area 211:Commonization department 212: Information transmission and reception unit 22: Storage section 23: Communications Department 3: Game engine device 31: Processor 310: Spatial output section 311: Simulation Department 312: Information transmission and reception unit 32: Storage section 33: Communications Department 4: Agent device 41: Processor 42: Storage section 43: Communications Department 44: Sensing section 45: Drive unit 5a: Camera 51a: Processor 52a: Storage section 53a: Communications Department 54a: Imaging section 5b: Sensor 51b: Processor 52b: Storage section 53b: Communications Department 54b: Sensing section 5c: Lighting 51c : Processor 52c: Storage section 53c:Communication Department 54c: Drive unit 5d: Smartphone 51d: Processor 52d: Storage section 53d: Communications Department 54d: Input section 55d: Output section 6: Human 7 : Object N: Network

Claims

1. An information processing system, at least one processor that executes the following steps: In the spatial reception step, real space information is received, The real space information is information indicating a structure of a real space, In the receiving step, sensing information is received, the sensing information is information obtained by sensing the real space, In the spatial output step, virtual space information is output in real time. The virtual space information is information indicating a virtual space in which a result of sensing indicated by the sensing information is reflected in a structure of the real space indicated by the real space information. Information processing system.

2. 2. The information processing system according to claim 1, The one or more processors further perform a simulating step; In the simulating step, simulated information is generated from the sensing information acquired in time series, and the simulated information is information obtained by simulating a prediction of a change in the real space in a virtual space. Information processing system.

3. 3. The information processing system according to claim 2, In the spatial output step, the simulated information is transmitted to an agent; The agent acts based on the simulated information. Information processing system.

4. 4. The information processing system according to claim 3, the sensing information includes skeletal information, the skeletal information being information about a skeleton of a human being present in the real space; The simulation information includes information obtained by simulating in a virtual space a prediction of the movement of the person in the real space. Information processing system.

5. 5. The information processing system according to claim 4, The agent includes a virtual agent that exists only in the virtual space, The virtual agent operates in coordination with the human's movements based on the simulated information. Information processing system.

6. 4. The information processing system according to claim 3, the sensing information includes temperature map information indicating temperatures corresponding to respective positions in the real space; the simulation information includes information obtained by simulating in a virtual space a prediction of a change in the temperature in the real space based on temperature map information acquired in a time series; The agent includes a robot that operates based on the simulated information. Information processing system.

7. 4. The information processing system according to claim 3, the sensing information includes weight information indicating a weight of a package present in the real space, The agent includes a robot having a predetermined weight of luggage that it can carry; The robot transports one or more loads that can be transported based on the weight information. Information processing system.

8. 4. The information processing system according to claim 3, the sensing information includes density information indicating a density of people for each section in the real space; the simulation information includes information obtained by simulating in a virtual space a prediction of the density of the people in the real space based on density information acquired in time series; The agent includes a drone capable of generating an alarm; The drone moves to a location where human density is estimated to be high based on the simulated information and generates an alarm. Information processing system.

9. 4. The information processing system according to claim 3, The sensing information includes baggage classification information and baggage position information; The baggage classification information includes information indicating a classification of the baggage, The luggage location information includes information indicating the location of the luggage, The agent transports the luggage to a predetermined location based on the luggage classification information and the luggage location information, The simulation information includes information simulating the remaining baggage capacity at the predetermined location based on the baggage classification information and the baggage position information acquired in time series. Information processing system.

10. 2. The information processing system according to claim 1, In the space output step, the virtual space information corresponding to the real space information is output in real time using a trained model trained to generate virtual space information from real space information. Information processing system.

11. 2. The information processing system according to claim 1, In the space output step, the virtual space information is output using a game engine. Information processing system.

12. A program, A computer is caused to function as the processor of the information processing system according to any one of claims 1 to 11. program.

13. An information processing method executed by an information processing system, comprising: The information processing system according to any one of claims 1 to 11, further comprising: Information processing methods.

Images