Method and apparatus for cognitive digital twinning based on advanced spatial situation recognition and reasoning process of multi-dimensional facility information model

Abstract

The present invention provides a method and apparatus for performing cognitive digital twinning using an advanced spatial context recognition and reasoning process based on a multi-dimensional facility information model. Based on multi-dimensional facility information for each of a plurality of facilities, a three-dimensional virtual space is implemented for each of the plurality of facilities, and a cell matrix is set for each of a plurality of planes, based on which a cell space list is set for each of a plurality of structure information, a plurality of facility resource information, and a plurality of vector space information, and the 3D virtual space for each of the plurality of facilities is a vector space and a cell represented in space, and based on information about a plurality of sensors pre-installed in each of the plurality of facilities and the multi-dimensional facility information.

Description

Technical Field

[0001] Embodiments of this disclosure relate to techniques for performing cognitive digital twins, and to methods and apparatus for performing cognitive digital twins using advanced spatial context recognition and reasoning processes based on multidimensional facility information models. Background Technology

[0002] Digital twin technology has recently garnered attention across industries and has emerged as a technology for monitoring, analyzing, and optimizing through digital replicas of physical spaces. Digital twins significantly improve operational efficiency and support decision-making accuracy in various industrial environments by connecting physical and virtual data in real time. However, advanced situational awareness and reasoning capabilities are crucial for digital twin technology to move beyond simple data reflection and provide an autonomous, intelligent operating system. This next-generation technology, known as "cognitive digital twins," is evolving into a system that learns from data generated in the physical environment and complex spatial information, enabling real-time responses and predictions through learning.

[0003] Existing digital twin-based systems rely on low-level spatial situational awareness to operate actuators, lacking the technical foundation to identify the correlation between spatial information and sensor data or to infer complex spatial situations, thus limiting the level of automation they can provide.

[0004] Furthermore, the inability to reflect the multidimensional spatial structure and relationships between resources within the facility makes advanced reasoning and real-time simulation based on spatial context difficult. Additionally, because the logical connections between sensors are not learned, data collected by each sensor is processed independently, preventing the utilization of data correlations and necessitating manual adjustments to the spatial model and data each time. The independent processing of data from each sensor, requiring preparation, presents significant financial and time costs. Therefore, a method and apparatus are needed for performing cognitive digital twins using advanced spatial context recognition and reasoning processes based on multidimensional facility information models. Summary of the Invention

[0005] The problem that the invention aims to solve

[0006] Embodiments of this disclosure may provide a method and apparatus for performing cognitive digital twins using advanced spatial context recognition and reasoning processes based on a multidimensional facility information model.

[0007] The technical challenges to be addressed in the embodiments are not limited to those described above, and those skilled in the art may consider other technical challenges not mentioned in the various embodiments described below.

[0008] Methods for solving problems

[0009] According to one embodiment, a method for a server to operate an autonomous actuator using cognitive digital twins includes: implementing a three-dimensional virtual space for each of a plurality of facilities based on multi-dimensional facility information for each of the plurality of facilities. The information includes information about a plurality of planes, each of which includes facility object information and facility space information, and the facility object information includes multiple structural information and multiple facility information. Resource information is included; the facility space information includes multiple vector space information and multiple unit space information; a unit matrix is ​​set for each of the plurality of planes; and the multiple space information is set based on the unit settings, providing a unit space list for each of the structural information, the multiple facility resource information, and the multiple vector space information; representing the three-dimensional virtual space of each of the plurality of facilities as a vector space and a unit space; and multiple information about multiple sensors pre-installed in each facility, based on the multi-dimensional facility information, arranging multiple pre-installed sensors in the three-dimensional virtual space for each of the plurality of facilities; collecting sensing data from the multiple pre-installed sensors; and information about the autonomous actuator based on information about the 3D virtual space and sensing data, information about multiple preset virtual scenes, and the 3D virtual space; determining basic control information using a spatial context inference model of a first neural network. Based on information about space, sensor data, information about the autonomous actuator, and basic control information, a spatial context response model using a second neural network is used to determine the corresponding information for each of multiple preset virtual scenes. This process may include sending information, including basic control information and the corresponding information for each of the multiple preset virtual scenes, to the autonomous actuator. When the autonomous actuator receives a value for a basic mode from the server, it operates based on the basic control information; and when it receives a value for a scene mode from the server, it operates based on multiple control information. Within the corresponding information for each preset virtual scene, operations can be performed based on the corresponding information that matches the value of the scene mode.

[0010] Invention Effects

[0011] As described above, the present invention has the following effects:

[0012] According to an embodiment, the server can utilize a multi-dimensional facility information model specifically designed for cognitive digital twins to accurately reflect the spatial structure and relationships between resources of a facility. This capability enables the server to perform advanced reasoning and real-time simulations based on spatial context, which is impossible with simple 2D or limited 3D data.

[0013] According to an embodiment, the server can systematically manage and process the relationships between the structure, resources, and space of a facility by dividing multidimensional facility information into vector space and cellular space. Based on this structure, the accuracy of object tracking and spatial situational data inference can be significantly improved by more effectively learning artificial intelligence models.

[0014] According to an embodiment, the server can use digital twins to reflect various types of sensors installed in the facility onto a three-dimensional virtual space, and train the corresponding spatial information and sensing data into an artificial intelligence model. In particular, by learning the interactions and correlations between sensors, sensor data can be used more effectively to precisely control and operate autonomous actuators.

[0015] According to an embodiment, the server simulates multiple virtual scenarios in a three-dimensional virtual space and can proactively respond to various situations using predictive data. For example, autonomous actuators can support efficient and autonomous facility management by responding promptly to anticipated environmental changes.

[0016] The effects that can be obtained from the embodiments are not limited to those described above, and other effects not mentioned can be clearly derived and understood by those skilled in the art based on the following detailed description. Attached Figure Description

[0017] Figure 1 The structure of multidimensional facility information according to an embodiment is shown.

[0018] Figure 2 An example of the position of the sensor in vector space and the position of the sensor in cell space according to one embodiment is shown.

[0019] Figure 3 An example of step-by-step spatial context recognition based on spatial scale is shown according to one embodiment.

[0020] Figure 4 This is a diagram illustrating the spatial concept of objects included in a traffic model of CityGML according to an embodiment.

[0021] Figure 5 This is a diagram illustrating multiple moving objects via cognitive digital twins at a 3D virtual city center intersection according to an embodiment. Detailed Implementation

[0022] The following embodiments combine elements and features of the embodiments in a predetermined manner. Unless otherwise explicitly stated, each component or feature can be considered optional. Each component or feature can be implemented without being combined with other components or features. Furthermore, various embodiments can be configured by combining some components and / or features. The order of operations described in the various embodiments can be changed. Some features or characteristics of one embodiment can be included in other embodiments, or can be replaced by corresponding features or characteristics of other embodiments.

[0023] In the description of the accompanying drawings, no processes or steps are described that may obscure the essence of the various embodiments, nor are any processes or steps that would be understood by one of ordinary skill in the art.

[0024] Throughout this specification, neural networks, neural network functions, and network functions can be used with the same meaning. A neural network may consist of a set of interconnected computational units, commonly referred to as "nodes." These "nodes" may also be called "neurons." A neural network consists of at least two or more nodes. The nodes (or neurons) that make up a neural network can be interconnected through one or more "links."

[0025] In neural networks, two or more nodes connected by links can form relative relationships of input and output nodes. The concepts of input and output nodes are relative; any node that is an output node to one node can potentially be an input node to another, and vice versa. As described above, input-to-output relationships can be created around links. One or more output nodes can be connected to an input node via links, and vice versa.

[0026] In a relationship between input and output nodes connected by a link, the value of the output node can be determined based on the data input to the input nodes. Here, the nodes connecting the input and output nodes can have weights. These weights can be variable and can be changed by the user or algorithm so that the neural network performs the desired function. The edges or links interconnecting the input and output nodes have weights that can be variably applied by the user or algorithm to perform the function required by the neural network. For example, when one or more input nodes are connected to an output node via their respective links, the output node is set as the output of the input nodes connected to the output node and the corresponding links for each input node. Node values ​​can be determined based on the weights.

[0027] As mentioned above, in a neural network, two or more nodes are interconnected by one or more links to form the input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links, the correlation between nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weights between the links, these two neural networks might be identified as different from each other.

[0028] Meanwhile, digital twin technology refers to the creation of digital virtual models for actual physical objects or systems, and the use of these models to monitor and analyze the status of the actual objects or systems in real time. In this context, the ultimate goal of digital twin technology may be to achieve intelligent facility management through autonomous operation. To achieve this goal, it is necessary to use digital twin virtual models to simulate various scenarios in advance and to implement installation and operation response plans based on the simulated models within the system.

[0029] Because current digital twin technology primarily focuses on visualization, linking it with simulation technology can be challenging. In other words, since digital twin virtual models cannot be directly used as input data for simulation solvers, separate input data preparation processes (preprocessors) must be performed for both the digital twin virtual model and the simulation solver. Due to the inconvenience and cost of performing these additional separate tasks, simulations using digital twin technology have not yet been widely adopted.

[0030] Cognitive digital twin technology is a more advanced form of digital twin technology that enables the system to possess advanced situational awareness capabilities, achieving autonomous operation of facilities, which is also the ultimate goal of digital twins. Various embodiments of this disclosure can be used in cognitive digital twin technology.

[0031] Figure 1 The structure of multidimensional facility information according to an embodiment is shown. Figure 2 An example of the position of the sensor in vector space and the position of the sensor in cell space according to one embodiment is shown. Figure 3 An example of step-by-step spatial context recognition based on spatial scale is shown according to one embodiment. Figures 1 to 3 The embodiments can be combined with various embodiments of this disclosure.

[0032] The server can generate multidimensional facility information specifically for cognitive digital twins.

[0033] The server can be a server that uses cognitive digital twins to operate autonomous actuators. For example, the server can create a three-dimensional virtual space for each of multiple facilities by constructing a cognitive digital twin for each of multiple facilities based on multi-dimensional facility information. For example, the server can be a server that uses the three-dimensional virtual space in which the cognitive digital twins are constructed to operate autonomous actuators located in each of multiple facilities.

[0034] An autonomous actuator can be a device that autonomously identifies set conditions or situations and operates automatically based on those conditions or situations. For example, an autonomous actuator can include a control module for various facilities, camera devices, and mobile bodies (such as robots). For example, if the autonomous actuator is a control module, it can operate automatically based on specific conditions to maintain optimal performance. For example, the control module can operate the facility based on at least one of various variables such as temperature, humidity, lighting, air quality, noise, and the number of occupants. Air quality can include carbon dioxide concentration and concentration of hazardous substances. For example, if the autonomous actuator is a camera device, the camera device can detect the location of a disaster (such as a fire, flood) or the presence of a specific occupant. For example, the camera device can photograph a disaster object while tracking its movement. For example, the camera device can capture images of a specific occupant while tracking their movement. For example, when the camera device detects a disaster or a specific occupant, it can send a visual notification to a pre-connected management terminal or output an audio notification for the area. For example, if the autonomous actuator is a mobile body such as a robot, the mobile body can detect the presence of at least one of the occupants or workers, move to the location of the occupant or worker, and interact with the occupant or worker.

[0035] Multidimensional facility information can be an information container that includes various semantic and spatial information required for cognitive digital twins. For example, a server can use multidimensional facility information to perceive, identify, reason, predict, and execute the entire process of spatial context.

[0036] Reference Figure 1 Multidimensional facility information can include information about multiple plans. Here, a plane can refer to a floor of the corresponding facility. For example, each piece of information about multiple planes can include facility object information and facility space information. For example, facility object information can include multiple structural information and multiple facility resource information. For example, structural information 410 can be information about the structures that constitute the facility, such as walls, doors, windows, and columns. For example, in structural information 410, information about walls can be set at a higher level, and information about doors, windows, and columns can be set at a lower level than information about walls. For example, facility resource information 420 can be information about the resources that constitute the facility (such as furniture and equipment). For example, facility resource information 420 can include information about furniture and information about the facility. For example, facility space information can include multiple vector space information and multiple unit space information.

[0037] Vector spatial information 430 can be information representing the size and shape of each space on a plane based on vectors. For example, vector spatial information 430 can be information representing the location and size of structural and facility resources in vector form. For example, vector spatial information 430 can include at least one surface information about the space and edge information about each surface. For example, at least one surface information can be set for a plane. Surface information can include edge information constituting the surface, start coordinates and end coordinates, and vector directions of the corresponding edges. For example, the edge information of each surface can include the coordinate information of each vertex of the surface. For example, the size of a room can be represented as a vector connecting the four corners of the room, and the shape of the room can be represented as a polygonal shape formed by these vectors. For example, vector spatial information 430 can be used to precisely manage the location of individual objects.

[0038] Cell space information 440 can be information indicating the size and shape of each space on a plane based on a cell. In other words, cell space information 440 can be information used to divide space into a grid and manage the space of each region. For example, cell space information 440 can include a cell matrix and cell spaces. The cell matrix can be information indicating the cells (cell spaces) occupied by each space on a grid structure that divides the corresponding plane into multiple cells. Cell spaces can represent at least one cell occupied by a structure, at least one cell occupied by facility resources, and at least one cell occupied by each vector space.

[0039] For example, spatial information 440 can be set by performing a spatial assessment on each cell. For example, a spatial assessment can be performed on each cell to determine which structure or vector space the corresponding cell is included in. For example, it can first be determined whether the corresponding cell is included in a structure or vector space based on the bounding box of a specific structure, a specific facility resource, or a specific vector space. This can speed up the assessment by reducing unnecessary computation. In other words, cells outside the bounding box can be considered not to belong to a structure, facility resource, or vector space. For example, by performing an in / out check on the cells included in the bounding box, the structure, facility resource, or vector space included in the cell can be determined. For example, an in / out check can be a process of determining whether the cell's location is inside or across the boundary of a real structure, real facility resource, or real vector space. For example, at least one of raycasting, polygon interior point, or bounding volume hierarchy (BVB) techniques can be used for in / out checks.

[0040] For example, a cell matrix can be set for each of the multiple planes. Then, a cell space list can be set for each of the structural information 410, facility resource information 420, and vector space information 430 based on the cell matrix. For example, the cell space list for structural information 410 could be a list of cells occupied by that structure. For example, the cell space list for facility resource information 420 could be a list of cells occupied by facility resources. For example, the cell space list for vector space information 430 could be a list of cells occupied by vector spaces.

[0041] For example, community spatial information 440 can be used to determine energy consumption, airflow, or the distribution of occupants.

[0042] For example, the 3D virtual space of each of multiple facilities can be expressed as a vector space and a cell space.

[0043] For example, a server can generate multidimensional facility information based on BIM (Building Information Modeling) information. This multidimensional facility information can have an object-based hierarchical structure, and the server can use it to create a 3D virtual space. In this case, the object-based hierarchy expresses the elements of the facility hierarchically and can represent the relationships between these elements. For example, facility elements can include attribute information, such as characteristics, materials, uses, and dimensions. For instance, the server can use multidimensional facility information to track the location of objects in the virtual space.

[0044] For example, a server can receive BIM information for each of multiple facilities from an external source. The server can then generate multidimensional facility information based on the BIM information of each facility and use this information to create a 3D virtual space for each facility. Here, BIM is a 3D digital modeling technology used to manage facilities, encompassing various information such as the physical composition, architectural elements, construction schedule, and maintenance. For instance, BIM information may include information about the facility's external shape, internal shape, dimensions, and components. Information about the facility's components refers to the various elements that make up the facility and may include information about the shape, size, and material of components such as doors, windows, walls, and stairs. Multidimensional facility information can express space usage, access to and from the space, facilities within the space, and connectivity between spaces through vector and cellular spaces. At this point, the server can implement a 3D virtual space based on the multidimensional facility information using predetermined logic or algorithms.

[0045] Servers can collect sensor data from sensors to perceive spatial conditions.

[0046] Spatial situational awareness can be the process by which a server identifies the current state and changes in space based on sensing data collected by sensors. For example, a server can store sensing data collected from sensors in real time using multiple variables.

[0047] Sensors can include various types of sensors. For example, sensors include temperature sensors that measure temperature, illuminance sensors that measure ambient light, humidity sensors that measure air humidity, motion sensors that detect human movement, and differential pressure sensors that detect indoor airflow. In a specific area, it might include carbon dioxide sensors that measure the concentration of carbon dioxide in the air, sound sensors that measure the level of noise generated in the space, and volatile organic compound (VOC) sensors that measure the concentration of harmful substances in the air. For example, sensing data can include temperature data, illuminance data, humidity data, differential pressure data, carbon dioxide concentration data, noise data, and harmful substance concentration data.

[0048] Reference Figure 2 The three-dimensional virtual space of a layer included in the facility can be represented as a vector space map 510 and a unit space map 520.

[0049] For example, vector space map 510 may include three spaces. For example, vector space map 510 may be created based on multi-dimensional facility information. In this case, the vector space map may include spatial polygon information for each space, information about doors connecting to that space, information about windows placed on the walls, and information about the external space. For example, Figure 5 The vector space map 510 may include first to third space polygons, first to third doors, and external space. For example, the vector space map 510 may represent a floor or a specific space within the facility. In this case, the external space may be represented by the number 0, and the first to third space polygons may be represented by the numbers 1 to 3. The first to third entrance doors may be represented by the numbers 0 to 3. The external space may also be identified as a single space, and the polygon representing its outline may be omitted. For example, each of the first to third space polygons may be defined as four clockwise vectors. For example, for each of the first to third space polygons, the four corner coordinates of the corresponding space can be determined based on the four clockwise vectors. For example, the first to third doors may be defined as clockwise vectors. For each of the first to third doors, the starting coordinates, center coordinates, and ending coordinates of the door can be determined based on the clockwise vectors.

[0050] For example, the locations of multiple sensors pre-installed in the facility can be represented in the vector space map 510. For example, the server can display sensors (A, B, C) that match the vector space map 510 based on information about the multiple pre-installed sensors. For example, each sensor (A, B, C) can be displayed in the vector space map 510 according to its location coordinates. For example, the server can reflect the locations of sensors A, B, and C in the vector space map 510 based on information about the multiple pre-installed sensors.

[0051] For example, cell space map 520 can be generated based on vector space map 510. For example, cell space map 520 can display different vector spaces as cells of different colors. For example, cells of different colors can have different identifier values. For example, the server can generate cell space map 520 by setting each of the first to third spatial polygons included in vector space map 510 as a cell with a different identifier value.

[0052] For example, cell space map 520 can display the locations of multiple sensors pre-installed in the facility. For example, the server can display sensors (A, B, C) that match the cell space map 520 based on information about the multiple pre-installed sensors. For example, in cell space map 520, cells matching the location coordinates of each of sensors A, B, and C can be set as the cells where the corresponding sensors are located. For example, the server can set cells corresponding to the locations of sensors A, B, and C in cell space map 520 as the cells for the corresponding sensors based on information about the multiple pre-installed sensors.

[0053] For example, the server can match multiple sensors pre-installed in the facility with multiple vector space maps 510 and multiple unit space maps 520 of the facility. Subsequently, the server can store the sensing data collected from the multiple pre-installed sensors in association with the multiple vector space maps 510 and multiple unit space maps 520. For example, the server can distinguish between using vector space maps 510 and using unit space maps 520 based on the characteristics of the objects to be analyzed in the multiple spaces included in the facility.

[0054] For example, the vector space map 510 can be used to detect specific occupants, detect moving objects such as robots, or detect disasters (e.g., fire or flood) to determine precise locations.

[0055] For example, the cell space diagram 520 can be used to analyze the flow of energy, fluid, and moving objects in the space. In this case, the server can use CA (cellular automata) analysis technology to analyze the flow of energy, fluid, and moving objects in the space based on the cell space diagram 520 and the sensing data associated with it.

[0056] For example, monitoring conditions in space solely through sensor data cannot leverage artificial intelligence or automation. Therefore, the actuators are manually operated, and users may need to directly control them by reviewing data and making judgments.

[0057] The server can analyze patterns in location-based sensor data at four spatial levels and identify the current spatial conditions.

[0058] Reference Figure 3 The four spatial levels can represent progressively different spatial levels based on spatial scale. For example, the four spatial levels could include unit space level, floor level, facility level, and comprehensive level. For example, the server can analyze patterns at the unit space level based on sensing data of a single unit space. For example, the server can analyze floor patterns based on sensing data of a single floor of a facility. For example, the server can analyze facility-level patterns based on sensing data of various facilities. For example, the server can analyze complex-level patterns based on sensing data of multiple facilities.

[0059] Identifying the current spatial condition can be a process of determining the spatial state in real time based on a pattern of analysis across four spatial levels. For example, based on sensor data per unit space, when the number of occupants in a specific room exceeds a set limit, the server can determine whether to increase cooling or heating, or increase lighting. For example, if the server detects a decline in air quality across an entire floor based on sensor data from one floor of a facility, it can decide to operate the ventilation system or optimize lighting based on the number of occupants. For example, when the server detects a fire on a specific floor based on sensor data from relevant facilities, it can send alarms to other floors in the facility and determine the situation to implement emergency measures. For example, when the server detects that the power consumption increase rate in a specific area of ​​the complex exceeds a threshold based on sensor data from multiple facilities, it can adjust the power consumption of equipment in other facilities with importance less than the threshold to optimize power consumption.

[0060] For example, actuators can be automatically executed through basic spatial situational awareness. At this stage, simple rule-based logic (e.g., turning on cooling when the temperature exceeds a certain level) can be applied to automatically operate the actuators.

[0061] The server can analyze the convergence patterns of multiple sensor data and infer the current spatial conditions.

[0062] Multisensor data can be sensing data collected from various types of sensors. For example, inferences about spatial conditions can be made based on the individual sensing data provided by each sensor and the correlations between the sensors.

[0063] A convergence pattern can be a pattern that performs analysis by integrating data from multiple sensors. In other words, a convergence pattern can be a pattern that provides an understanding of complex situations that are difficult to obtain using information from a single sensor. For example, convergence pattern-based analysis might involve jointly analyzing data from illuminance and motion sensors to determine lighting needs, or fusing data from temperature and carbon dioxide sensors to monitor air quality and assess cooling and heating efficiency.

[0064] Specifically, for example, the server can determine the position of each sensor in both vector space and cell space. The server can perform individual spatial situational awareness and identification for each space. The server can enhance spatial situational awareness and identification through complex sensors in various spaces. The server can enhance the perception of spatial conditions through multiple sensor data from across the entire space. At this point, the server can leverage artificial intelligence that has learned the correlations between sensors to infer the spatial situation.

[0065] Inferring the current spatial conditions can be a process of determining control information for optimization by analyzing real-time spatial conditions based on multiple sensor data. For example, a server can analyze the correlations between multiple sensor data by learning an inference model of the correlations between sensors and determine control information suitable for the current spatial conditions.

[0066] For example, a first space can be represented by a first illuminance sensor and a first temperature sensor, a second space by a second illuminance sensor and a second temperature sensor, and a third space by a third illuminance sensor and a third temperature sensor. This can be represented by a cell space diagram. Furthermore, the number of occupants in each space can be calculated by detecting the sensors. For example, the number of occupants in the first space could be 2, the number of occupants in the second space could be 0, and the number of occupants in the third space could be 9. The server can then receive sensor data from the sensors in real time.

[0067] For example, a server can use a spatial context inference model based on sensor data and cell spatial information to correlate illuminance values ​​at a specific time with the location of windows, thereby determining whether the illuminance value in that area is natural or artificial light.

[0068] For example, a server can determine lighting efficiency based on the illuminance and number of people in a specific space using a spatial context inference model, and then decide whether to turn the lighting on or off.

[0069] For example, a server can determine the lighting efficiency of a floor based on lighting energy consumption and the number of residents using a spatial context inference model.

[0070] For example, a server can use a spatial context inference model based on temperature data and the number of occupants in a specific space to determine the efficiency of cooling and heating based on the temperature value and the number of occupants at a specific time.

[0071] For example, servers can be included in the facility based on the total energy consumption of the entire facility, the energy consumption of each space within the facility, and the lighting efficiency of each space within the entire facility, using a spatial condition inference model. The autonomous control direction of each autonomous actuator can then be determined.

[0072] For example, data collected from multiple sensors can be fused to perform complex situational awareness, and actuators can be executed based on the results. In this case, the actuators can make more intelligent decisions by combining various data such as temperature, lighting, and number of occupants, rather than using a single sensor.

[0073] The server can simulate from multiple angles and predict future spatial conditions based on the scenario.

[0074] Multi-angle simulation can be the process of setting up various virtual conditions and performing various simulations for each condition.

[0075] For example, since the spatial model required for each simulation may differ, the server can convert the 3D virtual spaces of multiple facilities into a 3D virtual space suitable for the simulation. For instance, in a simulation experiment analyzing heating and cooling efficiency, the server can create a 3D virtual space capable of analyzing temperature and energy. Similarly, in a simulation experiment analyzing human movement, the server can create a 3D virtual space capable of analyzing human position and movement.

[0076] Scenario-based prediction of future space situations is a process of establishing various hypothetical scenarios, simulating possible future space events, and deriving expected outcomes and response plans.

[0077] The server can generate a set of predictable hypothetical scenarios. For example, a set of predictable hypothetical scenarios can include multiple preset scenarios. These preset scenarios could be, for instance, "a scenario where the external temperature of the facility decreases from 0 degrees Celsius at a rate of 1 degree per hour during winter," or "a scenario where the number of residents in the facility increases at a rate of 1 degree per hour," etc. It can include various scenarios, such as "a fire scenario." For example, each of the multiple preset scenarios can include values ​​for the variables and conditions required for that scenario.

[0078] For example, the server can preprocess the data to be used in each scenario and set the necessary variables or conditions through the preprocessor to simulate each scenario. For example, in a scenario where the external temperature of a facility drops from 0 degrees Celsius at a rate of 1 degree Celsius per hour in winter, the server can set the initial temperature value inside the facility and set the operating range of the heating and cooling systems.

[0079] For example, a server can run simulations in a 3D virtual space using a simulation solver based on a scene and a pre-processed dataset. For example, a server can simulate various changes in space (e.g., temperature, lighting, energy consumption patterns) based on conditions assumed for each scene using a simulation analysis unit. For example, the simulation analysis unit can use a spatial condition prediction model that leverages neural networks. For example, a server can generate simulation results using a spatial condition prediction model based on a given scene and pre-processed data.

[0080] For example, the server can determine the response information for each scenario through the simulation result reporting unit (post-processor). For example, the server can determine the response information for simulations performed based on assumed conditions for each scenario through the simulation result reporting unit. For example, the simulation result reporting unit (post-processor) can include a spatial situation response model using a neural network. For example, the server can determine the response scheme for each scenario based on the simulation results of each scenario using the spatial situation response model.

[0081] The server can execute autonomous actuators based on inferences about the current spatial conditions and predictions about the future spatial conditions.

[0082] For example, spatial conditions can be predicted based on multiple hypothetical scenarios, and actuators can be pre-executed according to the expected results. Since past data and predictive models can be used to predict future conditions, actuators can be autonomously executed based on inferences about the current spatial conditions and predictions of future spatial conditions. Thus, an autonomously operating actuator can be called an autonomous actuator.

[0083] According to one embodiment, the server can realize a three-dimensional virtual space for each of the multiple facilities based on multi-dimensional facility information for each of the multiple facilities.

[0084] For example, multidimensional facility information can include information about multiple planes. Multiple planes can refer to multiple floors that make up the facility. In other words, an airplane might represent one floor of the facility.

[0085] For example, each piece of information regarding multiple planes may include facility object information and facility spatial information. For example, facility object information may include structural information and facility resource information. For example, facility spatial information may include vector space information and unit space information.

[0086] For example, a cell matrix can be set for each of multiple planes. For example, a cell space list can be set up based on the cell matrix for each of the structural information, facility resource information, and vector space information.

[0087] For example, the 3D virtual space of each of multiple facilities can be expressed as a vector space and a cell space.

[0088] For example, the three-dimensional virtual space of a facility may include multiple layers, and each of the multiple layers may include multiple spaces in which multiple structures and multiple facility resources are interconnected.

[0089] For example, the indoor and outdoor spaces of a facility implemented as a 3D virtual space by a server can be composed of combinations of unit spaces. The external space is composed of small areas, and the internal space is divided into floors, each with unit spaces, connected by movable doors. In this case, the server can manage long spaces, such as corridors, by dividing them into multiple unit spaces. The server can configure the indoor and outdoor spaces of the facility into a spatial network. By placing camera equipment within each unit space, a spatial camera network can be formed. The camera equipment can be cameras installed at multiple locations within the facility. For example, camera equipment can capture still images and videos of specific areas within the facility to monitor those areas. For example, camera equipment may include a processor, memory, communication module, one or more lenses, image sensor, image signal processor, or flash. The server can analyze the video in real time through all camera equipment and store occupant attributes (gender, height, clothing color, behavior, etc.). In this case, occupant attribute information detected by a specific camera device at a specific time can be stored as metadata along with information about the corresponding space. Each piece of metadata can be stored in the form of a person (time, gender, height, clothing, action, space). At this point, not only can the identification value of the detected occupant's space be stored in the metadata, but also the 3D spatial coordinates can be stored in the metadata. For example, the metadata can be stored as a table in RDB format. To extract the occupant's 3D spatial coordinates, the server can extract the occupant's image from the camera device's image and determine the occupant's 3D coordinates in the virtual model view. At this point, the image from the camera device can be matched with the virtual model.

[0090] The server can place multiple pre-installed sensors in a three-dimensional virtual space for each of the multiple facilities based on information about multiple sensors pre-installed in each of the multiple facilities and multi-dimensional facility information.

[0091] Multiple pre-installed sensors may include various types of sensors pre-installed in a facility. Information about the multiple pre-installed sensors includes, for each of the multiple pre-installed sensors, values ​​for the sensor type, values ​​for the sensor's position coordinates, values ​​for the sensor's sensing direction, values ​​for the sensor's sensing range, and it may include sensor identification values. For example, the sensor's position coordinate values ​​may include an identifier of the space in which the sensor is located and the sensor's three-dimensional coordinates within that space. For example, preset identification values ​​may be provided for each of the multiple spaces included in each of the multiple facilities.

[0092] For example, the server determines the space matching the identifier value of the sensor's location based on multidimensional facility information, and determines the position coordinates in the vector space and cell space matching the sensor's 3D coordinate values. The position coordinates in this space can be determined. The position coordinates in the vector space can include the identifier value of the corresponding vector space and the position coordinates within that vector space. The position coordinates in the cell space can include the identifier value of the corresponding cell space and the cell index value within that cell space.

[0093] For example, a server can place a virtual sensor, matched with sensor type and sensor identification values, at a location in a 3D virtual space corresponding to its 3D coordinates. The server can then configure the area that the virtual sensor can detect in the 3D virtual space based on the sensor's detection direction and detection range. For instance, the server can pre-store virtual sensors matched according to combinations of sensor type and sensor identification values.

[0094] The server can collect sensing data from multiple pre-installed sensors.

[0095] Sensing data can include temperature data, illuminance data, humidity data, differential pressure data, carbon dioxide concentration data, noise data, and hazardous substance concentration data. Temperature data is the temperature measured by the sensor, and the unit may be degrees Celsius. Illuminance data is the brightness measured by the sensor, and the unit may be lux. Humidity data is the humidity measured by the sensor, and the unit may be a percentage representing relative humidity. Sensing data is the data on movement detected by the sensor, and may include the time the movement was detected. Differential pressure data is the pressure difference measured by the sensor in the pipes installed in the facility, and the unit may be Pascals. Carbon dioxide concentration data is the carbon dioxide concentration measured by the sensor, and the unit may be ppm. Noise data is the noise data measured by the sensor, and the unit may be decibels (dB). Hazardous substance concentration data is the concentration of hazardous substances measured by the sensor, and the unit may be ppm.

[0096] The server can collect sensing data from multiple pre-installed sensors at preset time intervals. The preset time can be 1 second.

[0097] The server can determine basic control information based on information about the autonomous actuator, information about the 3D virtual space, and sensor data by using a spatial context inference model of a first neural network.

[0098] Information about autonomous actuators can include information about their type, location, and operation. For example, information about autonomous actuators can be pre-stored on a server.

[0099] Information about the type of autonomous actuator can include one of several values ​​for different types of autonomous actuators. For example, values ​​for multiple types of autonomous actuators could include those for lights, air conditioners, heaters, ventilation systems, camera devices, and robots. The location information of the autonomous actuator can include an identifier of the space in which the autonomous actuator is located and its three-dimensional coordinates within that space. Information related to the operation of the autonomous actuator can include the settings required for its operation and the average hourly energy consumption of the autonomous actuator. For example, the information related to the operation of the autonomous actuator can vary depending on the type of autonomous actuator. For example, the settings required for the operation of an autonomous actuator might be the brightness level in lighting conditions, the settable temperature in air conditioning or heating conditions, the adjustable airflow in ventilation system conditions, and for camera devices, it might include adjustable vertical and horizontal angle values ​​in the case of a light source, and in the case of a robot, it might include a value for the movement speed. For example, information about autonomous actuators can be pre-stored on a server for each type of autonomous actuator.

[0100] For example, the server determines a space matching the identifier value of the autonomous actuator's location based on multidimensional facility information, and determines the position coordinates and cells in a vector space that match the following 3D coordinate values: The server can determine the position coordinates of the autonomous actuator within that space. For example, the server converts a virtual autonomous actuator that matches the actuator's type value, the settings required for its operation, and its hourly energy consumption into corresponding 3D virtual coordinate values. These 3D coordinate values ​​can be placed anywhere in the space. For example, virtual autonomous actuators matched based on combinations of the actuator's type value, the settings required for its operation, and its hourly energy consumption can be pre-stored on the server.

[0101] Information about the 3D virtual space can include information about the 3D virtual space where the autonomous actuator is located and information about the 3D virtual space where multiple pre-installed sensors are located. For example, information about the 3D virtual space where the autonomous actuator is located includes structural information, facility resource information, and vector space information. The location of the autonomous actuator and the space it occupies can include cell space information. Similarly, information about the 3D virtual space where multiple pre-installed sensors are located includes structural information, facility resource information, and vector space information. The sensor space and its information can include cell space information. For example, information about the 3D virtual space can be obtained based on a 3D virtual space implemented by a server.

[0102] According to one embodiment, the spatial context inference model may include a first CNN model based on a vector space reflecting spatial features, a second CNN model based on a cell space, and a bidirectional long short-term memory (LSTM) model reflecting temporal features.

[0103] Here, the LSTM model can effectively model time series information because it is typically a recurrent neural network (RNN), and the hidden layer values ​​stored internally for the current input are considered in the output of the next input. However, since RNNs are structures that rely on past observations, they can suffer from vanishing gradients or extremely large gradient values ​​(gradient explosion). The model that addresses this problem is the LSTM. By replacing the nodes inside the LSTM with memory cells, information can be accumulated or some past information can be deleted, thus mitigating the problems of RNNs. Furthermore, bidirectional LSTMs are LSTMs that are bidirectional and can include both forward and backward LSTMs. For example, the activation function of a forward LSTM can be a linear function, while the activation function of a backward LSTM can be the sigmoid function.

[0104] For example, by performing a first data preprocessing on information and sensing data from a 3D virtual space, the sensing vector includes the position coordinates of each of multiple sensors in a vector space and the sensing value of each of the multiple sensors. The sensors can be created in parts over time. The position coordinates of each of the multiple sensors in the vector space can include the identified value of the vector space and the sensor's position coordinates in the vector space. The sensing values ​​of each of the multiple sensors are temperature values, illuminance values, humidity values, the time of detected movement, pressure differential values, carbon dioxide concentration values, noise values, and toxicity values. Substance concentration values ​​may also be included. For example, the server performs a first data preprocessing on information and sensing data from the 3D virtual space to determine multiple sensors placed in the 3D virtual space and their position coordinates in the vector space. A sensing vector containing the sensing values ​​of each of the multiple sensors can be generated for each time period. For example, this time interval can be a preset time interval.

[0105] For example, based on the sensing vectors input to the first CNN model for each time period, a first spatial feature vector for each time period can be output. For instance, the server can obtain the first spatial feature vector for each time period by inputting the sensing vectors for each time period into the first CNN model.

[0106] For example, a first CNN model may include a first input layer, one or more first hidden layers, and a first output layer. Learning data, consisting of multiple perceptual vectors and multiple correct first spatial feature vectors, is input to the first input layer, passes through one or more first hidden layers and the first output layer, and is output as an output vector. This output vector is the input to a first loss function layer connected to the output layer. The first loss function layer uses a first loss function to compare the output vector of each training data point with the correct answer vector to output a first loss value. The parameters of the CNN model in the first loss function layer are the first loss value. Learning can proceed in the direction of decreasing loss.

[0107] The first spatial feature vector can be a vector representing the spatial features of a sensing vector composed of learned data. For example, the correct answer suggests that the first spatial feature vector could include density values ​​from multiple sensors, center coordinates of multiple sensors, correlations between sensing values ​​and location coordinates, and correlations between different types of sensing values. It can contain values.

[0108] For example, the density value of multiple sensors represents the number of sensors per unit area of ​​space within a facility, and can indicate the sensor density. For example, the density value of multiple sensors is [value] per square meter. It could be the number of sensors per sensor. For example, the center coordinates of multiple sensors are the average coordinates of the locations of multiple sensors in the facility, and can represent the center of the space where sensors are distributed in each space. For example, the center coordinates of multiple sensors can be three-dimensional coordinate values. For example, the correlation between a sensing value and its location coordinates can be represented spatially as the correlation between the location of each sensor and the sensing value (temperature, illuminance, etc.). The correlation value between different types of sensing values ​​can be represented spatially as the correlation between different types of sensing values, such as temperature and humidity, or illuminance and temperature. For example, the correlation can be determined as a value greater than 0 and less than 1. The closer to 1, the higher the correlation.

[0109] One or more first hidden layers may include one or more convolutional layers and one or more pooling layers. For example, one or more first hidden layers may learn vector-based spatial patterns through one or more convolutional layers and one or more pooling layers. For example, basic relationships (e.g., distances between sensors) may be learned in the first layer, and high-dimensional relationships (e.g., correlations, directions, etc. of sensor values) may be learned in subsequent layers.

[0110] For example, sensing vectors can be filtered in convolutional layers, and feature maps can be formed through convolutional layers. For instance, convolutional layers can divide sensing vectors into blocks of a preset size and learn local patterns between each sensing value and its position coordinates. For example, after performing convolution operations in convolutional layers, a rectified linear unit (ReLU) function can be applied. This allows the non-linear characteristics of the sensing vectors to be realized.

[0111] For example, by selecting a fixed vector related to the feature from the feature map formed in the pooling layer for dimensionality reduction and subsampling the formed feature map, the correct first-space feature vector and related features can be extracted. In other words, the pooling layer can remove unnecessary details by reducing the feature map, reflecting only the important information. For example, the pooling layer can be a max pooling layer that extracts the maximum value. For example, the pooling layer can be an average pooling layer that extracts the average value. For example, in this case, the parameters of the first CNN model can include parameters related to the convolutional and pooling layers (feature map size, filter size, depth, stride, zero padding).

[0112] For example, through a second data preprocessing of information and sensing data about the 3D virtual space, flow vectors including the position coordinates of each moving object in the cell space, energy, and fluid in the 3D virtual space can be generated for each time period. For example, a moving object in the 3D virtual space refers to an object whose position changes; a moving object in the 3D virtual space can include an occupant. For example, energy within the 3D virtual space can include thermal energy, electrical energy, and light energy. For example, fluids in the 3D virtual space may contain carbon dioxide and harmful substances.

[0113] In the 3D virtual space, the position coordinates of each moving object, energy, and fluid within a cellular space represent a cellular space defined for each resident. The 3D virtual space for thermal energy, electrical energy, light energy, carbon dioxide, and hazardous substances can include identifier values ​​and corresponding cell index values ​​within the cellular space. The values ​​of each cell index for thermal energy, electrical energy, light energy, carbon dioxide, and hazardous substances can be determined as the cell index value matching the position where the sensed value associated with each object is greater than or equal to the value of the cell index. Preset thresholds are used. For example, the sensed values ​​associated with thermal energy, electrical energy, light energy, carbon dioxide, and hazardous substances can be temperature, energy consumption, illuminance, carbon dioxide concentration, and hazardous substance concentration.

[0114] For example, the server generates a flow vector that includes the position coordinates of each moving object, energy, and fluid in the 3D virtual space within a unit space by performing a second data preprocessing on information about the 3D virtual space and sensed data It. This can be created in parts. For example, the time interval can be a preset time interval.

[0115] For example, based on the flow vectors input to the second CNN model for each time interval, a second spatial feature vector for each time interval can be output. For instance, the server can output the second spatial feature vector for each time interval by inputting the flow vectors for each time interval into the second CNN model.

[0116] The second CNN model may include a second input layer, one or more second hidden layers, and a second output layer. Learning data, consisting of multiple flow vectors and multiple correct answer second-space feature vectors, is input to the second input layer, passes through one or more second hidden layers and a second output layer, and is output as an output vector. This output vector serves as the input to a second loss function layer, which is connected to the output layer. The second loss function layer uses a second loss function to output a second loss value. This second loss function compares the output vector with the correct answer vector at each training iteration, and the parameters of the CNN model in the second loss function layer are used to minimize the first loss value. It can learn directions.

[0117] A: The second spatial feature vector can be a vector representing the spatial features of a flow vector composed of learned data. For example, the correct answer is that the second spatial feature vector includes the density value of the moving body, the density value of the energy, the density value of the fluid, the positional correlation value between the moving body and the object, and the density values ​​of the moving body and the energy. It can include the positional correlation values ​​between the fluid and the positional correlation values ​​between the energy and the fluid.

[0118] For example, the density value of a moving object can represent the density of moving objects in space. Similarly, the energy density value can represent the density of energy in individual spaces. The fluid density value can represent the density of fluid in space. The positional correlation between a moving object and energy can represent the spatial correlation between the moving object's location and the energy consumption point. The positional correlation between a moving object and fluid can indicate the spatial correlation between the moving object's location and the space where the fluid is distributed. The positional correlation between energy and fluid can represent the spatial correlation between the energy consumption point and the fluid distribution space. For example, correlation can be defined as a value greater than 0 and less than 1. In this case, the closer to 1, the higher the correlation.

[0119] One or more second hidden layers may include one or more convolutional layers and one or more pooling layers. For example, one or more second hidden layers may learn cell-based spatial patterns through one or more convolutional layers and one or more pooling layers. For example, in the first layer, you learn basic relationships (e.g., distances between moving objects, energy, and fluids in space), and in later layers, you learn higher-dimensional relationships (e.g., correlations, directions, etc., between moving objects, energy, and fluids).

[0120] For example, multiple flow vectors can be filtered within convolutional layers, and feature maps can be formed through these layers. For instance, convolutional layers can divide flow vectors into blocks of a preset size and learn local patterns between each object and its location coordinates. For example, after performing convolution operations in a convolutional layer, a rectified linear unit (ReLU) function can be applied. This allows the non-linear characteristics of the flow vectors to be represented.

[0121] For example, by selecting a fixed vector related to the feature based on the feature map formed in the pooling layer for dimensionality reduction, and subsampling the formed feature map, the correct second-space feature vector can be obtained from the vectorized data. For example, the pooling layer could be a max-pooling layer that extracts the maximum value. Alternatively, the pooling layer could be an average-pooling layer that extracts the average value. In this case, the parameters of the second CNN model could include parameters related to the convolutional and pooling layers (feature map size, filter size, depth, stride, zero-padding).

[0122] For example, by preprocessing information about autonomous actuators, an actuator vector can be generated, which includes values ​​related to the autonomous actuator type, the location coordinates of the autonomous actuator, and values ​​related to the operation of the autonomous actuator. For instance, a server can generate an actuator vector by preprocessing information about autonomous actuators.

[0123] The value for the type of autonomous actuator can include any one of several values ​​for the type of autonomous actuator. The position coordinates of the autonomous actuator can include the identification value of the space in which the autonomous actuator is located and the three-dimensional coordinates of the autonomous actuator within that space. Values ​​related to the operation of the autonomous actuator can include the setpoints required for the operation of the autonomous actuator and the average hourly energy consumption of the autonomous actuator.

[0124] For example, basic control information can be determined based on the composite feature vector input to the bidirectional LSTM model and the actuator vector, which combines the first and second spatial feature vectors for each time period. For instance, the server can obtain basic control information by generating a composite feature vector (combining the first and second spatial feature vectors) for each time period and inputting the composite feature vector and the actuator vector into the bidirectional LSTM.

[0125] A bidirectional LSTM model can include a third input layer, one or more third hidden layers, and a third output layer. For example, one or more third hidden layers may include one or more forward LSTM blocks and one or more backward LSTM blocks, and each LSTM block may include a storage unit, an input gate, a forget gate, and an output gate. A storage unit is a node that outputs the result of an activation function. Storage units can perform recursive operations, using the value output by the storage unit in the previous iteration as the input at the current time. For example, at time t, the value output by a storage unit at time t may be influenced by the values ​​of past storage units. Storage units can output a cell state (Ct) value and a hidden state (ht) value. That is, the memory cell can use the cell state value (Ct-1) and hidden state value (ht-1) passed by the memory cell at time t-1 as input values ​​to calculate the cell state value and the hidden state value at time t.

[0126] For example, a forward LSTM module can process the integrated feature vectors in chronological order to understand how past data influences the present and future. A backward LSTM block reverses the chronological order and reflects the current-future integrated feature vectors, allowing the present to consider future scenarios to evaluate the optimal settings for each condition. In this way, the gradient loss problem can be reduced while maintaining non-linearity. For example, high-dimensional feature vectors can be generated using forward and backward LSTM blocks to compute motion scores for each condition based on the features of each time period and the circumstances of each condition. For example, a high-dimensional feature vector might include the state value and the rate of change over time for each variable.

[0127] For example, the output layer calculates the operation score for each of the multiple settings for each condition based on the high-dimensional feature vector, and determines the setting with the highest operation score as the target setting for that condition.

[0128] For example, control indicators for each type of autonomous actuator can be preset on the server. For instance, the weights and rates of change for each control indicator can be pre-set on the server. Weights can be set to values ​​greater than 0 and less than 1.

[0129] In this way, the degree of conformity with the target state can be reflected by using the difference between the absolute values ​​of the motion values, and the stability of the motion can be determined by considering the rate of change.

[0130] In other words, the server inputs learning data, consisting of multiple integrated feature vectors, multiple actuator vectors, and multiple basic control information for correct answers, into the third input layer and one or more third hidden layers and third output layers of the bidirectional LSTM model. A third output vector is then output, which is fed into the third loss function layer connected to the third output layer. The third output vector and the third loss function layer are used for each training iteration. Data is input through the third loss function layer. The loss function outputs a third loss value, and the parameters of the bidirectional LSTM model are trained in the direction that the third loss value decreases.

[0131] Basic control information may include multiple condition values, target setpoints for each condition, and values ​​for control indices. For example, condition values ​​may include lower and upper limits for temperature, humidity, lighting, air quality, noise, and the number of occupants. Target setpoints may represent the target settings for the autonomous actuator under the corresponding conditions. For example, target setpoints may differ for each type of autonomous actuator. For example, in the case of lighting, the target setpoint may be a target illuminance value. For example, in the case of air conditioning or heating, the target setpoint may be a target temperature value. For example, in the case of a ventilation system, the target setpoint may be a target ventilation rate. For example, in the case of a camera device, the target setpoint may be a target position value. For example, in the case of a robot, the target setpoint may include at least one of a target position value or a target velocity value. Values ​​for control indices may be values ​​of specific actions or operational intensities that must be performed to achieve a target state.

[0132] Based on information about multiple preset virtual scenes, information about 3D virtual space, sensor data, and basic control, the server provides information about each of the multiple preset virtual scenes using a spatial context response model of a second neural network. The corresponding information can then be determined.

[0133] For example, a simulation vector for each of the multiple preset virtual scenes can be generated by preprocessing information from multiple preset virtual scenes and information from three-dimensional virtual space.

[0134] Information about the hypothetical scenario may include values ​​for the initial state and values ​​for the changing conditions. For example, values ​​for the initial state may include values ​​for temperature, humidity, lighting, air quality, and noise in each external and internal space of the facility, as well as values ​​for the occupant distribution in each internal space of the facility. Values ​​for the changing conditions may include the rate of increase or decrease in external temperature, humidity, illuminance, air quality, and noise, the rate of increase or decrease in the number of occupants, or it may at least include the location of the fire and the simulation execution time.

[0135] For example, a server can virtually deploy autonomous actuators and multiple pre-installed sensors in the three-dimensional virtual space of a facility based on information about the virtual space. For instance, a server can perform a simulation based on information about the virtual scene and basic control information within a three-dimensional virtual space containing autonomous actuators and multiple pre-installed sensors.

[0136] Alternatively, for example, simulations can be performed using spatial situation prediction models. In other words, simulation vectors can be created using spatial condition prediction models.

[0137] Specifically, for example, the server can generate an initial vector including initial state values, changing condition values, and simulation execution time values ​​by preprocessing information about the virtual scene. For example, the server can generate a state vector including the position coordinates of the autonomous actuator and multiple pre-installed sensors by preprocessing information about the three-dimensional virtual space. For example, the server can generate a setup vector including multiple condition values ​​and target settings for each condition by preprocessing basic control information.

[0138] For example, a server can generate simulated vectors by inputting initial vectors, state vectors, and setup vectors into a spatial situation prediction model. For instance, the bidirectional LSTM model described above can be used as a spatial situation prediction model.

[0139] For example, multiple high-dimensional prediction vectors can be generated using forward LSTM blocks and backward LSTM blocks based on the initial vector, state vector, and setup vector for each time interval. For instance, a high-dimensional prediction vector might include candidate change values ​​and the rate of change over time for each variable at each time interval.

[0140] The simulation vector can include the changes in temperature, humidity, illuminance, air quality, and noise in the facility's external and internal spaces over each time period, as well as the changes in their distribution over each time period. It can also include the occupants of each space within the facility.

[0141] In addition, the output layer of the spatial context prediction model can calculate the prediction score of each of multiple high-dimensional prediction vectors and determine the high-dimensional prediction vector with the highest prediction score as the simulation vector.

[0142] For example, the response information of each of the multiple preset virtual scenarios can be determined based on the simulation vector, integrated feature vector, and actuator vector of each of the multiple preset virtual scenarios input into the spatial context response model.

[0143] Transformer models can be used as spatial context response models. For example, a Transformer model can process information for each time period through multiple layers of encoders and decoders, generating appropriate information for the given context. The Transformer's encoder learns the relationships between input vectors, and the decoder outputs the optimal corresponding information based on the learned information.

[0144] For example, a spatial context response model may include multiple encoder layers and multiple decoder layers.

[0145] For example, the encoder layer can convert the input analog vector, the synthesized feature vector, and the actuator vector into a context vector.

[0146] For example, an encoder layer may include an input embedding layer, a multi-head self-attention layer, and a feedforward layer.

[0147] For example, the input embedding layer can convert analog vectors, ensemble feature vectors, and actuator vectors into embedding vectors.

[0148] For example, the embedding vector E can be determined as X·WE. Here, WE can be the embedding weight matrix. For example, the embedding weight matrix can be determined with initial random values ​​and then adjusted through backpropagation during the learning process. That is, the embedding weight matrix can be updated while learning the relationship between a specific input and output. For example, the embedding vector can be a 512-dimensional vector.

[0149] For example, a multi-head self-attention layer can determine the weights of elements in an embedding vector. For example, a multi-head self-attention layer can compute the weights of elements in an embedding vector through attention operations. For example, a multi-head self-attention layer can determine attention weights by generating query vectors, key vectors, and value vectors. In this case, the multi-head self-attention layer can learn multiple relationships in parallel using multiple attention heads. For example, a query vector Q is a vector used to determine the importance of specific input data and can be generated by applying the weight matrix WQ of the query vector to the embedding vector E. For example, a key vector K is a vector indicating the relationship (i.e., similarity) between each input data and other input data and can be generated by applying the weight matrix WK of the key vector to the embedding vector E. For example, a value vector V is a vector that returns highly relevant data and represents key information and can be created by applying the weight matrix WV of the value vector to the embedding vector E. For example, like the embedding weight matrix, the weight matrix for each query, key, and value can be determined as initial random values ​​and then adjusted through backpropagation during the learning process. Each attention head can output a vector Z, which is the value obtained by applying the attention weights to the value vector. In other words, a multi-head attention layer can output a multi-head attention result H. h is the number of attention heads, and WO can be the integral weight matrix. Similar to the embedding weight matrix, the ensemble weight matrix can be determined with initial random values ​​and then adjusted via backpropagation during the learning process. `concat()` can be an operation that joins two or more data sets.

[0150] For example, a feedforward layer can generate a context vector by performing a nonlinear transformation on the computation results of a multi-head self-attention layer. For instance, the context vector F can be used to determine the feedforward layer. The bias vector is a learnable parameter and can be a vector added to the output of each node to output a specific value, even if the input data is 0. ReLU can be a rectified linear unit (ReLU) function.

[0151] For example, the decoder layer can determine the relevant information based on the initial values ​​in the simulator vector and the context vector.

[0152] For example, a decoder layer may include an input embedding layer, a masked multi-head self-attention layer, an encoder-decoder attention layer, and a feedforward layer. For instance, the input embedding layer may embed initial values ​​into an analog vector and use it as the input vector. For example, the decoder's embedding weight matrix may be determined with initial random values ​​and then adjusted via backpropagation during the learning process. That is, as the relationship between a specific input and output is learned, the decoder's embedding weight matrix can be updated.

[0153] For example, a masked multi-head self-attention layer can predict the next state based solely on current time and past information. By masking, it prevents the referencing of data from later points in time, allowing predictions to be made only based on the current time and past data. In this case, multiple attention heads can analyze the data from various perspectives, identifying relationships and understanding which elements are important for prediction. For instance, a masked multi-head self-attention layer can determine the masking attention weights by generating the decoder's query vector, key vector, and value vector.

[0154] For example, an encoder-decoder attention layer can learn the relevance to simulated data by combining the decoder's input vector with the encoder's context vector. For instance, the encoder-decoder attention layer can use the context vector generated by the encoder and the current decoder's input vector to compute encoder-decoder attention weights.

[0155] For example, a feedforward layer can enhance the applicability of the information by performing a nonlinear transformation on the results of the operations of the masked multi-head attention layer and the encoder-decoder attention layer. For instance, a feedforward layer can generate Hi, a vector that combines the results of the operations of the masked multi-head attention layer and the encoder-decoder attention layer.

[0156] For example, the corresponding information for each of multiple preset virtual scenarios may include the target state value of the autonomous actuator, the value of the control index of the autonomous actuator, and the value of the operation duration of the autonomous actuator. For example, the corresponding information may vary depending on the type of autonomous actuator. The target state value of the autonomous actuator may be the value of the environmental state that the autonomous actuator must achieve. The value of the control index of the autonomous actuator may be the value of the specific action or the intensity of the operation that must be performed to achieve the target state. The value of the operation duration of the autonomous actuator may be the time it must maintain operation to achieve the target state. For example, the value of the operation duration of the autonomous actuator may include at least one of the following: the time until the target state is achieved, a specific time value, or a value of periodic operation. The value of periodic operation may include the value of operation time and the value of downtime.

[0157] For example, under lighting conditions, the target state value of the autonomous actuator is 500 lux illuminance, the control index value of the autonomous actuator is 70% luminance and color temperature of 3500K, and the operation duration of the autonomous actuator can be 1 hour.

[0158] For example, in the case of a ventilation system, the target state values ​​of the autonomous actuator are the values ​​of carbon dioxide concentration and harmful substance concentration, the control index values ​​of the autonomous actuator are the values ​​of ventilation volume and fan speed, and the value of the autonomous actuator's operating duration is the value of periodic operation, which may be 10 minutes to turn on and 5 minutes to turn off.

[0159] For example, for a heater or air conditioner, the target state value of the autonomous actuator is 22 degrees Celsius, the control index value of the autonomous actuator is 70% of the output intensity, the value of the blowing phase, and the value of the operating duration of the autonomous actuator can be 1 hour.

[0160] For example, for a camera, the target state value of the autonomous actuator is the value at a specific location, the control index value of the autonomous actuator is the value of resolution and magnification, and the operating duration of the autonomous actuator is the value of resolution and magnification, which may be 6 hours.

[0161] For example, in the case of a robot, the target state value of the autonomous actuator is the value of a specific position, the control index value of the autonomous actuator is the value of the moving speed and any one of the values ​​in multiple modes, and the value of the operating duration of the autonomous actuator can be 2 hours.

[0162] The server can send configuration information, including basic control information and response information for each of multiple preset virtual scenarios, to the autonomous actuator.

[0163] The configuration information can be used to configure the autonomous actuator.

[0164] For example, when the autonomous actuator receives the value of the basic mode from the server, it can operate based on basic control information. For instance, the default mode might be how the autonomous actuator operates based on real-time data about the current state of space. For example, the basic mode could be a mode that does not correspond to multiple hypothetical scenarios, i.e., a mode that operates under normal conditions.

[0165] For example, when an autonomous actuator receives a scene pattern value from a server, it can operate based on the corresponding information in the corresponding information of each of multiple preset virtual scenes that matches the value of the scene pattern. For example, a scene pattern could be a way in which the autonomous actuator operates in response to a preset hypothetical scenario (e.g., an emergency, an expected environmental change). For example, a scene pattern could be a mode in which the autonomous actuator operates based on response information previously generated by the server. For example, the server can determine, based on information from multiple preset virtual scenes, whether there exists a virtual scene with a similarity greater than or equal to a preset similarity to the current spatial state. For example, if a virtual scene with a similarity higher than the preset similarity exists, the server can generate a value indicating the virtual scene and send this value to the autonomous actuator. In this case, the value indicating the virtual scene could be the value of the scene pattern. For example, the autonomous actuator can determine the response information matching the corresponding virtual scene based on the value indicating the corresponding virtual scene in the corresponding information of each of multiple preset virtual scenes, and operate based on the existing corresponding information.

[0166] Figure 4 This is a diagram illustrating the spatial concepts of objects included in a traffic model of CityGML (City Geographic Markup Language) according to an embodiment. Figure 5 This is a diagram illustrating multiple moving objects via cognitive digital twins at a 3D virtual city center intersection according to an embodiment. Figure 4 and Figure 5 The embodiments can be combined with various embodiments of this disclosure.

[0167] See Figure 4 CityGML's traffic models can represent traffic-related objects such as roads, railways, and plazas, and can include the geometric and semantic features of traffic networks. CityGML's traffic models support various levels of detail (LoD), allowing for expressions ranging from simple two-dimensional to complex three-dimensional representations as needed. CityGML's traffic models can also include additional information, such as the surface materials, signage, and lighting of traffic facilities. CityGML's traffic models can be integrated with other CityGML modules to build comprehensive city models. CityGML's traffic models can be used in various fields, including urban planning, traffic simulation, and navigation system development.

[0168] In CityGML, the spatial concepts of traffic objects mainly include Traffic Area, Traffic Space, and Clearance Space. Traffic Area represents the surface and detailed components of a road, Traffic Space represents the actual space vehicles can travel in, and Clearance Space represents the safety clearances required for traffic flow. This allows for the monitoring of traffic objects using security camera footage. For example, traffic areas can be monitored by matching the video from security cameras with a virtual space created through CityGML.

[0169] The server can obtain spatial data of intersections in the city center, as well as information about multiple security camera terminals at the city center intersections.

[0170] The server represents multiple moving objects in a 3D virtual city center intersection based on CityGML, using images received from multiple security camera terminals. It determines the movement information of these moving objects and manages them based on this information. For example, the server can identify abnormal moving objects among the multiple moving objects represented at the 3D virtual city center intersection and send information about these abnormal objects to an administrator terminal. The server may also include... Figure 1 Server 108.

[0171] An administrator terminal can be a server-implemented user terminal for managing a downtown intersection. For example, the administrator terminal can output images of a 3D virtual downtown intersection provided by the server to its own screen, i.e., a monitor. For example, the administrator terminal can display moving objects on the 3D virtual downtown intersection displayed on its screen based on information about moving objects provided by the server. For example, the administrator terminal can be pre-connected to the server. For example, the administrator terminal can be... Figure 1 Electronic equipment 101.

[0172] Multiple security camera terminals can be camera terminals installed at multiple locations within a downtown intersection. For example, security camera terminals can capture still images and images of specific areas to monitor specific areas within a downtown intersection. For example, a security camera terminal may include a processor, memory, a communication module, one or more lenses, an image sensor, an image signal processor, or a flash.

[0173] Intersection spatial data can include topographic data, infrastructure data, and traffic data. For example, intersection spatial data can be collected from public data portals, geographic information systems, or map services. Topographic data may include data on the elevation, surface materials, and road structure of a downtown intersection. For example, topographic data may include the height of the intersection and surrounding intersections, materials and pavement conditions (e.g., asphalt and concrete), road width, the distinction between roads and sidewalks, and the location of pedestrian crossings. Infrastructure data may include data on traffic-related objects such as traffic lights, streetlights, and signs installed at downtown intersections. Traffic data may include data on lanes, signaling systems, and traffic flow at downtown intersections. For example, traffic data includes the number of lanes, lane width, lane spacing information, lane changing permissions, signal waiting times, signal interval data, vehicle movement and average speed by time period, intersection congestion, speed limits, traffic conditions, etc. It may include the frequency, type, and causes of accidents.

[0174] Information about multiple security camera terminals may include internal information about each of the multiple security camera terminals and external information about each of the multiple security camera terminals.

[0175] The internal information is about the lens and sensor characteristics of the security camera terminal, and may include, for example, the focal length and distortion coefficient of the security camera terminal's lens, as well as the optical center of the security camera terminal.

[0176] External information is information about the location and orientation of the surveillance camera terminal, such as the three-dimensional coordinates and rotation matrix of the surveillance camera terminal (e.g., horizontal rotation (Pan), vertical rotation (Tilt), rotational motion (Roll)).

[0177] For example, the server can acquire pre-stored intersection spatial data and information from multiple surveillance camera terminals. Alternatively, the server can receive cross-spatial data and information about multiple camera terminals from external devices.

[0178] The server can create 3D virtual city center intersections using CityGML based on intersection spatial data.

[0179] According to one embodiment, the server can set multiple spatial concepts for each LOD (Level of Detail) based on intersection spatial data using the CityGML traffic model.

[0180] Multiple spatial concepts can include traffic areas, secondary traffic areas, intersection areas, traffic spaces, safety spaces, and urban facilities.

[0181] Traffic zones represent the road surface and can specifically define lanes, pedestrian crossings, and parking areas. For example, a traffic zone defines the area, surface material (e.g., asphalt, concrete), and lane type (e.g., solid lines, dashed lines, road arrows) of lanes, pedestrian crossings, stop lines, and center lines. The length and width of pedestrian crossings can be included as attribute values. For example, the area of ​​each lane within a traffic zone can be modeled as a polygon, and the lane's direction, type, and material can be assigned as separate attributes. For example, stop lines and center lines can be represented as linear boundaries (LineString), and the type and color of the road lines can be set as attributes.

[0182] Auxiliary traffic areas represent auxiliary spaces such as sidewalks, bike lanes, and road shoulders, and can include attribute values ​​such as sidewalk width and length, surface material, and whether there are ramps. For example, pedestrian and bike lanes can be represented as polygons separate from regular roads.

[0183] An intersection area represents the space where roads intersect as a polygon and can include intersection type (e.g., intersection, T-junction, roundabout), traffic volume, and signal cycle as attribute values. For example, defining an intersection area as a polygon and connecting adjacent traffic areas to the intersection center forms a traffic network relationship.

[0184] Traffic space is a spatial concept reflecting the elevation values ​​within a traffic area. It expresses the actual driving space in three dimensions, including attributes such as the number of lanes, lane width, speed limit, and vehicle type. At this point, each lane can be defined in detail according to the traffic area. For example, after establishing traffic space, the traffic area can be integrated into its management.

[0185] Safe space represents the free space and safe zone required for traffic flow, and can include the minimum distance between vehicles and surrounding structures (e.g., lateral clearance), height restrictions, and the location and coordinates of boundary areas as attribute values. For example, the spatial relationship between safe space and traffic space can be defined.

[0186] Urban facilities refer to facilities (traffic lights, signs, etc.) installed on roads and at intersections. These facilities can include attribute values ​​such as the location, height, signal status, type and installation height of traffic signs, and the location and brightness of streetlights. For example, each facility can be defined as a point or entity object and can be assigned location coordinates and attribute values.

[0187] For example, topological relationships can be established between multiple spatial concepts. Traffic spaces can be connected to traffic areas, traffic areas can be connected to urban facilities, and intersection areas can be connected to traffic areas. In this way, by establishing nodes and links between roads, lane connections and road entrance and exit directions can be defined.

[0188] For example, spatial data and attribute data of multiple spatial concepts can be integrated and stored in XML format.

[0189] For example, in LOD 0, each road within a traffic area, pedestrian and bicycle lanes within a secondary traffic area, and intersections within an intersection area can be set as two-dimensional boundaries and polygons. In this case, lanes and roads can be represented by linear expressions, and the areas of roads and intersections can be represented by area expressions. For instance, an intersection can be represented as a polygon independent of the road. That is, intersections and roads can be distinguished by polygons.

[0190] The server uses CityGML to model intersections, using a first color to represent road segments within an intersection area and a second color to represent the same road segments. For example, the first and second colors can be set to different colors; for instance, the first color could be brown and the second color could be blue.

[0191] In addition, the server can store and manage the semantic spatial information of road segments using two representation methods: regional representation and linear representation.

[0192] For example, in LOD 1, traffic space is set to reflect the height of roads within traffic areas, and the space actually available to vehicles is expressed as a three-dimensional block. Pedestrian and bicycle lanes in secondary traffic areas and intersections in intersection areas can also be represented as three-dimensional blocks that reflect height values.

[0193] For example, in LOD 2, road components are represented on the road surface in the traffic area, slope, width, surface, and material represent pedestrian and bicycle lanes in the secondary traffic area, and intersections are represented in the intersection area. This expresses lane connectivity and the composition of intersections, represents the available space for each lane of the road in the traffic space, and allows setting upper limits for road and intersection heights and safety margins.

[0194] The components of a road can include elements that make up the road, such as lanes, center lines, stop lines, and pedestrian crossings. Lane connections include merging of multiple lanes into one lane, branching of a lane into multiple lanes, intersections where two lanes intersect, parallel lanes, U-turns, left turns, right turns, and lanes can include exits at the end of a new lane or entrances at the beginning of a new lane. The components of an intersection can include elements that make up the intersection, such as lanes, center lines, stop lines, pedestrian crossings, traffic islands, and lane-changing prohibition lines.

[0195] For example, in LOD 3, the texture, curvature, and slope of each road in the traffic area, pedestrian and bicycle lanes in the secondary traffic area, and intersections in the intersection area are represented, along with physical obstacles that can be represented in a safe space.

[0196] Physical barriers may include physical boundary elements such as curbs, obstacles, and railings.

[0197] Additionally, for example, at LOD 3, maintenance elements are represented in the traffic area, the infrastructure of each pedestrian and bicycle lane in the secondary traffic area and the intersection in the intersection area, as well as the vehicle type in the safety space. Height limits and margins can be represented using traffic space, and lane restriction elements can also be represented using traffic space.

[0198] Maintenance factors may include maintenance-related factors such as road surface cracks, wear, potholes, drainage conditions, and line wear. Infrastructure can be road infrastructure such as power lines, communication lines, drainage ditches, and underground passages. Lane restriction factors may include factors that restrict lane access, such as accidents, construction, vehicle type restrictions, and time zone restrictions.

[0199] For example, each of multiple spatial concepts can be selectively implemented as one of LOD 0 to LOD 3. For instance, the server can determine one of LOD 0 to LOD 3 for each of the multiple spatial concepts and use the determined LOD to implement the multiple spatial concepts, thereby creating a 3D virtual city center intersection. In this way, the server can control the detailed representation of each of the multiple spatial concepts. Alternatively, for example, the server can dynamically determine the LOD of each of the multiple spatial concepts based on the data throughput of each spatial concept's importance, and use the determined LOD to implement the multiple spatial concepts. This allows control over the detailed representation of the 3D virtual city center intersection.

[0200] Reference Figure 5 The server can use digital twins to identify moving objects and manage events. In other words, the server can construct a digital twin of a downtown intersection and then store and manage this virtual space as CityGML. For example, three security cameras cover areas A, B, and C. The server records the position and movement of moving objects such as vehicles, pedestrians, and bicycles within areas A, B, and C, and can save and manage direction and speed in real time. For example, the server can individually store instances where the position of each moving object deviates from permitted spatial areas (e.g., pedestrian walkways, roads, bike lanes) as specific events and report them to the administrator terminal. For example, information about previously stored moving objects can be reconstructed along a timeline. For example, information about stored moving objects can be used for simulation analysis.

[0201] According to one embodiment, the abnormal signal based on the second neural network analyzes the interaction between multiple moving objects and their compliance with traffic rules based on the movement information of multiple moving objects and spatial information about a 3D virtual city center intersection. It can determine the judgment model, the abnormal symptom object, and the abnormal symptom type of the abnormal symptom object.

[0202] According to embodiments of this disclosure, a system for sharing building space information based on a blockchain network includes: multiple provider nodes providing building space information; and at least one provider node managing a global model for federated learning related to digital twins. Among the manager node and the multiple provider nodes, at least one learner node may be included to participate in the federated learning related to digital twins. In this case, the multiple provider nodes and at least one manager node can be configured with a blockchain network and an IPFS network. For example, the multiple provider nodes and at least one manager node can participate in the blockchain network and utilize the IPFS network to store and share building space information. The federated learning related to digital twins can be federated learning utilized in the implementation or optimization of digital twin technology. For example, through federated learning related to digital twins, the digital twin of each building can analyze space utilization data and learn a global space optimization model through federated learning. For example, in federated learning related to digital twins, the digital twin of each building learns energy usage data within the building to determine optimized energy management strategies, and integrates and learns the strategies of each building through federated learning to create the overall network's energy efficiency. Improvements are possible. For example, federated learning related to digital twins uses digital twins to analyze the space utilization and movement patterns of buildings in cities. It can also derive the optimal spatial arrangement among multiple buildings. For instance, in federated learning related to digital twins, digital twins learn structural data from buildings to simulate disasters such as earthquakes and fires. Through federated learning, disaster response scenarios in various environments can be integrated to prevent global disasters. Response models can be built. In this way, federated learning involving various types of digital twins can be conducted.

[0203] For example, a provider node could be a node that stores building space information and provides that information to the system. For instance, a node could include... Figure 1 Electronic devices 101, smartwatches, laptops, desktops and Figure 1Server 108. Nodes can be referred to by other terms such as device, apparatus, terminal, user equipment (UE), mobile station (MS), wireless device, or handheld device. For example, a provider node may include a terminal used by a building owner, a terminal used by a building manager, and a terminal used by a building user. For example, a provider node may provide building space information to the blockchain network and receive compensation for providing building space information.

[0204] For example, a learner node can be a node among the provider nodes participating in federated learning related to digital twins. For instance, a learner node participates in federated learning related to digital twins, providing the results learned based on its local data to the manager node via the IPFS network, and earns rewards for contributing to the learned results based on its local data.

[0205] For example, learner nodes register in an IPFS network, share a group key, and can access each other's shared storage on the IPFS network using the shared group key. The group key can be an encryption key used to control and secure data access within the IPFS network.

[0206] For example, a learner node can create a local model by storing the current model and its parameters from the IPFS network and learning the current model based on the local dataset. Here, the current model is a pre-learned model used to create the local model; it could be the initial global model stored in the IPFS network when the management node registers federated learning related to the digital twin or when the management node performs federated learning. It may be a global model that is continuously updated in the process.

[0207] Learner nodes can update their local models based on parameters from the global model on the manager node. The amount of local data shared by learner nodes with other nodes may vary depending on the type of federated learning.

[0208] For example, an administrator node could be a node that manages and deploys a global model of federated learning related to digital twins. For instance, a manager node could include a server utilizing cognitive digital twin technology.

[0209] For example, administrator nodes can manage the storage of data and the addition, modification, deletion, and registration of computing resources within the blockchain network. Administrator nodes can provide authentication and network structure settings related to nodes in the blockchain network. Administrator nodes can periodically measure the performance of the blockchain network, such as TPS, block creation time, block confirmation time, and network availability. Administrator nodes provide and can modify the consensus algorithm of the blockchain network. For example, administrator nodes can use a proof-of-authority consensus algorithm or a Byzantine Fault Tolerance (BFT) consensus algorithm. Administrator nodes can add, delete, and set permissions for terminals participating in the blockchain network.

[0210] For example, a manager node can manage communication protocols implemented in the form of smart contracts or chaincode. For instance, a manager node can handle communication protocols between blockchain networks and perform normal operational tasks such as link establishment, synchronization, transmission control, and error control.

[0211] For example, blockchain networks may have smart contracts, which are program codes that run automatically when certain conditions are met. Through these smart contracts, contract execution and verification can be automated. For instance, a blockchain network can create transactions to manage building space information through smart contracts. For example, when certain conditions are met, a blockchain network can use a smart contract to allow access to building space information for a specific terminal.

[0212] For example, a management node can upload a global model and its parameters to the IPFS network for federated learning related to digital twins. For example, a management node can receive the CID of the global model from the IPFS network. For example, an administrator node can deploy a second smart contract to the blockchain network, containing information about the global model for federated learning related to digital twins. Information about the global model for federated learning related to digital twins can include link information to the global model stored in the IPFS network, the maximum number of learner nodes that can participate, and the participation conditions for federated learning related to digital twins. At this point, the CID of the global model executing the federated learning related to digital twins can be transmitted to the IPFS network along with the maximum number of learner nodes that the second smart contract can participate in. Then, for example, the manager node can update the global model through the IPFS network based on the parameters of the local models generated by the learner nodes. For example, the manager node can collect the parameters of the local models created by the learner nodes through the IPFS network and update the global model with the collected data.

[0213] The above embodiments can be implemented using hardware components, software components, and / or combinations of hardware and software components. For example, the apparatus, methods, and components described in the embodiments may include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, and field-programmable gate arrays (FPGAs). They can be implemented using one or more general-purpose or special-purpose computers, such as arrays, programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, it may be described as using a single processing device; however, those skilled in the art will understand that a processing device may include multiple processing elements and / or various types of processing elements. For example, a processing apparatus may include multiple processors or one processor and one controller. Furthermore, other processing configurations, such as parallel processors, are also possible.

[0214] Software can include computer programs, code, instructions, or combinations thereof, which can configure processing units to operate as needed, or can independently or jointly command devices. Software and / or data can be used on any type of machine, component, physical device, virtual device, computer storage medium, or device to be interpreted by or to provide instructions or data to processing devices, or can be permanently or temporarily embodied in transmitted signal waves. Software can be distributed across networked computer systems and stored or executed in a distributed manner. Software and data can be stored on one or more computer-readable recording media.

[0215] The method according to the embodiments can be implemented in the form of program instructions executable by various computer devices and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., individually or in combination. The program instructions recorded on the medium may be specifically designed and configured for the embodiments, or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; and magnetic media such as floppy disks—including optical media (magneto-optical media). Hardware devices specifically designed for storing and executing program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as code generated by a compiler, and high-level language code that can be executed by a computer using an interpreter. The aforementioned hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0216] Although embodiments have been described above using limited accompanying drawings, those skilled in the art can make various modifications and variations based on the above application of techniques. For example, the described techniques may be performed in a different order than the described methods, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described methods, or other components may be substituted or replaced with equivalents, and appropriate results may be obtained.

[0217] Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

1. A method for operating an autonomous actuator using a cognitive digital twin on a server, comprising: Based on the multi-dimensional facility information of each of the multiple facilities, a three-dimensional virtual space is realized for each of the multiple facilities; Multidimensional facility information includes information on multiple planes. Each plane contains facility object information and facility spatial information. The facility object information includes multiple structural information and multiple facility resource information. The facility spatial information includes multiple vector space information and multiple unit space information. A cell matrix is ​​set for each of the multiple planes, and a cell space list is set for each of the multiple structural information, multiple facility resource information, and multiple vector space information based on the cell matrix. The three-dimensional virtual space of each of the multiple facilities is represented as a vector space and a cellular space. Based on information about multiple sensors pre-installed in each of multiple facilities and multi-dimensional facility information, multiple pre-installed sensors are arranged in a three-dimensional virtual space for each of the multiple facilities; Collect sensor data from multiple pre-installed sensors; Based on information about the autonomous actuator, information about the three-dimensional virtual space, and sensor data, a first neural network is used to determine basic control information through a spatial context inference model. The spatial context response model using the second neural network determines the response information based on information about multiple preset virtual scenarios, information about the three-dimensional virtual space, sensor data, information about the autonomous actuator, and basic control information for each of the multiple preset hypothetical scenarios. as well as The system sends settings information, including basic control information and corresponding information for each of the multiple preset virtual scenarios, to the autonomous actuator. Autonomous actuator, When the value of the basic mode is received from the server, the operation is performed based on the basic control information. When the value of the scene mode is received from the server, the operation is performed based on the corresponding information of each scene. Multiple preset virtual scenes operate according to the corresponding information that matches the value of the scene mode.

Images