Method and system for physical ai based simulation

Abstract

The present invention relates to a simulation method and system, and more specifically to a technology that provides a simulation method and system for implementing physical AI.

Description

Technology Field

[0001] The present invention relates to a simulation method and system, and more specifically to a technology that provides a simulation method and system for implementing physical AI. Background Technology

[0002] Recently, automated operations utilizing multiple robots are expanding in smart logistics, smart factory, and autonomous factory environments, and the importance of simulation technology is increasing to verify performance differences resulting from robot operation strategies and spatial layouts in advance. In particular, in environments where autonomous vehicles, mobile robots, and work robots operate in combination, a process of analyzing interaction results based on spatial structure information and object placement information in advance is required.

[0003] While current simulation platforms provide various physics-based modeling and visualization capabilities, they often operate with the separation of environment layout design and control strategy application. This presents a limitation in that it entails the burden of reconfiguration when components are changed or strategies are modified. Consequently, temporal and procedural constraints may arise during the process of comprehensively verifying the correlation between environment configuration and control strategies.

[0004] Furthermore, while various studies are underway to expand into physical AI environments, there are instances where environment configuration, control strategy application, and execution result analysis are not organically integrated within simulation systems that reflect actual operating environments. In particular, despite the need to analyze performance changes based on robot movement status, workflow, resource allocation, and temporal operating conditions from multiple perspectives, users face limitations in comprehensively perceiving the overall operational status when simulation results are not provided in an integrated form combining intuitive visualization and quantitative performance indicators.

[0005] These constraints ultimately make simulation-based verification difficult in environments requiring rapid decision-making, such as physical AI.

[0006] Against this backdrop, there is a need for technology capable of overcoming resulting limitations by performing simulations at the level of implementing a physical AI environment and providing the execution results in an intuitive and integrated manner. The problem to be solved

[0007] The present invention is intended to provide a physical AI-based simulation method capable of reproducing robot operation interacting with a physical environment in a virtual space.

[0008] The present invention aims to provide a simulation method capable of verifying various operating conditions in a consistent structure, based on a data processing pipeline that links environmental information and control strategies.

[0009] The present invention aims to provide a simulation method that enables verification of multiple operating conditions by integrally reflecting environmental information and intelligent control logic.

[0010] The present invention aims to provide a new pipeline-based simulation method capable of comprehensively interpreting spatiotemporal changes and performance characteristics occurring during the operation of a robot.

[0011] The present invention is intended to provide a simulation method that can be utilized in environments requiring rapid decision-making. means of solving the problem

[0012] To solve the problem described above, the present invention presents a data processing method that executes a robot simulation based on an environment layout and a control strategy model and outputs the execution results.

[0013] Specifically, a physical AI-based simulation method according to the present invention, performed by a computer, may include the steps of: providing a first user interface for generating an environment layout related to robot operation; interacting at least one control strategy model with at least one environment layout generated by the first user interface; and executing a robot simulation for robot operation based on the interacted environment layout and the control strategy model, and outputting the execution result of the robot simulation in a second user interface.

[0014] In an embodiment, the method may further include the step of generating the environment layout based on the first user interface and the step of saving the generated environment layout in a file format.

[0015] In an embodiment, the generating step may include the step of setting a grid for a space related to robot operation, the step of placing at least one of a robot and a component related to robot operation on the space where the grid is set, and the step of setting operation parameters related to robot operation.

[0016] In an embodiment, the operational parameter may include a parameter that defines at least one of a workflow, resource allocation, and temporal operational conditions.

[0017] In an embodiment, the first user interface is provided with a dropdown menu, and the dropdown menu may be configured to select at least one of the robot and component and place it as a dropdown in the space.

[0018] In an embodiment, the robot operation includes the operation of an autonomous vehicle, and the environment layout may define an operation environment that includes at least one of spatial structure information and object placement information related to a plurality of autonomous vehicles.

[0019] In an embodiment, the step of interoperating may include selecting one of a plurality of control strategy models and interoperating the selected control strategy model based on at least one environment layout and standard interface generated by the first user interface.

[0020] In an embodiment, the second user interface may be provided with a selection area for selecting any one of the plurality of control strategy models.

[0021] In an embodiment, the second user interface may include an animation display area that displays the movement state of the robot in an animation form.

[0022] In an embodiment, the second user interface may further include an indicator display area that displays performance indicators related to the operation of the robot.

[0023] In an embodiment, the control strategy model may include at least one of a rule-based algorithm, a reinforcement learning-based policy model, and a language model-based planner.

[0024] In an embodiment, the step of executing a driving decision of at least one robot based on the execution result of the robot simulation may be further included.

[0025] In an embodiment, the step of executing a driving decision of at least one robot may include the step of generating a plurality of candidate data in a search mode based on the execution result of the robot simulation, and the step of determining whether to terminate the search mode by comparing the candidate data with a preset transition specification and activating a decision mode to execute a driving decision of the robot.

[0026] In an embodiment, the control strategy model may include a spatial token generation module and a time token generation module that generate spatial features and time features by separating them from data.

[0027] In an embodiment, the control strategy model can perform an attention operation that applies a separate spatiotemporal attention mask to the space token and the time token.

[0028] In an embodiment, the control strategy model can perform rotation-based position encoding based on the local and global order of tokens.

[0029] In an embodiment, the control strategy model may include a mapping module that generates contextual spatial tokens by fusing structured location information and language information that specify spatial information.

[0030] In an embodiment, the mapping module can perform a cross-attention operation between a space token encoded with the location information and a language token encoded with the language information.

[0031] A physical AI-based simulation method according to the present invention, performed by a computer, may include the steps of: generating an environment layout related to robot operation; linking at least one control strategy model with the environment layout; separating and generating spatial features and temporal features from environment state data generated by the control strategy model based on the environment layout; interpreting spatiotemporal relationships between a robot and an object based on the spatial features and the temporal features; and executing a robot simulation based on the interpretation results and outputting the execution results of the robot simulation in a user interface.

[0032] A physical AI-based simulation system according to the present invention comprises one or more processors and a memory for storing instructions executed by the one or more processors, wherein the one or more processors provide a first user interface for generating an environment layout related to robot operation by executing the instructions, and at least one control strategy model interacts with at least one environment layout generated by the first user interface, and executes a robot simulation for robot operation based on the interacted environment layout and the control strategy model, and can output the execution result of the robot simulation through a second user interface.

[0033] An artificial intelligence inference method according to the present invention, which is performed by a computer, presents an artificial intelligence inference method capable of solving various problems through inference using a learned prediction model. More specifically, the artificial intelligence inference method according to the present invention may include the steps of receiving a prediction request, inputting the prediction request into a learned prediction model such that representation vectors corresponding to a plurality of attributes are reflected in an embedding, and generating a prediction result based on at least one specific attribute forming the prediction request by the prediction model.

[0034] In an embodiment, the representation vector corresponding to the plurality of attributes is generated during the learning process of the prediction model based on the difference between the first embedding data and the second embedding data, and the first embedding data and the second embedding data can each be generated by data distinguished according to whether or not at least one attribute is included.

[0035] In an embodiment, the second embedding data may be embedding data generated by inputting the data from which at least one attribute has been removed into the prediction model.

[0036] In an embodiment, the prediction model can be trained so that the representation vectors have similar representations in a vector space.

[0037] In the embodiment, the representation vector may be a vector corresponding to the same attribute included in a different combination of attributes.

[0038] In an embodiment, the representation vector can be combined as part of the embedding to form an embedding component corresponding to the specific attribute.

[0039] In an embodiment, the embedding may include a first embedding component representing a unique meaning corresponding to each attribute and a second embedding component related to the interaction of a plurality of attributes.

[0040] In an embodiment, the embedding may be formed by the combination of the first embedding component and the second embedding component.

[0041] In an embodiment, the prediction result may be generated based on the first embedding component corresponding to the specific attribute.

[0042] In an embodiment, even if the combination of attributes including at least one specific attribute is not used in the learning process of the prediction model, the prediction model can generate a prediction result based on a representation vector corresponding to the at least one specific attribute.

[0043] In an embodiment, the expression vector may be formed to represent a meaning corresponding to the specific attribute independently of other attribute combinations that include the specific attribute.

[0044] In an embodiment, the prediction model is trained to output a solution to a combinatorial optimization problem as the prediction result, the combinatorial optimization problem includes at least one of a resource placement, path determination, or schedule establishment problem combined with a plurality of constraints, and the plurality of attributes may include attributes defining the constraints.

[0045] A learning method for artificial intelligence according to the present invention may include the steps of: receiving data containing attribute information corresponding to each of a plurality of attributes; inputting the data into a prediction model to generate first embedding data; inputting data from which at least one attribute has been removed into the prediction model to generate second embedding data; and training the prediction model such that a representation vector corresponding to the at least one attribute is formed based on the difference between the first embedding data and the second embedding data.

[0046] In an embodiment, the first embedding data and the second embedding data may be generated based on whether the at least one attribute is included for the same data.

[0047] In an embodiment, the second embedding data may be generated by inputting the data from which at least one attribute has been removed into the prediction model.

[0048] In an embodiment, the prediction model can be trained so that the representation vectors corresponding to the same attribute included in different attribute combinations have similar representations in a vector space.

[0049] In an embodiment, the prediction model can be trained such that a first embedding component representing a unique meaning corresponding to each attribute and a second embedding component corresponding to the interaction of a plurality of attributes are formed separately.

[0050] In an embodiment, the first embedding data can be learned to be formed by the combination of the first embedding component and the second embedding component.

[0051] An artificial intelligence inference system according to the present invention comprises one or more processors configured to perform operations by executing one or more instructions and a memory connected to said processors, wherein a prediction model learned such that a representation vector corresponding to a plurality of attributes is reflected in an embedding is stored in said memory, and said processors may be configured to receive a prediction request, input said prediction request into said prediction model, and generate a prediction result based on at least one specific attribute among a plurality of attributes constituting said prediction request.

[0052] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored on a computer-readable recording medium, and may include instructions for performing the steps of receiving a prediction request, inputting the prediction request into a prediction model learned such that a representation vector corresponding to a plurality of attributes is reflected in an embedding, and generating a prediction result based on at least one specific attribute forming the prediction request by the prediction model. Effects of the invention

[0053] According to the present invention, since a physical AI environment can be reproduced in a virtual space by integrally reflecting an environment layout and a control strategy model, it becomes possible to pre-verify various operating conditions prior to actual physical deployment.

[0054] According to the present invention, since a plurality of control strategy models can be applied to the same environment layout or the same control strategy model can be applied to a plurality of environment layouts for comparison and verification, consistency and reliability in determining the suitability of an operation strategy can be improved.

[0055] According to the present invention, since it is possible to perform a simulation that comprehensively reflects performance changes based on the robot's movement state, work flow, resource allocation, and temporal operating conditions, it becomes possible to conduct a multifaceted analysis of spatiotemporal operating characteristics.

[0056] According to the present invention, since simulation execution results can be provided in a form that integrates intuitive visualization and quantitative performance indicators, users can quickly and clearly perceive the overall operational status, and in particular, make decisions regarding strategy changes or resource reallocation within a short period of time in environments requiring rapid decision-making, such as field operations.

[0057] According to the present invention, various types of control strategy models can be linked in a standardized structure, thereby ensuring scalability and flexibility of intelligent control technology.

[0058] In particular, since the control strategy model of the present invention processes spatial and temporal features separately and performs robot simulations based on them, the precision of operational performance analysis under complex spatiotemporal conditions can be improved. Accordingly, not only the accuracy of physical AI-based decision-making but also the efficiency of real-time strategy formulation can be enhanced. Brief explanation of the drawing

[0059] FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system, as an embodiment of a computing system in which the present invention can be implemented. Figure 3 illustrates an example of a block diagram from another perspective of a computing device, which is one of the components of a computing system. FIG. 4 is a conceptual diagram illustrating a physical AI-based simulation system according to the present invention. FIG. 5 is a flowchart illustrating a physical AI-based simulation method according to the present invention. FIGS. 6a to 6c are conceptual diagrams for explaining the process of creating an environment layout through a first user interface according to the present invention. FIG. 7 is a conceptual diagram illustrating the process of executing a robot simulation through a second user interface according to the present invention and providing the execution results of the robot simulation. FIGS. 8a to 8h are conceptual diagrams for explaining an example of an environment layout setting, a simulation result, and a data processing flow according to an embodiment to which the present invention is applied. FIG. 9 is a block diagram of an analysis module implementing an image analysis service based on separate analysis of spatial features and temporal features according to another embodiment. FIG. 10 illustrates an exemplary configuration of an analysis architecture that performs image analysis based on separate analysis of spatial features and temporal features according to one embodiment. FIG. 11 is intended to illustrate a method for encoding location information for an input sequence of a large language model according to one embodiment. FIG. 12 is a block diagram illustrating the configuration of a space-language mapping module according to one embodiment. FIG. 13 is intended to explain the function of a space-language mapping operation module according to one embodiment. FIG. 14 is a flowchart of a scene data analysis method according to one embodiment. FIG. 15 is a flowchart of a learning method for at least one artificial intelligence model used in the method of FIG. 14. FIG. 16 is a flowchart illustrating the steps that may be included in the step of encoding location information of the method of FIG. 15. FIG. 17 is a flowchart illustrating the steps for generating contextual space tokens through vision-language mapping that the method of FIG. 15 may further include. FIG. 18 is a conceptual diagram illustrating the structure for applying spatiotemporal separation analysis to initial scene data according to one embodiment. FIG. 19 is a conceptual diagram illustrating a structure for performing spatiotemporal separation analysis on scene sequence data generated during a strategy execution process according to one embodiment. FIG. 20 is a conceptual diagram illustrating a structure for performing comparative analysis between strategies based on multiple strategy execution history and environmental condition data according to one embodiment. FIGS. 21, FIGS. 22, FIGS. 23 and FIGS. 24 are conceptual diagrams for explaining an attribute processing module according to an embodiment of the present invention. FIG. 25 illustrates examples of attributes according to one embodiment of the present invention. FIG. 26 is a flowchart illustrating a method of inference for an attribute processing module according to an embodiment of the present invention. FIG. 27 is a flowchart illustrating a learning method of an attribute processing module according to the present invention. FIGS. 28, FIGS. 29, FIGS. 30, FIGS. 31, FIGS. 32, FIGS. 33, FIGS. 34, FIGS. 35, FIGS. 36, FIGS. 37, FIGS. 38, FIGS. 39, FIGS. 40, FIGS. 41, FIGS. 42, FIGS. 43, FIGS. 44, FIGS. 45, FIGS. 46, FIGS. 47, FIGS. 48, and FIGS. 49 are conceptual diagrams and mathematical expressions illustrating a learning and inference method of an attribute processing module according to the present invention. Specific details for implementing the invention

[0060] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components are assigned the same reference number regardless of the drawing symbols, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not have distinct meanings or roles in themselves. Furthermore, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0061] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0062] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0063] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0064] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0065] Hereinafter, the present invention will be examined in more detail with reference to the attached drawings. FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. FIG. 2 illustrates an example of a block diagram of a computing device that may be included in a user computing device, a server computing system, and a training computing system as an embodiment of a computing system in which the present invention can be implemented. FIG. 3 illustrates an example of a block diagram of a computing device in another aspect that is one of the components of a computing system.

[0066] Meanwhile, FIG. 1 illustrates an example of a block diagram of a computing system in which the present invention can be implemented. In this regard, the image-based user interface operation inference system according to the present invention can be implemented through a computing device described below and can perform data processing related to the physical AI-based simulation method described above.

[0067] Referring to FIG. 1, a computing system (10000) for performing a physical AI-based simulation method of the present invention may include at least one computing device. At this time, the at least one computing device may be a single processor or a multi-processor computing device.

[0068] The components of at least one computing device of the present invention may include at least one processor, memory, other hardware, and various hardware components such as a system bus (not shown) that connects various system components so that they can transmit and receive data to and from each other (e.g., telecommutatively connected, physically connected, electrically connected), and the components of at least one computing device are not limited thereto and may be very diverse.

[0069] Meanwhile, at least one computing device included in a computing system (10000) that performs a physical AI-based simulation method may be connected to communicate via a network (1070). For example, at least one computing device included in the computing system (10000) may be clustered or part of a local area network (LAN). Additionally, at least one computing device may be part of a wide area network (WAN) or connected to at least one of a client-server network and a peer-to-peer network within the cloud.

[0070] Meanwhile, when at least one computing device is used in at least one of a network environment and a cloud computing environment, the at least one computing device may be connected to at least one of a public and private network through a network interface or adapter. In one embodiment, other communication connection devices, such as a modem, may be used to establish communication through the network. The modem may be at least one of an internal modem and an external modem, and may be connected to a system bus through a network interface or a specific mechanism, etc. A wireless network component consisting of an interface and an antenna may be coupled to the network through a device such as an access point, a peer computer, etc. In the present invention, the method of connecting at least one computing device to communicate through the network (1070) is not limited, and it may be connected to communicate in a manner different from the described example.

[0071] Furthermore, other computer-type devices and / or systems not shown in FIG. 1 may also interact technically with at least one computing device or other system through one or more connections to the network (1070) via a network interface. Here, the network interface may include network interface equipment such as a physical network interface controller (NIC) or a virtual network interface (VIF).

[0072] The network (1070) of the present invention may include various forms such as the Internet, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wireless USB (Wireless Universal Serial Bus), etc., and in the present invention, data transmission may be performed based on standard communication protocols such as TCP / IP, HTTP, SSL, etc.

[0073] A computing system (10000) for performing a physical AI-based simulation method according to the present invention may include at least one of a user computing device (1010), a training computing system (1050), and a server computing system (1030).

[0074] A user computing device (1010) according to the present invention may be understood as a computing device comprising at least one processor (1011) and memory (1012) for performing a physical AI-based simulation method. For example, the user computing device (1010) may include at least one computing device among a smartphone, a smart TV, a laptop computer, a desktop computer, a digital broadcasting terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, and a head-mounted display).

[0075] At least one processor (1011) constituting the user computing device (1010) may include one or more general-purpose processors and / or one or more special-purpose processors. For example, at least one processor (1011) constituting the user computing device (1010) may be composed of at least one of a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), an application integrated circuit, an application semiconductor (ASIC), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or electrical units for performing other functions, or a plurality of electrically connected processors.

[0076] Furthermore, at least one processor (1011) may be configured to execute computer-readable instructions contained in memory (1012) and / or other instructions described herein.

[0077] The memory (1012) constituting the user computing device (1010) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media and / or other types of physically durable storage media.

[0078] For example, the memory (1012) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and may include web storage of a server that performs the storage function of memory on the internet. This memory (1012) may store data and instructions necessary for the at least one processor (1011) to perform the operation of an application for generating predictive action data from an image to be analyzed and providing an artificial intelligence agent trained using training data including the predictive action data.

[0079] A user computing device (1010) may include one or more user input components (1021) that detect user input. For example, the user input component (1021) may also be referred to as a user interface module. The user input component (1021) may include a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, or other similar devices. However, the present invention does not limit the type of user input component (1021). In this case, the user input component (1021) in the present invention does not necessarily mean a hardware means, but can be understood as a channel for receiving input from a user. Meanwhile, the user of the present invention may refer to an automated agent, script, playback software, etc., that operates on behalf of one or more people.

[0080] A user can interact with a computing system (10000) including at least one computing device through input text, touch, voice, movement, computer vision, gestures and / or other forms of input / output using a user input component (1021). For example, the user input component (1021) may include one or more of a command line interface (CLI), a graphical user interface (GUI), a natural user interface (NUI), a voice command interface and / or other user interface (UI) representations.

[0081] Between the user input component (1021) and the user computing device (1010), one or more application programming interface (API) calls may be made based on user input received from the user interface and / or network.

[0082] Here, the expression "based on" may be interpreted to include cases where it is based on the use of a specific configuration, modified from, derived from, influenced by, dependent on, or otherwise derived from a specific configuration. In some embodiments, an API call may be configured for a specific API, which may be interpreted or converted into an API call configured for another API. Here, an API may refer to a defined interface or connection between computers or between computer programs.

[0083] In one embodiment, the user computing device (1010) may store at least one machine learning model (1020). For example, the user computing device (1010) may be various machine learning models, such as a plurality of neural networks (e.g., deep neural networks) that perform at least one of generating predictive action data and performing the operation of an artificial intelligence agent, or other types of machine learning models including non-linear models and / or linear models, and may be composed of a combination thereof.

[0084] According to an embodiment of the present invention, a user computing device (1010) may perform a physical AI-based simulation method using a local or / and external machine learning model (1020). Alternatively, the user computing device (1010) may perform a physical AI-based simulation method using a machine learning model (1040) provided by a server.

[0085] In addition, according to another embodiment of the present invention, a server computing system (1030) communicating with a user computing device (1010) may provide predicted operation data from an image to be analyzed according to the present invention to the user computing device (1010) on an application or / and the web in response to a request from a user received through the user computing device (1010).

[0086] In addition, according to another embodiment of the present invention, by performing a physical AI-based simulation method of a user computing device (1010) and a server computing system (1030), a task result corresponding to user input data can be provided to the user through the operation of an artificial intelligence agent trained with training data including predicted action data.

[0087] Additionally, according to various embodiments of the present invention, a user computing device (1010) and / or a server computing system (1030) can learn machine learning models (1020, 1040) performed in a physical AI-based simulation method through interaction with a training computing system (1050) that is communicatedly connected via a network (1070). In this case, the training computing system (1050) may be a computing system separate from the server computing system (1030). Alternatively, in some embodiments, the training computing system (1050) may be part of the server computing system (1030) or part of the user computing device (1010).

[0088] Meanwhile, the server computing system (1030) may include at least one processor (1031) and memory (1032). Here, the processor (1031) may be composed of at least one or a plurality of electrically connected processors among a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), an application integrated circuit, an application semiconductor (ASIC), an arithmetic logic unit (ALU), a floating-point arithmetic unit (FPU), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, and / or other electrical units for performing functions. For example, at least one processor (1031) may include a circuit and a transistor configured to execute instructions from memory (1032).

[0089] The memory (1032) constituting the server computing system (1030) according to the present invention may include volatile memory, non-volatile memory, fixed media, removable media, magnetic media, optical media, semiconductor media, and / or other types of physically durable storage media. For example, the memory (1032) may include one or more non-transient / transient computer-readable storage media such as RAM, ROM, HDD (Hard Disk Drive), SSD (Solid State Disk), SSD (Silicon Disk Drive), EEPROM, EPROM, flash memory device, magnetic disk, etc., and combinations thereof, and may include web storage of a server that performs the storage function of memory over the internet. Additionally, the server computing system (1030) may further include a data storage (data store). For example, the data storage may be composed of at least one of a relational database, a NoSQL database, a data warehouse, and a local file system.

[0090] In the memory (1032) constituting the server computing system (1030) according to the present invention, data and instructions necessary for the at least one processor (1031) to perform the operation of an application for generating prediction operation data and operating an artificial intelligence agent may be stored.

[0091] In one embodiment, the server computing system (1030) may be composed of a single device or a plurality of computing devices, and may be configured to operate according to a sequential or parallel computing architecture. Additionally, a distributed processing system may be configured with a plurality of networked devices.

[0092] Meanwhile, the training computing system (1050) may include at least one processor (1051) and memory (1052). The model trainer (1060) is a logical component that executes the training of at least one machine learning model (1020, 1040) and may be implemented in the form of hardware, firmware, or software. For example, the model trainer (1060) may be executed by the processor (1051) after loading training data (1061) stored in a storage device into memory (1052). For example, the model trainer (1060) may be configured to execute one or more operations (e.g., model training, model reconstruction, model validation, model testing) on ​​at least one machine learning model.

[0093] The machine learning model of the present invention may include at least one of a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network, a Word2Vec model, a Bag of Words model, a TF-IDF (document frequency-inverse document frequency) model, a GPT (Generative Pre-trained Transformer) model (or other autoregressive models), a PPO (Proximal Policy Optimization) model, a nearest neighbor model (e.g., a k-nearest neighbor model), a linear regression model, a K-means clustering model, a Q-learning model, a TD (Temporal Difference) model, a Deep Adversarial Network model, and all other types of models further described herein.

[0094] Specifically, the model trainer (1060) may execute operations to train a machine learning model, and said operations may include at least one of adding, removing, and modifying model parameters. At this time, the training of the machine learning model may be at least one of supervised learning, semi-supervised learning, and unsupervised learning. In one embodiment, the training of the machine learning model may include the step of repeatedly inputting training data (1061) based on epochs and repeatedly performing the machine learning model training process configured in this way. Here, an epoch may refer to a unit in which the entire set of training data (1061) undergoes forward and backpropagation processing once. In some implementations, different levels of training methods (e.g., supervised learning, semi-supervised learning, unsupervised learning) may be used for different epochs.

[0095] The training data (1061) of the present invention may include input data and / or data previously output from at least one machine learning model (e.g., recursive learning feedback).

[0096] At least one parameter of a machine learning model may include at least one of a seed value, a model node, a model layer, an algorithm, a function, connections between different machine learning models, connections between parameters, machine learning model constraints, and other digital components that influence the output of the machine learning model. In this case, model connections between different machine learning models may include or represent relationships between model parameters and / or models, which may be dependent or interdependent, hierarchical, and / or static or dynamic. The combinations and configurations of model parameters described herein may be too complex to be maintained or utilized by human cognitive abilities.

[0097] In the present invention, the machine learning parameters described according to the embodiments are not limited, and a single machine learning model may further include a plurality of model parameters.

[0098] Meanwhile, FIG. 2 illustrates an example of a block diagram of a computing device (1100) that may be included in a user computing device (1010), a server computing system (1030), and a training computing system (1050), as an embodiment of a computing system (10000) in which the present invention can be implemented.

[0099] As illustrated in FIG. 2, the computing device (1100) may include at least one application (e.g., Application 1 to Application N), and each of the at least one application may include a machine learning library and a model execution environment for performing a machine learning-based physical AI-based simulation method. The at least one application included in the computing device (1100) may communicate with the sensor, context manager, device state manager, or additional component(s) within the computing device (1100) via an Application Programming Interface (API). In one embodiment, the at least one application may interface with device components, such as receiving sensor data or state data or transmitting prediction results to an output device via a public or private API.

[0100] Meanwhile, FIG. 3 illustrates an example of a block diagram in another aspect of a computing device (1200), which is one of the components of a computing system (10000) that performs a physical AI-based simulation method according to an embodiment of the present invention.

[0101] A computing device (1200) according to the present invention may include at least one application (e.g., Application 1 to Application N), and at least one application may communicate with a central intelligence layer (1210). Each application may interact with a shared model within the central intelligence layer (1210) through an API (e.g., a common API).

[0102] The central intelligence layer (1210) includes one or more machine learning models and may share them among multiple applications or provide them independently to each. In one embodiment, the central intelligence layer (1210) may be integrated as part of an operating system or implemented as a separate logical layer.

[0103] Additionally, the central intelligence layer (1210) can communicate with the central device data layer (1220). The central device data layer (1220) can integrate and store at least one of the analysis target image, scene description information, basic scene data, scene data, basic motion data, and predicted motion data in the computing device (1200), and provide this as input data required for training (e.g., fine-tuning) at least one training target model. Each device component (e.g., sensor, state manager, etc.) can communicate with the central device data layer (1220) through a private API, etc.

[0104] The technology described herein may be composed of a single or multiple computing devices, and a machine learning model performing a physical AI-based simulation method may be executed sequentially or in parallel on one component or multiple distributed components. Data storage, machine learning models, and applications may be distributed and operated locally or over a network, and these configurations can be flexibly applied to various system architectures.

[0105] The above describes a computing system in which the physical AI-based simulation method according to the present invention can be implemented. Below, the image-based user interface operation inference according to the present invention will be described in general.

[0106] As illustrated in FIG. 4, a physical AI-based simulation system (5100) according to one embodiment of the present invention may include at least one of a storage unit (5110), a user interface unit (5120), a communication unit (5130), and a control unit (5140). Additionally, the control unit (5140) may include at least one of an environment layout generation unit (5141), an environment state management unit (5142), a model linkage unit (5143), and a simulation execution unit (5144). The system (5100) according to the present invention may be operated on a computing system as described in FIGs. 1 to 3.

[0107] The simulation system (5100) of the present invention may have a structure that performs simulations by reflecting changes in environmental conditions. In one embodiment, the simulation system may operate as a time step-based iterative structure or an event occurrence-based structure, and may be implemented in a form in which the interaction between the environmental conditions and the control strategy model is performed iteratively.

[0108] For example, the control unit (5140) manages the time index t of the simulation and can drive the control strategy model based on the current environment state S(t) at each time step. The time step may be defined according to a fixed time interval or may be defined based on the occurrence of a specific event.

[0109] In this case, at each time step or event point, the environment state S(t) is provided as input to the control strategy model, the control strategy model produces an action A(t), and the action is converted into a control command and reflected in the state transition operation so that the next environment state S(t+1) can be generated.

[0110] With this structure, the simulation can proceed through the interaction between the environmental state and the control strategy model.

[0111] In the following, for convenience of explanation, the simulation system (5100) of the present invention is exemplified as having a closed-loop simulation structure having a time step-based iterative structure.

[0112] To this end, the environment state management unit (5142) is configured to generate and manage the environment state S(t) at each time step. The environment state may be represented as a single global vector, or as a structure including a set of state vectors for each of multiple objects and relationship information between objects. As an example, the environment state may include object-specific state information (location, speed, direction, work status), goal information, distance or collision risk between objects, spatial constraint information, etc. The state data is represented in the form of structured data and may undergo normalization and dimension alignment processes before being transmitted to the model linkage unit (5143).

[0113] Here, the object can be freely configured as a robot, autonomous vehicle, drone, autonomous ship, etc., which are the subject of the simulation. However, for convenience, robots will be used as examples in the following explanation.

[0114] In the present invention, “strategy” refers to a judgment rule, computational model, or decision logic for determining an action in response to an environmental state, and is not limited to a specific implementation method. For example, the strategy may be implemented by at least one of a rule-based model, a learning-based model, or a statistical model.

[0115] “Action” refers to an operational alternative selected or generated in response to a given environmental state, and may be expressed in the form of discrete selection values, continuous values, or structured data. “Control command” refers to input data reflected in state update operations within the simulation environment based on the aforementioned action, and is not limited to signals that directly drive actual physical devices.

[0116] For example, when reinforcement learning is applied, “strategy” refers to a rule or policy for selecting an action in response to an environmental state, and “action” refers to a control alternative selected at a specific time step t, which can be expressed in the form of a discrete choice or a continuous control value.

[0117] From the perspective of data flow, the current environmental state S(t) is provided as input to a control strategy model, said control strategy model produces an action A(t), and said action can be converted into an executable control command. The control strategy model can produce action information based on the current environmental state S(t), and said action information can be reflected in a state update operation within the simulation.

[0118] The above control command can be applied to state update operation logic to calculate the next environment state S(t+1). The above state update operation logic may include a calculation rule or processing algorithm that calculates a subsequent state based on the current environment state and the control command.

[0119] The above state update operation logic may be included in the simulation execution unit (5144) and may include operation logic for calculating the next environment state based on the current environment state and control commands. In this case, the state update operation logic may be implemented in various ways.

[0120] As one example, the state update operation logic may include a movement operation that calculates a movement result based on the position, speed, and direction information of a robot or vehicle. As another example, the state update operation logic may include interaction determination operations such as calculating the distance between objects, determining whether a collision has occurred, determining a passable area, or determining whether a task has been completed. As yet another example, the state update operation logic may include a state update according to a predefined rule, a state change calculation according to a probabilistic transition rule, or a state prediction operation based on a learned transition function.

[0121] Accordingly, the state update operation logic may be implemented by at least one of a physics-based model, a rule-based model, a probability model, or a learning-based model, and is not limited to a specific implementation method. As an example, the state update operation logic includes a state transition model, and the state transition model may be a model including operation logic including dynamic operations and collision judgment, a physics model, or a dynamics model.

[0122] As explained above, the data flow can be sequentially configured in the order of environment state S(t), calculation of action A(t) by the control strategy model, generation of control command, state update operation (e.g., application of state transition model), and generation of the next state S(t+1).

[0123] The above state update operation logic may include, but is not limited to, movement operations, determination of interaction between objects, constraint verification, or determination of collision. Accordingly, control commands may be reflected in the current state to calculate a new state.

[0124] The control unit (5140) manages the time step progression of the simulation and, when a control command is received from the control strategy model, may request the simulation execution unit (5144) to perform a state update operation. Accordingly, the simulation execution unit (5144) may perform a state transition operation based on the current state S(t) and the control command A(t) and generate the next state S(t+1).

[0125] Here, the control command A(t) corresponds to the result of an action selection performed by the control strategy model. That is, the action selection refers to a decision-making process of selecting one of multiple possible control alternatives based on the current environmental state S(t). The control strategy model can receive state data as input and produce a specific action A(t) according to a policy function, a rule set, or an inference result.

[0126] The selected action A(t) is converted into an executable control command by the control unit (5140) and transmitted to the simulation execution unit (5144), and the next state S(t+1) is determined by applying the action as an input to the state update operation logic. Therefore, even if the same initial state is given, if the result of the action selection is different, the subsequent state transition path and simulation development may differ.

[0127] The simulation execution unit (5144) can calculate performance evaluation data based on the state transition result. The performance evaluation data may include operational performance indicators such as work completion time, collision occurrence, resource efficiency, and energy consumption. For example, the performance evaluation data is calculated in the form of a compensation value for each time step, and to this end, the simulation execution unit (5144) may be configured to apply a compensation function R(t) based on the state transition result S(t+1). The compensation function may be defined by considering the work completion time, collision occurrence, resource efficiency, energy consumption, etc.

[0128] An experience data set including the current state, selected action, reward value, and next state may be generated at each time step. For example, execution history data including the state, action, performance evaluation data, and next state may be generated at each time step. The experience data set may be generated by the simulation execution unit (5144) and recorded in the storage unit (5110).

[0129] The above storage unit (5110) may include hardware elements for storing data. For example, it may include the aforementioned memory (1012), and furthermore, web storage of a server.

[0130] The above experience data set can be stored in the storage unit (5110) along with the strategy model identifier and the environment layout identifier. If the control strategy model includes a learnable structure, the execution history data or performance evaluation data can be used as reference information in the policy adjustment or model improvement process.

[0131] In the present invention, when multiple control strategy models exist, the control unit (5140) can manage to provide the same environmental state S(t) to the multiple strategy models in common. Each strategy model can independently calculate an action A_i(t), and the simulation execution unit (5144) can create independent instances for each strategy to perform state transitions. Accordingly, data such as state history, action history, and performance evaluation history (reward history, performance indicators, etc.) can be managed independently for each strategy. The fairness of the evaluation can be ensured by performing comparisons between strategies under the same environmental layout identifier and the same initial conditions.

[0132] The control strategy model may be at least one of a rule-based model, a reinforcement learning-based policy model, or a large-scale language model-based strategy model.

[0133] A rule-based model can be configured to select an action based on the result of rule evaluation by substituting the current state into a conditional expression. A reinforcement learning-based policy model can be configured to input a state vector into a policy function to produce an action probability distribution or a continuous control value. A large-scale language model-based strategy model can be configured to input state data into the model in the form of a prompt and convert the generated strategy output into a control command. The control unit (5140) can selectively apply different data transfer methods and execution methods depending on the model type.

[0134] In the present invention, the environment layout includes spatial structure, object density, constraints, etc., which can directly affect the difficulty and performance of the strategy model. Even for the same strategy model, the state transition path and accumulated reward value may vary depending on the environment layout conditions. Therefore, by managing the environment layout identifier and the strategy model identifier in conjunction, it becomes possible to ensure the reproducibility of strategy performance evaluation and manage them at the experimental level.

[0135] Below, based on the above description, the configuration of the present invention will be examined in more detail.

[0136] The storage unit (5110) is configured to store data and program commands necessary for performing a simulation. The storage unit (5110) can store environment layout data, operating condition data, environment status data, control strategy model information, simulation result data, and performance indicator data. Program commands stored in the storage unit (5110) can be executed by the control unit (5140).

[0137] The user interface section (5120) can provide visual information, auditory information, etc. to the user through the user terminal (5001).

[0138] The user interface unit (5120) can receive user input through the user terminal (5001) and visually output simulation results. The user interface unit (5120) can be implemented in the form of a web-based interface, an application-based interface, or a remote access interface to provide interaction between the user and the system. For convenience of explanation, an application-based graphical user interface (GUI) will be described below as an example.

[0139] The communication unit (5130) is configured such that the simulation system (5100) communicates with an external storage (5200) and an external server (300). For example, the communication unit may include an antenna, a network interface card (NIC), a wireless transceiver module, a communication control chip, or a hardware component that performs communication.

[0140] The communication unit (5130) can perform model execution requests and receive results when the control strategy model is deployed to an external server, and can communicate with an external storage (5200) for storing or loading simulation data.

[0141] The control unit (5140) may be implemented by one or more processors and is configured to perform a simulation by executing program commands stored in the storage unit (5110). The control unit (5140) may include at least one of an environment layout generation unit (5141), an environment state management unit (5142), a model linkage unit (5143), and a simulation execution unit (5144).

[0142] Here, the environment layout generation unit (5141), environment state management unit (5142), model linkage unit (5143), and simulation execution unit (5144) are not limited to physically separated hardware modules, but can be implemented by program code or software function blocks executed on a processor constituting the control unit (5140). That is, the environment layout generation unit (5141), environment state management unit (5142), model linkage unit (5143), and simulation execution unit (5144) each refer to function modules logically separated to perform specific functions, and can be implemented integrally on a single processor or distributed across multiple processors.

[0143] Additionally, the environment layout generation unit (5141), environment state management unit (5142), model linkage unit (5143), and simulation execution unit (5144) may be implemented in an integrated or separate manner as needed, and some functions may be performed by software modules running on an external server or in a cloud environment. Therefore, the names of the components are for functional distinction only and do not limit the separation of physical configurations.

[0144] Meanwhile, the environment layout generation unit (5141) can generate a virtual physical environment related to robot operation or autonomous driving based on information input through the first user interface (5121). The generated environment layout can be expressed as a data structure including spatial structure, object placement, constraints, etc.

[0145] The environment state management unit (5142) is a component that generates and manages the state of the robot or vehicle and the surrounding environment state at each time step of the simulation. The environment state data may include location, speed, direction, work status, surrounding object information, etc., and may be updated according to the simulation execution results. The environment state management unit (5142) may generate and manage the environment state at each time step in the form of structured data. The environment state may be expressed as a data structure including object-specific state information, spatial constraint information, and relationship information between objects, and may be defined as a common state expression format to be used as input to a control strategy model. The state expression may be defined as a common data structure that is not dependent on a specific control strategy model structure.

[0146] The model linkage unit (5143) may be configured to convert environment state data provided from the environment state management unit (5142) into an input format for the control strategy model and to call the control strategy model.

[0147] In the present invention, the model linkage unit (5143) can function as an interface layer that normalizes and structures environmental state data to match the input specifications required by the control strategy model. Specifically, the state set data generated by the environmental state management unit (5142) is not directly transmitted to the model, but can be converted into a model input vector or tensor form through a state feature extraction and dimension alignment process. In this way, by forming a standardized data interface between the environmental state data and the control strategy model, multiple control strategy models having different structures can be interchangeably linked based on the same environmental data.

[0148] More specifically, the model linkage unit (5143) may comply with a standardized interface specification applicable to a multi-agent environment. The standard interface may be formed to manage the state, behavior, reward, and state transition information of the multi-agents according to a common data structure.

[0149] For example, the above standard interface may follow a standard API (Application Programming Interface) structure applicable to a multi-agent reinforcement learning environment, but is not limited thereto.

[0150] The above standard interface may include a common execution protocol that defines the data exchange format and calling procedure between the simulation environment and the control strategy model. The execution protocol may be implemented as a cyclic interaction structure including the initialization of the simulation environment, the provision of agent-specific state information, the reception of behavior information from the control strategy model, and the generation and return of state transition results based on the behavior information.

[0151] By adopting such a common execution protocol, a decoupling structure can be formed in which the coupling between the internal implementation structure of the simulation environment and the implementation structure of the control strategy model is reduced. Accordingly, as long as the control strategy model satisfies the above standard interface specifications, it can be replaced or applied in parallel without any structural changes.

[0152] The model linkage unit (5143) can perform state feature extraction, input format conversion, and model execution request generation, and can receive control commands or strategy information output from the control strategy model and transmit them to the simulation execution unit (5144).

[0153] The control strategy model may be at least one of a reinforcement learning-based model, a rule-based model, or a large-scale language model-based model, and may be deployed on an internal or external server (300) of the simulation system (5100). It goes without saying that the control strategy model may be named a “model” or a “behavior model,” etc.

[0154] The above control strategy model is not limited to a single model, and multiple models having different structures can be operated simultaneously or selectively.

[0155] In the present invention, the control strategy model is not limited to a configuration that operates independently and autonomously, and can be driven by the control unit (5140). More specifically, the model linkage unit (5143) operates under the control of the control unit (5140) and can call the control strategy model based on environment state data transmitted from the environment state management unit (5142).

[0156] The control unit (5140) can request the model linkage unit (5143) to execute the model by using the progression of a time step in the simulation or the occurrence of an event as a trigger. Accordingly, the control strategy model is driven at each time step or whenever a specific condition is satisfied, so that an action selection for the current state can be performed.

[0157] In the present invention, “action selection” refers to a decision-making process that determines one of a plurality of possible control alternatives in a task to be simulated, such as the operation of a robot or an autonomous vehicle, based on environmental state data at a current time. The environmental state data may include the robot’s position, speed, direction, work status, and information regarding relationships with surrounding objects, and a control strategy model may receive the state information as input and produce a specific action according to predefined rules, learned policies, or inference results. Here, “action” may include control judgments that directly affect robot operation, such as selecting a direction of movement, determining a path, adjusting speed, assigning tasks, determining priorities, and collision avoidance strategies.

[0158] The aforementioned action selection is not limited to merely generating control signals but can be understood as a judgment step of strategically deciding one of multiple candidate control alternatives corresponding to the current state. The control strategy model can select a specific action or calculate continuous control values ​​based on the state-action mapping relationship, and the result of the selection can be converted into an executable control command by the control unit and reflected in the simulation execution unit. Accordingly, even if the same initial environmental conditions are given, the transition path of the subsequent environmental state and the simulation progression (scene evolution) may differ depending on the result of the action selection.

[0159] Furthermore, the aforementioned action selection can be performed repeatedly based on time steps or events, and the selection result in each iteration may affect cumulative performance indicators or performance evaluation data. Accordingly, the “action selection” in the present invention serves as a core decision-making step for determining robot operation strategies and performs a pivotal function that connects environmental states with subsequent state transitions.

[0160] The model linkage unit (5143) can transmit input data to the control strategy model and receive the model execution result according to the execution control of the control unit (5140). At this time, the control strategy model may be called synchronously by the control unit (5140) or executed asynchronously to return a result.

[0161] The control unit (5140) can interpret the strategy selection result received from the model linkage unit (5143) and transmit it to the simulation execution unit (5144) so ​​that specific control execution can be performed.

[0162] Accordingly, the control strategy model is an execution target managed, called, and controlled by the control unit (5140), and the model linkage unit (5143) can function as an interface layer that mediates the execution of the control strategy model.

[0163] Meanwhile, when a plurality of control strategy models are operated in the present invention, the control unit (5140) can generate a plurality of model execution requests in parallel, and the model linkage unit (5143) can generate an independent execution session for each model. When executing a plurality of control strategy models in parallel, the simulation system (5100) can generate an independent simulation instance corresponding to each strategy and record a state history, a performance evaluation history, and performance indicators individually for each instance. The instances can be managed as units for comparative experiments between strategies by sharing the same environment layout identifier and the same initial condition information.

[0164] The control unit (5140) can manage to provide the same environment state data to each model in common, and can receive control command data returned from each model along with an identifier to maintain a control execution flow separated by strategy.

[0165] Accordingly, the control unit (5140) can function as a higher-level control entity that integrally manages the execution of multiple strategy models.

[0166] In the present invention, there may be multiple control strategy models, and they may be operated in different ways depending on their type. As an example, if the control strategy model is a rule-based model, the control unit (5140) can transmit current state data to a rule engine and drive it to select an action based on the result of a condition judgment. As another example, if the control strategy model is a reinforcement learning-based model, the control unit (5140) can input a state vector into a policy function and drive it to calculate an action probability distribution or a continuous control value. If necessary, it may also include a structure that reflects the result of calculating cumulative performance indicators back into the learning process. As yet another example, if the control strategy model is a large-scale language model-based strategy model, the control unit (5140) can configure state data in the form of a prompt and transmit it to the model, and convert the high-level strategy generated by the model into a control-executable command. In this way, the control unit (5140) can selectively apply different driving methods and data transmission methods depending on the type of control strategy model.

[0167] The control unit (5140) receives state update data from the simulation execution unit (5144), reflects it in the environment state management unit (5142), and can call the control strategy model again based on the updated state. Accordingly, a repeating structure of environment state, strategy model execution, strategy selection, control execution, state update, and re-call is formed, and the control strategy model can be repeatedly executed during the simulation process.

[0168] The simulation execution unit (5144) is configured to perform a simulation by reflecting control commands received from the control strategy model into the environment state. The simulation execution unit (5144) can update the state using state update operation logic, determine whether there is a collision or a constraint violation, and calculate performance indicators such as throughput, average operation time, and reward value. State history data, control command history data, and performance evaluation history data generated during the simulation process can be stored in the form of a dataset and can be used as data for training or verification of the control strategy model in the future.

[0169] Meanwhile, each element constituting the simulation system of the present invention will be described in more detail as follows.

[0170] The user interface unit (5120) provides interaction between the user and the simulation system through the user terminal (5001). The user interface unit (5120) performs both input reception and output display functions and can receive environment configuration information, model selection information, operation parameter information, and simulation execution control information from the user.

[0171] The user interface unit (5120) may be configured to allow the user to create or modify an environment layout and may be formed to generate environment data by receiving inputs for robot placement, object placement, path structure, or operation parameters. Additionally, the user interface unit may be configured to select or specify at least one of a plurality of control strategy models and may be formed to receive inputs for linking the selected control strategy model with the environment layout. Furthermore, the user interface unit may be configured to visually display simulation execution results and may be formed to output state changes, object movements, performance evaluation results by strategy, or cumulative performance indicators in the form of graphs, tables, animations, or other visual representations.

[0172] Additionally, the user interface section (5120) may include at least one of a first user interface (5121) and a second user interface (5122). For example, the first user interface may be formed as an input-oriented interface that performs environment layout design and strategy selection, and the second user interface may be formed as an output-oriented interface that outputs simulation results. Although the first user interface and the second user interface are logically separated, it is also possible for them to be implemented on the same physical device.

[0173] The first user interface (5121) is an input screen for creating an environment layout related to robot operation, and may include spatial structure settings (grid or coordinate-based spatial definition), object placement settings (robot, vehicle, obstacle, etc.), and operation parameter settings (workflow, resource placement, time conditions, etc.). Additionally, it may be configured so that a user can select components and place them in the space through a drop-down menu, a selection area, or a graphic placement interface. The input information is structured into environment setting data and transmitted to the environment layout generation unit (5141).

[0174] The second user interface (5122) is an output screen that provides simulation execution results to the user and may include an animation display area that displays the movement status of a robot or vehicle in an animation form, and an indicator display area that displays performance indicators such as throughput, average operation time, and number of collisions. Additionally, it may include a selection area that allows selecting or comparing any one of a plurality of control strategy models.

[0175] A physical AI-based simulation method according to the present invention may include the steps of generating an environment layout related to robot operation through a first user interface (5121), interlinking at least one control strategy model with the environment layout, and executing a simulation of robot operation based on the environment layout and the control strategy model and outputting the execution result through a second user interface (5122).

[0176] When an environment layout related to robot operation is generated through the first user interface (5121), the environment layout can be converted into a data structure by the environment layout generation unit (5141) and stored in the storage unit (5110) or transmitted to the environment state management unit (5142).

[0177] The environment state management unit (5142) can generate initial environment state data based on the environment layout data and generate dynamic state data by time step by reflecting state update data during the simulation execution process. The generated environment state data is provided to the model linkage unit (5143).

[0178] The model linkage unit (5143) can convert the environment status data into an input format for the control strategy model and generate control command data by functionally linking with the selected control strategy model. The control strategy model may be implemented within the system or deployed to an external server via the communication unit (5130), and the model linkage unit (5143) can transmit status data to the control strategy model and receive control command data through a standard interface.

[0179] Additionally, the model linkage unit (5143) can manage multiple control strategy models simultaneously. The multiple control strategy models may be based on different algorithm structures or learning methods, and the model linkage unit (5143) can execute multiple control strategy models in parallel on the same environmental state data.

[0180] To this end, the model linkage unit (5143) normalizes the environment state data into a common standard input format, thereby enabling multiple control strategy models to use the same initial environment layout and state data of the same time step as input. Accordingly, each control strategy model calculates a strategy under the same conditions, and a basis for quantitatively comparing performance differences between models can be established.

[0181] Through this, the present invention has an extended structure that is not dependent on a specific model structure and can compare and evaluate reinforcement learning-based policy models, rule-based models, large-scale language model-based strategy models, etc., based on the same environment layout and state data structure.

[0182] The simulation execution unit (5144) receives the control command data, updates the state of the robot or vehicle, determines whether a collision has occurred or whether a constraint has been violated, and can calculate performance indicator data. The state update data is transmitted to the environment state management unit (5142) to be reflected in the state generation of the next time step, and the performance indicator data and event log data can be output through the second user interface (5122).

[0183] Accordingly, the simulation system (5100) of the present invention has a structure that links a control strategy model based on an environment layout defined through a first user interface (5121), repeatedly executes robot operation in a virtual environment by reflecting control commands calculated by the control strategy model, and visually provides the execution results through a second user interface (5122).

[0184] In particular, when multiple control strategy models are configured to be selectable, different control strategy models can be applied under the same environment layout conditions to generate respective simulation results. The simulation execution unit (5144) can calculate state update data and performance indicator data corresponding to each control strategy model, and the second user interface (5122) can output the performance indicator data in a form that can be compared.

[0185] Accordingly, the present invention provides a structure capable of quantitatively comparing and evaluating multiple strategies under the same physical environment conditions, and can have the technical effect of providing objective grounds for strategy selection.

[0186] Additionally, the simulation execution unit (5144) can produce performance evaluation data for each strategy by performing a performance evaluation operation based on the state transition results corresponding to each strategy model. The performance evaluation criteria may be defined by considering at least one of the task completion time, whether a conflict occurred, whether a constraint was violated, resource utilization efficiency, or energy consumption. A performance evaluation value may be calculated for each time step or event unit. The performance evaluation value may be aggregated and stored as a cumulative performance indicator along with the strategy model identifier, or used for comparative analysis.

[0187] As an example of this, the simulation execution unit (5144) can calculate strategy-specific reward data by applying a reward function based on the state transition result corresponding to each strategy model.

[0188] When multiple control strategy models are executed in parallel, the simulation execution unit (5144) can maintain independent reward calculation paths corresponding to each strategy model, and state transitions and reward histories for each strategy can be recorded under the same environment layout and the same initial conditions. Accordingly, comparisons between strategies can be performed according to the same reward criteria, and objectivity and fairness in strategy selection can be ensured.

[0189] The above reward data may be stored in the storage unit (5110) along with the strategy model identifier and the environment layout identifier, and may be visualized as a cumulative reward trend or comparison result per strategy through the second user interface (5122). Additionally, if the control strategy model is a learnable model, the control unit (5140) may be configured to feed the above reward data back to the corresponding strategy model to be used for policy updates.

[0190] At this time, the control strategy model generates a control command based on environmental state data, and the simulation execution unit (5144) updates the state by reflecting the control command in the environmental state and can repeatedly produce the result.

[0191] Meanwhile, as previously discussed, when multiple control strategy models are executed in parallel, the simulation execution unit (5144) may maintain a separate state update path corresponding to each control strategy model, or create and execute independent simulation instances based on the same initial state. Accordingly, control commands for each control strategy model are independently reflected under the same environment layout and the same initial conditions, and respective state change history and performance indicator data are generated.

[0192] The above performance indicator data can be stored together with model identification information or transmitted to the second user interface (5122).

[0193] The second user interface (5122) can display performance indicator data corresponding to multiple control strategy models in a comparable form. For example, throughput, average operation time, number of crashes, reward values, etc., per model can be displayed in parallel on the same screen or superimposed in the form of a graph. Accordingly, the user can intuitively compare the performance of multiple strategies under the same environmental conditions.

[0194] Meanwhile, the environment layout generation unit (5141) may be configured to define a virtual physical environment based on environment setting data input through the first user interface (5121). The environment layout generation unit (5141) may interpret spatial structure information and object placement information input through the first user interface (5121) to internally generate a spatial coordinate system or a grid-based data structure. The data structure may include attributes of each location within the space, passage connection relationships, movable areas, obstacle locations, etc.

[0195] Additionally, the environment layout generation unit (5141) sets initial placement information for objects, such as robots or autonomous vehicles, and reflects operational parameters (workflow, resource placement, temporal operation conditions, etc.) in the data structure. The generated environment layout data forms static structure data, which is stored in the storage unit (5110) or transmitted to the environment state management unit (5142). The environment layout data functions as a reference environment for simulation and serves as a standard for subsequent state generation and strategy calculation.

[0196] The environment state management unit (5142) may be configured to generate and manage the initial state of the simulation and the dynamic state of each time step by referencing the environment layout data transmitted from the environment layout generation unit (5141). The environment state management unit (5142) generates state vectors or state set data including the position, speed, direction, work status, target information, and distance relationship with surrounding objects of each robot or vehicle.

[0197] When state update data is transmitted from the simulation execution unit (5144), the environment state management unit (5142) can update the state data for the next time step by reflecting this. In this process, if multiple robots or vehicles exist, the state of each object can be integrated to form a complete set of environment states. The generated environment state data can be provided as input to the model linkage unit (5143).

[0198] The model linkage unit (5143) may be configured to convert environmental state data transmitted from the environmental state management unit (5142) into an input format that the control strategy model can process, and to perform the role of an interface with the control strategy model.

[0199] The model linkage unit (5143) can extract state feature data necessary for strategy determination from state data. For example, it can generate feature vectors by calculating the distance between objects, collision risk, congestion, error relative to the target, queue length, etc. The generated feature vectors are converted into an input format for the control strategy model (e.g., multidimensional tensor, structured data, rule-based input structure, etc.).

[0200] Additionally, the model linkage unit (5143) selects one of a plurality of control strategy models, transmits input data to the selected control strategy model through a standard interface, and receives control command data generated as a result of model execution. The control strategy model may be implemented inside the simulation system or may be deployed to an external server via the communication unit (5130). The model linkage unit (5143) can transmit the received control command data to the simulation execution unit (5144).

[0201] The control strategy model is a component that calculates a robot operation strategy based on environmental conditions and may include a reinforcement learning-based policy model, a rule-based model, or a large-scale language model-based strategy model. The control strategy model processes model input data transmitted from the model linkage unit (5143) to generate control command data.

[0202] The above control command data may include the movement path of the robot or vehicle, speed control value, task assignment information, priority determination information, etc. The generated control command data may be transmitted to the simulation execution unit (5144) through the model linkage unit (5143).

[0203] The simulation execution unit (5144) is configured to receive control command data transmitted from the control strategy model and update the state of the robot or vehicle by reflecting it in the state update operation logic. The simulation execution unit (5144) can perform movement operations, collision judgment, constraint verification, etc., and generate updated state data.

[0204] Additionally, the simulation execution unit (5144) calculates performance evaluation values ​​or performance indicator data such as throughput, average work time, number of crashes, and reward value, and generates event log data. The generated state update data is transmitted to the environment state management unit (5142) and used to generate the state of the next time step, and the performance indicator data and event log data can be transmitted to the second user interface (5122) and output.

[0205] Accordingly, the present invention provides a closed-loop simulation structure capable of executing multiple control strategy models in parallel under the same environmental layout and the same initial state conditions, and quantitatively comparing state changes and performance indicators according to each strategy. As a result, unlike a simple robot operation simulation system, it can function as an evaluation platform for strategy selection and strategy improvement.

[0206] Meanwhile, the environment layout data generated by the environment layout generation unit (5141) may be stored in a file format containing a unique identifier (layout identifier). The unique identifier may be generated by combining, for example, layout version information, creation time information, spatial structure hash value, object placement configuration value, operation parameter set identification information, etc. As an example, the unique identifier may be configured in the form of a combination of “layout type code-version number-hash value”, and may be configured so that the same hash value is generated when having the same spatial structure and object placement.

[0207] Additionally, the unique identifier may be stored together with an environment layout file and may be mapped to layout data stored in a storage unit (5110) or an external storage (5200). Accordingly, by reloading an environment layout file corresponding to a specific unique identifier, the same spatial structure, the same object placement, and the same operational parameter conditions can be reproduced.

[0208] Furthermore, the unique identifier can be stored in conjunction with a simulation instance identifier and a control strategy model identifier. For example, multiple control strategy model identifiers can be mapped to a single layout identifier, and the experimental results for each model can be recorded as a combination of “layout identifier-model identifier-experiment round identifier.” Accordingly, the performance of multiple strategies can be repeatedly compared and verified under the same layout conditions, and the objectivity and reproducibility of the strategy performance comparison can be ensured.

[0209] In addition, the unique identifier may be stored together with random number seed information or initial state setting information, and when a simulation is re-run based on the same identifier combination, a strategy comparison may be performed under the same initial conditions. Accordingly, it can be guaranteed that the performance difference between strategies is due to differences in strategy selection rather than deviations in environmental conditions.

[0210] In this way, by introducing a unique identifier structure for the environment layout, the present invention can provide a platform structure capable of managing experimental units for strategy evaluation and data-based analysis, going beyond simple simulation execution.

[0211] Furthermore, in the present invention, the environment layout can be configured to model various physical AI application scenarios, such as automated logistics systems, order picking systems, warehouse operation environments, production line automation environments, multi-robot collaboration environments, and autonomous vehicle operation environments. Accordingly, the present invention can be expanded into a general-purpose strategy evaluation platform that is not limited to specific industrial fields.

[0212] In the above description, the simulation system (5100) of the present invention has been described, and it can implement the physical AI-based simulation method described below.

[0213] Hereinafter, the physical AI-based simulation method according to the present invention will be described in more detail with reference to FIG. 5 together with FIG. 6a to 6c and FIG. 7. FIG. 5 is a flowchart for explaining the physical AI-based simulation method according to the present invention, FIG. 6a to 6c are conceptual diagrams for explaining the process of creating an environment layout through a first user interface according to the present invention, and FIG. 7 is a conceptual diagram for explaining the process of executing a robot simulation through a second user interface according to the present invention and providing the execution result of the robot simulation.

[0214] In the physical AI-based simulation method of the present invention, a step of first providing a first user interface that generates an environment layout related to robot operation may be performed (see FIG. 5, S5100).

[0215] “Robot operation” according to the present invention may refer to an overall operational process of controlling various types of robots to perform at least one task. The robot operation is not limited to the motion control of a single robot and may include robot operation in which multiple robots are coordinated to perform the same or different tasks. As an example, robot operation may include performing tasks such as transporting goods, loading and unloading, order picking, material transfer between production processes, pathfinding and collision avoidance, task assignment and priority determination, resource distribution, and traffic flow management.

[0216] In this case, the robot according to the present invention may include an autonomous driving robot (or vehicle) including a movement function, an unmanned transport vehicle, an industrial manipulator, a collaborative robot, a material transport robot, or a combination thereof, and is not limited to a specific type of robot. For example, robot operation according to the present invention may include the operation of an autonomous driving vehicle, and it can be understood that the autonomous driving vehicle may drive in various spaces such as a logistics warehouse, inside a factory, a production line, an outdoor environment, or a road environment.

[0217] Therefore, robot operation in the present invention is not limited to a specific industry or specific type of work, but can be understood as a concept encompassing the overall motion control and task execution of a robot (or autonomous vehicle) operating within a physical space.

[0218] As illustrated in FIG. 6a, the first user interface (5121) according to the present invention may include a layout setting area (5400) for creating an environment layout related to robot operation. Here, the environment layout may refer to a robot operation environment in which at least one of the spatial structure and operation parameters in which the robot is operated is set. For example, the environment layout may refer to a robot operation environment in which at least one of i) the size of the space in which the robot moves and performs work, ii) the number of grid units dividing the space, iii) the size of said grid units, iv) the work flow, v) resource placement, and vi) temporal operation conditions is reflected. As an example, the environment layout may define an operation environment that includes at least one of spatial structure information and object placement information related to a plurality of autonomous vehicles.

[0219] The control unit (5140) can generate an environment layout related to robot operation based on the first user interface (5121). Specifically, the first user interface (5121) according to the present invention may include at least one layout setting area capable of setting different environment layout characteristics. For example, the user interface unit (5120) may provide a first user interface (5121) comprising a first layout setting area (5410) capable of setting spatial structure characteristics related to robot operation. Here, the spatial structure characteristics may include at least one of the number of rows dividing the space, the number of columns, and the size of the grid unit (Cell) dividing the space where the robot is operated.

[0220] The user interface unit (5120) can receive (or detect) user input that sets spatial structure characteristics related to robot operation in the first layout setting area (5410). Furthermore, the control unit (5140) can set a grid for the space related to robot operation according to the spatial structure characteristics related to robot operation set according to the received user input. At this time, setting a grid for the space related to robot operation may mean dividing the space into multiple unit sections to form a reference structure in which the movement, placement, and interaction of the robot can be defined.

[0221] Specifically, the environment layout generation unit (5141) can set a grid for a space related to robot operation based on at least one of the number of rows and columns that partition the space set in the first layout setting area (5410) and the size of the grid unit (Cell) that partitions the space where the robot is operated.

[0222] Meanwhile, the first user interface (5121) according to the present invention may include a layout editing area (5500) that visually displays the space (5510a) where the grid is set and provides an editing function for the space (5510a) where the grid is set. Specifically, the environment layout generation unit (5141) may set a grid in a space related to robot operation according to the spatial structure characteristics set in the first layout setting area (5410). Furthermore, the user interface unit (5120) may output (or provide) a space (5510b) where the grid is set in a layout editing area (5500) located in an area separated from the layout setting area in the first user interface (5121).

[0223] At this time, if map data for the space where the robot is operated exists in advance, the control unit (5140) can set a grid for the space related to robot operation using said map data. Specifically, the control unit (5140) can extract spatial structural characteristics (e.g., space size, space partitioning, etc.) included in the map data for the space where the robot is operated. For example, the control unit (5140) can calculate the number of rows and columns for creating a grid in the space where the robot is operated based on the space size of the space where the robot is operated and the size of the pre-set grid unit (or grid resolution). As an example, the control unit (5140) can determine the number of columns by dividing the horizontal length of the space where the robot is operated by the grid unit size, and determine the number of rows by dividing the vertical length by the grid unit size.

[0224] Through this, the control unit (5140) can generate a grid structure that matches the spatial structure characteristics included in the map data, even without directly receiving specific spatial structure characteristics such as the number of rows, the number of columns, and the grid unit (Cell) size in the first layout setting area (5410) from the user terminal (5001).

[0225] Specifically, as illustrated in FIG. 6b, the user interface unit (5120) may provide at least one map data item (5411) for specifying map data for a space related to robot operation in the first layout setting area (5410). Specifically, the control unit (5140) may provide a map data item corresponding to each of at least one map data previously stored in the storage unit (5110) to the first layout setting area (5410).

[0226] The user interface unit (5120) can receive (or detect) user input for at least one map data item (5411) provided in the first layout setting area (5410). The environment layout generation unit (5141) can load map data corresponding to the map data item for which the user input was received. Furthermore, the environment layout generation unit (5141) can set a grid in the space related to robot operation based on the spatial structure characteristics included in the map data.

[0227] At this time, the environment layout generation unit (5141) can extract operational restriction area information from the loaded map data. Here, “operational restriction area” may refer to a spatial area where the movement, occupation, or performance of work by a robot or autonomous vehicle is restricted or prohibited. For example, the operational restriction area may include areas where access by a robot is not allowed or is allowed only restrictively, such as walls, pillars, facilities, fixed structures, safety zones, no-traffic zones, worker-only zones, and danger zones.

[0228] In the present invention, the method for extracting operational restriction area information from map data according to the present invention may be very diverse, and in this specification, any module capable of extracting operational restriction area information from map data is not limited to its type and method. The environment layout generation unit (5141) according to the present invention may further include at least one of a module and an algorithm for extracting operational restriction area information from map data. For example, the environment layout generation unit (5141) may recognize objects within space using an object detection model, an object segmentation model, a semantic segmentation model, or an edge detection algorithm, and extract operational restriction area information based on the type of the recognized object.

[0229] As another example, the environment layout generation unit (5141) can use a vision-language model to interpret visual objects and text information included in the map data together, and recognize semantic labels (or, “wall,” “pillar,” “safety zone,” etc.) to specify an operation restriction area in a robot operation-related area. In this case, the vision-language model may be a pre-trained multimodal model and can infer the functional meaning of an element in space by combining the visual features of the object and the text description.

[0230] Additionally, the environment layout generation unit (5141) may use a large-scale language model to analyze descriptive information or metadata included in the map data and determine whether a specific space is suitable for robot operation. For example, it may interpret text information such as “worker-only zone” or “hazard area” included in the map data to set the corresponding area as an operation restriction area. Furthermore, the environment layout generation unit (5141) may use a rule-based algorithm to detect spatial structural characteristics, for example, passages or closed areas with a width below a certain standard, and set the said area as an operation restriction area.

[0231] Accordingly, the method for extracting an operation restriction area according to the present invention is not limited to a specific model or algorithm and may include at least one of a coordinate analysis-based method, a rule-based method, a machine learning-based method, a deep learning-based method, or a multimodal inference-based method.

[0232] Furthermore, the user interface unit (5120) can output (or provide) a grid-set space (5510b) such that an operation restriction area (5511) is displayed in a layout editing area (5500) located in an area separated from the layout setting area in the first user interface (5121).

[0233] In the foregoing, an example of a first layout setting area among at least one layout setting area included in a first user interface, capable of setting spatial structure characteristics related to robot operation, has been described. Below, an example of a second layout setting area among at least one layout setting area included in the first user interface, for placing at least one component related to a robot and robot operation on the space where the grid is set, will be described.

[0234] Meanwhile, as illustrated in FIG. 6c, the user interface section (5120) may provide a first user interface (5121) comprising a second layout setting area (5420) capable of placing at least one of the robot and components related to robot operation on the space where the grid is set. Here, “component” may refer to any physical or functional component placed within the space where robot operation takes place and which affects the robot’s movement, task performance, resource supply, and environmental constraints. The component according to the present invention may include at least one of equipment, structures, and operating devices that are directly or indirectly related to robot operation.

[0235] For example, the component may include at least one of a belt conveyor, a roller conveyor, and other continuous transfer equipment for transporting goods, and at least one of a storage shelf, a storage rack, a loading structure, and an automatic storage facility for storing goods. As another example, the component may include at least one of a charging station, a waiting station, and a maintenance station for charging a battery equipped on a robot, and at least one of a work station, a picking station, and a loading and unloading station for order processing and task execution. As yet another example, the component may include spatial components such as at least one of a partition structure, a wall, a safety fence, a passageway, an intersection area, and a traffic control area for partitioning a space.

[0236] In other words, the components in the present invention can be understood as a concept encompassing all elements placed within the physical space where robot operation is performed that affect the robot's movement path, work flow, or resource utilization, and are not limited to a specific industrial field or specific type of facility.

[0237] For example, the user interface unit (5120) may provide at least one component item (5421) that allows the user to select at least one of the components related to robot and robot operation to be placed in the space where the grid is to be set in the second layout setting area (5420). Specifically, the control unit (5140) may provide the second layout setting area (5420) with component items corresponding to each of the components related to robot and robot operation that are pre-stored in the storage unit (5110).

[0238] The user interface unit (5120) receives (or detects) user input for at least one component item and can specify at least one component related to a robot and robot operation corresponding to the specific component item to which the user input is applied. Furthermore, the environment layout generation unit (5141) can place a graphic object corresponding to at least one of the specified components related to a robot and robot operation in a grid-set space (5510). As an example, the first user interface (5121) is provided with a drop-down menu (5520), and the drop-down menu (5520) can be configured to select at least one component related to a robot and robot operation and place it as a drop-down in the grid-set space (5510). At this time, the drop-down menu (5520) according to the present invention may be configured to be included in at least one of the layout setting area (5400) and the layout editing area (5500).

[0239] In the present invention, the dropdown menu (5520) is illustrated as being configured separately from the second layout setting area (5420), but it is not limited thereto. The dropdown menu (5520) may be implemented to be displayed in an area separated from the second layout setting area (5420), but in other embodiments, the dropdown menu (5520) may be implemented in a form integrated with the second layout setting area (5420). For example, when user input selecting at least one component item is received, the dropdown menu (5520) may be configured to drop down a graphic object corresponding to the selected component into a grid-set space (5510) without displaying a separate dropdown window. In this case, the user can place the component at a desired location on the grid-set space (5510) while the graphic object is attached to a location indicator (or cursor). Additionally, the dropdown menu (5520) can be implemented in the form of a popup, a side panel, a toolbar, and a context menu, and can be included or nested within the second layout setting area (5420).

[0240] The method of selecting at least one of the robot and components related to robot operation according to the present invention and placing them in a grid-set space (5510) is not limited to the drop-down method described above. For example, the user interface unit (5120) may be configured to select and place at least one of the robot and components by at least one of a click method, a drag-and-drop method, a touch input method, a coordinate input method, and a shortcut key input method. In addition, the user interface unit (5120) may place a graphic object corresponding to at least one of the robot and components in a grid unit by selecting a desired grid unit (Cell) on the grid-set space in order to place at least one of the robot and components selected according to user input.

[0241] As another example, the placement of at least one of the components related to the robot and robot operation may be performed based on an external data file or a predefined template. For example, the environment layout generation unit (5141) may place multiple components simultaneously in space by loading a stored placement configuration file or by selecting a basic placement template corresponding to a specific operation scenario. The environment layout generation unit (5141) may place at least one of the components related to the robot and robot operation in a grid-set space (5510) by a user interaction method including at least one of voice input and gesture input received from a user terminal (5001).

[0242] Accordingly, the component placement method according to the present invention is not limited to a specific user interface implementation method and may include any method capable of selecting and placing at least one of a robot and a component related to robot operation on a space where a grid is set.

[0243] In the foregoing, an example of a second layout setting area for placing at least one component related to a robot and robot operation among at least one layout setting area included in the first user interface onto the space where the grid is set has been described. Below, an example of a third layout setting area for setting operation parameters related to a robot and robot operation among at least one layout setting area included in the first user interface will be described.

[0244] Meanwhile, referring again to FIG. 6b, the user interface unit (5120) may provide a first user interface (5121) including a third layout setting area (5430) capable of setting operational parameters related to robot operation. Here, “operational parameters” may refer to setting values ​​that define the robot’s operation method, work execution conditions, and dynamic characteristics of the environment. The operational parameters according to the present invention may include parameters that define at least one of work flow, resource allocation, and temporal operation conditions. For example, the operational parameters may include parameters that define a work flow, which include at least one of an order creation method, a work creation cycle, a work priority rule, a work assignment method, a work processing order, a queue size, a work distribution algorithm, whether work is processed in parallel, and whether specific work conditions are activated. Specifically, the parameters that define a work flow may include items that set a work to be created according to a certain time interval (feed interval) or set a work to be created when a specific event occurs.

[0245] As another example, operational parameters may include parameters that define the placement of physical or functional resources used for robot operation. Specifically, parameters defining resource placement may include at least one of the number of robots, the initial placement location of the robots, the number of components, the size of the buffer area, and whether a waiting area is configured. Additionally, parameters defining resource placement may include the processing capacity of each resource, the quantity that can be processed simultaneously, the number of units that can be charged, the quantity that can be stored, and the spacing criteria between resources. For example, the number of tasks that can be processed simultaneously at a specific work station, the number of robots that can be charged simultaneously at a specific charging station, or the maximum load capacity of a storage area may be defined as resource placement parameters.

[0246] As another example, the operational parameters may include parameters that define temporal operational conditions defining the temporal characteristics of robot operation. Specifically, the parameters that define temporal operational conditions may include at least one of the robot's movement speed, maximum movement speed, acceleration or deceleration, rotational speed, time required to perform a task, and the operation speed (or operation time) of a component.

[0247] The operational parameters according to the present invention are not limited to the examples described above and may include various conditions, constraints, or setting elements related to robot operation. The operational parameters may be defined in at least one form among numeric setting values, logical conditions, thresholds, rule-based settings, conditional expressions, or combined setting values, and are not limited to a specific type of data format. Furthermore, the operational parameters may be added or changed according to user settings and may be configured to be expandable to suit specific industrial environments or application scenarios. Accordingly, the operational parameters according to the present invention should be understood as a concept encompassing various setting elements for defining and controlling the robot operation environment, and should not be interpreted as being limited to specific examples.

[0248] The user interface unit (5120) can receive (or detect) user input for setting operation parameters related to robot operation in the third layout setting area (5430). Furthermore, the environment layout generation unit (5141) can set the operation parameters in the environment layout based on the operation parameters related to robot operation set by the user input. Here, setting operation parameters related to robot operation in the environment layout may mean setting operation parameters in at least one of the grid-set space (5510), the robot placed in the grid-set space, and the components related to robot operation.

[0249] Furthermore, the control unit (5140) can save the generated environment layout in a file format. At this time, the saved environment layout may be named environment layout data. The environment layout data according to the present invention forms static structural data, which is stored in the storage unit (5110) or transmitted to the environment state management unit (5142). The environment layout data functions as a reference environment for simulation and can be understood as serving as a standard for subsequent state generation and strategy calculation.

[0250] In the present invention, the format in which environment layout data is stored can be very diverse and is not limited to a specific file format. For example, the control unit (5140) may store the generated environment layout data in a structured data file format. The file format may include JSON (JavaScript Object Notation) format, XML (eXtensible Markup Language) format, YAML format, and other hierarchical data representation formats. Additionally, the environment layout data may be stored in a binary format file, a database record format, or an object serialization format, and may be stored in the storage unit (5110) or an external storage (5200).

[0251] Specifically, environment layout data may refer to a set of information including spatial structure characteristics, robot placement characteristics, component placement characteristics related to robot operation, and operational parameter characteristics, and said information may be configured to reproduce the same spatial configuration and operating conditions upon reloading. Additionally, environment layout data may include at least one of version information, creation time information, and a unique identifier.

[0252] Accordingly, the storage method for environmental layout data according to the present invention is not limited to a specific data representation method and may include all formats capable of reproducibly storing the space related to robot operation, the arrangement of the robot and components related to robot operation, and the operation parameters related to robot operation.

[0253] Furthermore, the environment state management unit (5142) can generate environment state data based on the environment layout data generated from the environment layout generation unit (5141). Specifically, the environment state management unit (5142) can generate and manage the environment state at each time step during simulation in the form of structured environment state data. For example, the environment state data may include the robot's position, speed, direction, work status, and relationship information with surrounding objects. Specifically, the environment state management unit (5142) can generate initial state information for each robot by referencing the spatial structure characteristics, robot placement characteristics, component placement characteristics related to robot operation, and operation parameter characteristics included in the environment layout data. At this time, the initial state information may include the robot's coordinate position, initial speed, direction value, current work status, or target information.

[0254] Additionally, when multiple objects exist, the environment state management unit (5142) can organize state information for each object into a state set corresponding to the object. For example, it can generate a state vector for each robot and integrate it into an overall environment state set. Furthermore, the environment state management unit (5142) can calculate relationship information between objects. For example, it can calculate the distance between robots, the distance between a robot and an obstacle, a collision probability index, the remaining distance to a specific target point, and congestion information, and include them in the environment state data.

[0255] In the foregoing, the process of generating an environment layout related to robot operation through a first user interface was described. Below, the process of interacting at least one control strategy model with at least one environment layout generated by the first user interface through a second user interface will be described.

[0256] In the present invention, a step of interoperating at least one control strategy model with at least one environment layout generated by a first user interface may be performed (see FIG. 5, S5200).

[0257] As illustrated in FIG. 7, the second user interface (5122) according to the present invention may be configured to include a selection area (5610) for selecting one of a plurality of control strategy models (5620). As previously described, a control strategy model may refer to an artificial intelligence model that receives environmental state data as input and produces a specific action according to predefined rules, learned policies, and inference results. Here, "action" may include control decisions that directly affect robot operation, such as selecting a direction of movement, determining a path, controlling speed, assigning tasks, determining priorities, and collision avoidance strategies.

[0258] For example, the control strategy model may include at least one of a rule-based algorithm, a reinforcement learning-based policy model, and a language model-based planner. The control strategy model (5620) according to the present invention may be deployed inside the simulation system (5100) or on an external server (5300). It goes without saying that the control strategy model may be named a “model” or a “behavioral model,” etc. At this time, the control strategy model is not limited to a single model, and multiple models having different structures may be operated simultaneously or selectively. In the present invention, the control strategy model is not limited to a configuration that operates independently and autonomously, but may be driven by a control unit (5140).

[0259] For example, among the multiple control strategy models (5620), a rule-based model may be configured to select an action based on the result of rule evaluation by substituting the current state into a conditional expression. As an example, the rule-based model may include at least one of a heuristic-based model, an if-then rule set-based model, a decision tree-based model, a finite state machine-based model, a priority-based scheduling model, or a pathfinding algorithm-based model.

[0260] For example, a heuristic-based model may be configured to select the action with the highest priority by evaluating indicators such as the distance between the current robot and the target point, queue length, collision risk, and resource availability. Additionally, a condition-action rule-based model may be configured to perform a predefined action when specific conditions are met. Furthermore, the rule-based model may include a shortest path search algorithm (e.g., Dijkstra's algorithm, A* algorithm, etc.) and a search algorithm based on a specific evaluation function, and these algorithms may be used to determine at least one of the robot's movement path and task sequence. Accordingly, the rule-based model in the present invention includes any form of decision structure that performs state-action mapping according to predefined rules or evaluation criteria, and is not limited to a specific implementation method.

[0261] As another example, a reinforcement learning (RL)-based policy model may be configured to input a state vector into a policy function to produce a probability distribution of actions or a continuous control value. As an example, the reinforcement learning-based policy model may include at least one of a policy-based model, a value-based model, and an actor-critic structure. For example, a policy-based model may receive environment state data as input, output a probability distribution for each action, and select a specific action based on said probability distribution. A value-based model may estimate a state-action value function (Q-value) and select an optimal action based on said value function.

[0262] As another example, a large-scale language model (LLM)-based strategy model may be configured to input environment state data into the model in the form of a prompt, and to convert the generated strategy output into a control command. The control unit (5140) may selectively apply different data transfer methods and execution methods depending on the model type.

[0263] The user interface unit (5120) can receive (or detect) a user input selecting one of a plurality of control strategy models (5620) in a selection area (5610) included in the second user interface (5122). Furthermore, the control unit (5140) can select (or specify) at least one control strategy model to be interacted with at least one environment layout generated by the first user interface based on the received user input.

[0264] The control unit (5140) can link a selected control strategy model based on at least one environment layout and standard interface generated by the first user interface. Specifically, the model linkage unit (5143) may be configured to convert environment state data provided from the environment state management unit (5142) into an input format for the control strategy model and to call the control strategy model. For example, the model linkage unit (5143) may normalize and structure the environment state data to match the input specifications required by the selected control strategy model.

[0265] More specifically, the model linkage unit (5143) may comply with standardized interface specifications applicable to a multi-agent environment. The standard interface may be formed to manage the state, behavior, reward, and state transition information of multiple agents according to a common data structure. For example, the standard interface may follow a standard API (Application Programming Interface) structure applicable to a multi-agent reinforcement learning environment, but is not limited thereto.

[0266] The standard interface may include a common execution protocol that defines the data exchange format and calling procedure between the simulation environment and the control strategy model. The execution protocol may be implemented as a cyclic interaction structure including the initialization of the simulation environment, the provision of agent-specific state information, the reception of behavior information from the control strategy model, and the generation and return of state transition results based on the behavior information.

[0267] By adopting such a common execution protocol, a decoupling structure can be formed in which the coupling between the internal implementation structure of the simulation environment and the implementation structure of the control strategy model is reduced. Accordingly, as long as the control strategy model satisfies the above standard interface specifications, it can be replaced or applied in parallel without any structural changes.

[0268] As an example, the environment state data generated by the environment state management unit (5142) is not directly transmitted to the control strategy model, but can be converted into a model input vector or tensor form through the process of extracting state features and aligning dimensions. In this way, by forming a standardized data interface between the environment state data and the control strategy model, multiple control strategy models having different structures can be interchangeably linked based on the same environment data.

[0269] In the present invention, the control strategy model is not limited to a configuration that operates independently and autonomously, but can be driven by the control unit (5140). More specifically, the model linkage unit (5143) operates under the control of the control unit (5140) and can call the control strategy model based on environment state data transmitted from the environment state management unit (5142). Accordingly, the control strategy model is an execution target managed, called, and controlled by the control unit (5140), and the model linkage unit (5143) can mediate the execution of the control strategy model.

[0270] In the foregoing, the process of interacting at least one control strategy model with at least one environment layout generated by the first user interface through the second user interface was described. Below, the process of executing a robot simulation for robot operation based on the interacted environment layout and the control strategy model, and outputting the execution result of the robot simulation in the second user interface will be described.

[0271] In the present invention, a step of executing a robot simulation for robot operation based on the mutually interconnected environment layout and the control strategy model, and outputting the execution result of the robot simulation in a second user interface may be performed (see FIG. 5, S5300).

[0272] The control unit (5140) can control the execution of a robot simulation for robot operation by mutually linking the environment layout created through the first user interface (5121) and the control strategy model selected through the second user interface.

[0273] In this case, the control strategy model may include a spatial token generation module and a temporal token generation module that generate spatial and temporal features by separating them from the data. Through this, the control strategy model can perform an attention operation that applies separated spatiotemporal attention masks to the spatial tokens and the temporal tokens. For example, the control strategy model performs rotation-based position encoding based on the local and global order of the tokens.

[0274] Additionally, the control strategy model may include a mapping module that generates contextual spatial tokens by fusing structured location information and language information that specify spatial information. The mapping module may perform a cross-attention operation between a spatial token encoded with the location information and a language token encoded with the language information.

[0275] In addition, the separation of spatial and temporal features to process data in the present invention can be utilized not only in the control strategy model but also in generating an environment layout or in other data processing. This will be described in detail later with reference to FIGS. 9 to 13.

[0276] Here, “robot simulation” may refer to a series of processes that allow tasks (or actions) performed by a robot in a specific environment layout to be carried out in a virtual space. As an example, robot simulation may include virtually reproducing operational processes in automated logistics systems, smart warehouse environments, production line automation environments, multi-robot collaboration environments, autonomous vehicle operation environments, or other environments where physical AI is applied.

[0277] Specifically, the robot simulation may include various workflows performed by multiple robots interacting within the same space. For example, the robot simulation may enable various tasks to be performed in a virtual space, such as material transfer, loading and unloading, order picking and stowing, material transfer between production processes, collaborative work between work stations, pathfinding and collision avoidance, task assignment and reassignment, priority determination, queue management, resource distribution, and energy management.

[0278] Therefore, robot simulation in the present invention is not limited to a specific type of work or a specific industrial field, but can be understood as a concept that encompasses the overall execution process of implementing and evaluating the operation process of a robot performed in space within a virtual environment.

[0279] Referring again to FIG. 7, the second user interface (5122) according to the present invention may include a simulation control area (5630) for controlling a robot simulation for robot operation. At this time, the simulation control area (5630) may be configured to control the execution state, execution method, and simulation conditions (e.g., simulation execution speed, etc.) of the robot simulation. For example, the simulation control area (5630) may include an execution control item for controlling the start, pause, resume, or termination of the simulation. Additionally, the simulation control area (5630) may include a speed control item for adjusting the execution speed of the simulation, a step execution item for proceeding with execution in units of a certain unit of time or steps, and an initialization item for restoring to an initial state. Furthermore, the user interface unit (5120) may control the robot simulation for robot operation based on user input received through the simulation control area (5630).

[0280] Specifically, the control unit (5140) can control a robot simulation for robot operation using the simulation execution unit (5144). For example, the control unit (5140) can request the model linkage unit (5143) to execute a model by using the progression of a time step or the occurrence of an event of the simulation as a trigger. Accordingly, the control strategy model is driven at each time step or whenever a specific condition is satisfied, so that an action selection for the current state can be performed.

[0281] In the present invention, “action selection” means a decision-making process that determines one of a plurality of control strategies possible in a task (or operation) to be simulated, such as the operation of a robot or an autonomous vehicle, based on environmental state data at the current time.

[0282] Specifically, the control unit (5140) manages the time step progression of the simulation and, when a control command is received from the control strategy model, may request the simulation execution unit (5144) to perform a state update operation. Accordingly, the simulation execution unit (5144) may perform a state transition operation based on the current state S(t) and the control command A(t) and generate the next state S(t+1).

[0283] Here, the control command A(t) corresponds to the result of the action selection performed by the control strategy model. That is, the action selection refers to a decision-making process of selecting one of multiple possible control alternatives based on the current environmental state S(t). The control strategy model can take state data as input and produce a specific action A(t) according to a policy function, a rule set, or an inference result.

[0284] The selected action A(t) is converted into an executable control command by the control unit (5140) and transmitted to the simulation execution unit (5144), and the next state S(t+1) can be determined by applying the action as an input to the state update operation logic. Therefore, even if the same initial state is given, if the result of the action selection is different, the subsequent state transition path and simulation development may differ.

[0285] The state update operation logic may be included in the simulation execution unit (5144) and may include operation logic that calculates the next environment state based on the current environment state and control commands. In this case, the state update operation logic may be implemented in various ways. As an example, the state update operation logic may include a movement operation that calculates a movement result based on the position, speed, and direction information of the robot (or autonomous vehicle).

[0286] As another example, the state update operation logic may include interaction determination operations such as calculating the distance between objects, determining whether a collision has occurred, determining a passable area, or determining whether a task has been completed. As yet another example, the state update operation logic may include state updates based on predefined rules, calculation of state changes based on probabilistic transition rules, and state prediction operations based on a learned transition function.

[0287] The control unit (5140) receives state update data from the simulation execution unit (5144), reflects it in the environment state management unit (5142), and can call the control strategy model again based on the updated state. Accordingly, a repeating structure of environment state, strategy model execution, strategy selection, control execution, state update, and re-call is formed, and the control strategy model can be repeatedly executed during the simulation process.

[0288] For example, the control strategy model can perform an attention operation that applies separate spatiotemporal attention masks to spatial and temporal tokens. Furthermore, the control strategy model can perform rotation-based position encoding based on the local and global order of tokens.

[0289] As another example, the control strategy model may include a mapping module that generates contextual spatial tokens by fusing structured location information and linguistic information that specify spatial information. Specifically, the mapping module can perform cross-attention operations between spatial tokens encoded with location information and language tokens encoded with linguistic information.

[0290] In this way, the control unit (5140) can execute a robot simulation for robot operation by repeatedly driving a control strategy model based on environmental state data and reflecting the control command accordingly in the simulation execution unit (5144), based on a plurality of control strategy model structures. Accordingly, the present invention provides an integrated simulation execution structure that is not limited to a specific algorithm or a specific model structure, but can repeatedly drive various types of control strategy models under the same environmental state and execution structure.

[0291] As a result, the control unit (5140) can perform a complete simulation of robot operation by integrally controlling the environment layout, environment state management, strategy model linkage, and state update operation.

[0292] For the above, the process of executing a robot simulation for robot operation based on the interconnected environment layout and control strategy model has been described. Below, the process of outputting the execution results of the robot simulation to the second user interface will be described.

[0293] The control unit (5140) can execute a robot simulation for robot operation and output the execution result of the robot simulation in the second user interface. Referring again to FIG. 7, the second user interface (5122) according to the present invention may be configured to have an animation display area (5700) that displays the movement state of the robot in an animation form.

[0294] Specifically, the user interface unit (5120) may be configured to visually display the position, direction, speed, and work status of the robot at each time step in the animation display area (5700) based on state update data transmitted from the simulation execution unit (5144). For example, the user interface unit (5120) may display the position of at least one robot in a space where a grid is set in the animation display area (5700) as a robot graphic object corresponding to each robot, and implement the movement process in the form of an animation by continuously updating the position of the graphic object as the time step progresses. At this time, the movement path of the robot may be displayed in the form of a line or trajectory, and the target point or work target object may be distinguished through a separate visual identification means (color, icon, mark, etc.).

[0295] Additionally, the user interface unit (5120) may be configured to visually display when a pre-set event (e.g., occurrence of a crash, completion of a task, entry into a waiting state, or violation of a constraint) occurs. For example, the user interface unit (5120) may highlight the location when a crash occurs, or output a completion indicator icon to the animation display area (5700) when a task is completed.

[0296] Furthermore, when multiple control strategy models are executed in parallel, the user interface unit (5120) can distinguish the robot movement state corresponding to each strategy model and display it in the animation display area (5700). For example, the user interface unit (5120) can be configured to display animations of each of the executed multiple control strategy models in parallel through different colors or distinct display areas, so that the operational differences between the executed control strategy models in the animation display area (5700) can be intuitively compared.

[0297] Accordingly, the animation display area (5700) according to the present invention is not limited to a simple position display function, but can be understood as a visualization area that visually reproduces dynamic state information updated according to the time flow of the simulation, allowing the user to intuitively check the robot operation process and strategic effects.

[0298] Additionally, the second user interface (5122) according to the present invention may be configured to further include an indicator display area (5800) that displays performance indicators related to robot operation. Specifically, the control unit (5140) may receive performance indicator data calculated through the simulation execution unit (5144) and transmit it to the user interface unit (5120) to control output to the indicator display area (5800). For example, the indicator display area (5800) may display performance indicators including at least one of average order picking time, number of picking completions or completion rate, average stowing time, number of stowing completions or completion rate, average total work completion time, and cumulative rewards (RL).

[0299] At this time, the performance indicators provided in the indicator display area (5800) may be configured differently depending on the type of work processed by the robot. For example, if the robot operation includes an order picking task, the performance indicators may include at least one of the average order picking time, the number of completed pickings per unit time, the length of the picking queue, or the success rate of the picking task. As another example, if the robot operation includes a stowing task, the performance indicators may include at least one of the average stowing time, the number of completed stowings per unit time, the storage space utilization rate, or the time required for inventory reflection.

[0300] Accordingly, the performance indicators provided in the indicator display area (5800) according to the present invention are not limited to a specific type of work, and can be configured to dynamically configure and display different sets of performance indicators depending on the characteristics of the work performed by the robot.

[0301] In the above, the process of outputting the execution results of the robot simulation in the second user interface was explained.

[0302] Meanwhile, the physical AI-based simulation method according to the present invention may include a step of executing a driving decision of at least one robot based on the execution result of the robot simulation. That is, the physical AI-based simulation method may include a process of generating a control command corresponding to the driving decision of an actual robot. Below, the process of executing a driving decision of at least one robot based on the execution result of the robot simulation will be described.

[0303] The control unit (5140) can execute a driving decision for at least one robot based on the execution results of the robot simulation. Specifically, the control unit (5140) can generate multiple candidate data in a search mode based on the execution results of the robot simulation. Here, “search mode” may refer to a mode in which a preliminary review process is performed to generate and evaluate multiple possible control alternatives before executing a driving decision for the robot. For example, the control unit (5140) can generate multiple candidate data in a search mode based on at least one of the state transition path and action selection results selected (or calculated, derived) through the robot simulation.

[0304] At this time, the candidate data may include at least one of a robot driving path candidate, a speed control value candidate, an acceleration and deceleration profile candidate, a task execution sequence candidate, a task assignment candidate, an avoidance strategy candidate, and a control command sequence candidate. For example, the candidate data may include different sets of paths to reach the same target point, and for each path, evaluation indicators such as estimated time required, collision risk, energy consumption, and resource occupancy time may be included.

[0305] Furthermore, candidate data is not limited to control commands at a single point in time but can be configured in the form of a continuous sequence of control commands applied over a certain time interval. For example, a set of movement commands along a specific path, a set of speed change commands, and a set of avoidance action commands can be defined as a single set of candidate data.

[0306] The control unit (5140) can determine whether to terminate the search mode by comparing candidate data with a preset transition specification and activate a decision mode that executes a driving decision of the robot. Here, “decision mode” may refer to a mode in which at least one of the multiple candidate data generated in the search mode is determined as a final control alternative and executed as an actual driving decision.

[0307] Specifically, the control unit (5140) may select at least one candidate according to evaluation criteria if there is a candidate among multiple candidate data that satisfies the transition specifications. For example, the process of selecting at least one candidate may be performed based on at least one of minimum collision risk, minimum estimated time required, minimum energy consumption, maximum resource utilization efficiency, or a predefined priority rule.

[0308] The control unit (5140) can determine a control command corresponding to selected candidate data in response to the activation of the decision mode, and transmit the determined control command to at least one robot.

[0309] Below, an example of Order Picking to which the present invention is applied is described. Order Picking may refer to an operation in which multiple robots move between shelves and work stations within a warehouse to selectively pick products included in an order. In this example, a logistics environment in which such Order Picking is operated is simulated in a virtual environment, and a simulation is performed to evaluate the performance of the control strategy model.

[0310] FIG. 8a is a conceptual diagram illustrating the conceptual configuration of a physical AI-based simulation system (5100) applied to an Order Picking scenario.

[0311] As seen in FIG. 4, the first user interface (5121) may be configured as an input screen for setting an environment layout. The first user interface (5121) may receive environment setting information including a grid structure of the warehouse, shelf placement locations, work station locations, robot initial placement information, and operation parameters. The setting information may be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8110).

[0312] The left area of ​​FIG. 8a illustrates an order picking environment configured by the environment layout data (8110). This environment may represent a warehouse space in which a plurality of shelves, work stations, and robots are arranged, and this may function as a reference space structure for the simulation.

[0313] The right area of ​​FIG. 8a shows a simulation result report (e.g., performance indicator data) (8120) produced as a result of running a simulation. As seen in FIG. 4, the second user interface (5122) may be configured as a screen that visually outputs the performance indicator data. The performance indicator data (8120) may include average order picking time, picking completion rate, robot status distribution, cumulative rewards, etc.

[0314] Although not explicitly shown, when multiple control strategy models are applied, the performance indicator data (8120) may be linked with the strategy model identifier and displayed separately by strategy, or displayed for comparison in the form of a parallel or overlapping graph. Accordingly, performance differences by strategy can be intuitively verified under the same environment layout (8110) conditions.

[0315] FIG. 8b is a flowchart illustrating a process executed in a time-step-based closed-loop structure in the Order Picking illustrated in FIG. 8a. Referring to FIG. 8b, the system (5100) according to the present invention can receive configuration information for an order picking environment through a first user interface (5121) (S8110). The configuration information may include a grid structure of a warehouse, shelf placement locations, work station locations, robot initial placement information, and operational parameters, which can be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8110).

[0316] The generated environment layout data (8110) may be stored in the storage unit (5110) or provided to the environment state management unit (5142). The control unit (5140) may perform simulation environment initialization and data loading (S8120). In this case, the control unit (5140) may generate an initial environment state S(0)(8111) based on the environment layout data (8110). The initial environment state may be composed of a set of state vectors including the position, velocity, work status, and order queue information of a plurality of robots.

[0317] When the simulation is initiated, the control unit (5140) can manage time steps, and at each time step t, the environment state management unit (5142) can configure the current environment state S(t) (8112) (S8130). The environment state S(t) can be expressed as structured data including the position and work status of each robot, order waiting and progress status, distance between robots, collision risk, etc.

[0318] The model linkage unit (5143) can generate model input data (8113) by receiving the environment state S(t) (8112) and performing normalization and feature extraction to match the input specifications of the control strategy model (S8140). The generated model input data (8113) can be transmitted to one or more control strategy models.

[0319] The control strategy model may be at least one of a rule-based model, a reinforcement learning-based policy model, or a large-scale language model-based strategy model, and if multiple control strategy models are selected, the same environment state S(t) may be provided in common to each strategy model. Each strategy model may independently calculate an action A_i(t) corresponding to the current state S(t) (S8150, S8160). The action may include selecting a picking target, assigning tasks per robot, determining movement goals and paths, etc.

[0320] The control unit (5140) can convert the action A_i(t) calculated from each strategy model into executable control command data (8116) (S8170). Subsequently, the simulation execution unit (5144) can receive the current environment state S(t) (8112) and the control command data (8116) corresponding to each strategy and apply a state transition model (S8180). The state transition model may include movement operations, collision detection, and determination of whether a task is completed, and may generate state update data (8117) for each strategy. However, in this example, the state transition model is for convenience of explanation, and any one of the aforementioned state update operation logic or examples of logic may be applied.

[0321] Accordingly, the next environment state S_i(t+1) (8118) corresponding to each strategy model can be generated (S8190). The generated state can be managed again by the environment state management unit (5142) and used as the input state for the next time step. This process of updating the state and calling the strategy model can be repeated until a termination condition is satisfied, thereby forming a closed-loop structure that transitions from environment state S(t) to S_i(t+1).

[0322] However, as mentioned above, the closed-loop simulation structure having a time-step-based iterative structure is merely an example, and structures based on event occurrence are also possible.

[0323] The simulation execution unit (5144) can calculate compensation data (8119) by applying a compensation function R_i(t) for each strategy at each time step (S8191). The compensation function can be defined by considering whether an order is completed, the average order picking time, whether a conflict occurs, the waiting time, etc., and can be managed as a cumulative compensation value linked to a strategy model identifier.

[0324] In addition, performance indicator data (8120), including average order picking time, picking completion rate, robot status distribution, and cumulative reward value, can be calculated for each strategy (S8192). The generated performance indicator data (8120) can be visually provided through a second user interface (5122) (S8220), and if multiple strategy models are executed in parallel, the performance indicators for each strategy can be displayed for comparison in the form of parallel or overlapping graphs.

[0325] Accordingly, the flow illustrated in FIG. 8b can be configured as a time-step-based iterative closed-loop structure that includes starting from the creation of an environment layout, the creation of an initial state, parallel operation of strategy models, state transitions, compensation calculation, and performance indicator calculation, and can enable quantitative comparison and evaluation of multiple strategy models for an Order Picking scenario.

[0326] Thus, the present invention can be configured to provide the same environment layout (8110) and the same time step environment state S(t) (8112) to a plurality of control strategy models in common, while maintaining state transition paths and reward calculation paths corresponding to each strategy model independently of one another. In this way, the present invention is not limited to a structure that observes the execution results of a single strategy, but can provide a structure that drives multiple strategies in parallel under the same conditions and can individually generate and manage the action A_i(t), next state S_i(t+1) (8118), reward data (8119), and performance indicator data (8120) according to each strategy in conjunction with the strategy model identifier. Accordingly, it is structurally guaranteed that performance differences between strategies are caused by the strategy selection itself rather than variations in environmental conditions, and fairness, reproducibility, and the possibility of quantitative analysis of strategy comparisons can be simultaneously secured. The structure of providing the same state in common and maintaining independent closed loops for each strategy can form technical features as a strategy evaluation platform that is distinguished from simple simulation execution.

[0327] The following describes Stowing to which the present invention is applied by way of example. A Stowing embodiment may refer to an operation in which multiple robots transport goods received at an incoming station to appropriate shelf locations within a warehouse for stacking. In this example, a logistics environment in which such Stowing is operated is simulated in a virtual environment, and a simulation is performed to evaluate the performance of the control strategy model.

[0328] FIG. 8c is a conceptual diagram illustrating the conceptual configuration of a physical AI-based simulation system (5100) applied to a Stowing scenario.

[0329] As seen in FIG. 4, the first user interface (5121) may be configured as an input screen for setting an environment layout. In the Stowing embodiment, the first user interface (5121) may receive environment setting information including a grid structure of the warehouse, shelf placement locations, receiving station locations, robot initial placement information, and operation parameters. The setting information may be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8230).

[0330] The left area of ​​FIG. 8c may illustrate a stacking environment configured by the environment layout data (8230). The environment may include a spatial structure in which goods arrive at a receiving station and multiple robots move to a stacking target shelf to stack the goods.

[0331] The right area of ​​FIG. 8c may show performance indicator data (8240) calculated as a result of running a simulation. The performance indicator data (8240) may include average stacking time, stacking completion rate, robot state distribution, cumulative reward, etc.

[0332] When multiple control strategy models are applied, the performance indicator data (8240) may be linked to the strategy model identifier and displayed separately by strategy, or displayed for comparison in the form of a parallel or overlapping graph. Accordingly, differences in stacking performance by strategy can be verified under the same environment layout (8230) conditions.

[0333] FIG. 8d is a flowchart illustrating the process of executing the Stowing embodiment shown in FIG. 8c in a time-step-based closed-loop structure.

[0334] Referring to FIG. 8d, the system (5100) can receive configuration information for a storage environment through a first user interface (5121) (S8210). The configuration information can be interpreted by an environment layout generation unit (5141) and generated as environment layout data (8230).

[0335] The control unit (5140) can generate an initial environment state S (0) based on the environment layout data (8230) (S8193), and the initial state can be represented by a set of state vectors (8231) including robot position, a list of products waiting to be stacked, and information on the remaining space of the shelf.

[0336] When the simulation is initiated, the control unit (5140) can manage time steps, and at each time step t, the environment state management unit (5142) can configure the current environment state S(t) (8232) (S8230). The environment state may include the position of each robot, the quantity and position of goods waiting to be stacked, shelf occupancy rate, distance between robots, collision risk, etc.

[0337] The model linkage unit (5143) can generate model input data (8233) by receiving the environment state S(t) (8232) and performing normalization and feature extraction (S8240).

[0338] There may be one or more control strategy models, and if multiple control strategy models are selected, the same environmental state S(t) (8232) may be provided in common to each strategy model (S8250). Each strategy model may independently calculate an action A_i(t) corresponding to the current state (S8260), and said action may include selecting a shelf to be stacked, robot assignment, movement goal and path determination, etc.

[0339] The control unit (5140) can convert the action A_i(t) into executable control command data (8234) (S8270).

[0340] The simulation execution unit (5144) can receive the current environment state S(t) (8232) and control command data (8234) and apply a state transition model (S8280). The state transition model may include movement operations, collision detection, and determination of whether stacking is complete, and may generate state update data (8235).

[0341] Accordingly, the next environment state S_i(t+1)(8236) corresponding to each strategy model can be generated (S8290). The generated state can be managed again by the environment state management unit (5142) and used as the input state for the next time step.

[0342] Additionally, the simulation execution unit (5144) can calculate compensation data (8237) by applying a strategy-specific compensation function R_i(t) (S8300). The compensation function can be defined by considering the stacking completion time, space utilization efficiency, and whether a collision occurs.

[0343] In addition, performance indicator data (8240) for each strategy can be calculated (S8310). The generated performance indicator data (8240) can be visually provided through the second user interface (5122) (S8320), and can be displayed to compare the stacking performance for each strategy when multiple strategy models are executed in parallel.

[0344] The above process can be repeated until the termination condition is satisfied, and accordingly, a closed-loop simulation structure can be formed that can quantitatively compare and evaluate the stacking performance of multiple strategy models under the same environment layout (8230) conditions.

[0345] As described above, the present invention provides the same environment layout data (8230) and the same time step environment state S(t) (8232) to a plurality of control strategy models in common, while maintaining state transition paths and performance evaluation paths formed based on behavioral information generated by each control strategy model independently of each other.

[0346] For example, the state transition path and the compensation calculation path may be maintained independently of each other according to the results of the stacking target selection, robot assignment, and movement path determination corresponding to each strategy model. That is, the present embodiment is not limited to a structure that simply observes the execution results of a single stacking strategy, but can provide a structure that drives multiple strategies in parallel under the same receiving conditions, the same shelf arrangement structure, and the same initial state conditions, and can individually generate and manage the next state S_i(t+1) (8236), compensation data (8237), and performance indicator data (8320) according to each strategy in conjunction with the strategy model identifier.

[0347] In particular, since stacking operations are closely related to shelf remaining space constraints, the possibility of collisions between robots, and space utilization efficiency, differences in shelf selection criteria or robot allocation methods based on strategies can directly affect changes in space occupancy rates and work completion times. This embodiment structurally ensures fairness and reproducibility in strategy selection by combining a state transition model that reflects such physical space constraints to enable the comparison of stacking performance by strategy under identical conditions. Accordingly, this embodiment can function as a strategy comparison platform capable of quantitatively evaluating space-utilization-based logistics operation strategies, going beyond simple stacking simulations.

[0348] The following describes a Fulfillment Center to which the present invention is applied by way of example. This embodiment exemplifies a simulation structure for comprehensively simulating a complex logistics operating environment in which order picking and receiving stacking operations are performed simultaneously in a virtual environment, and for comprehensively evaluating the performance of multiple control strategy models.

[0349] FIG. 8e is a conceptual diagram illustrating the conceptual configuration of a physical AI-based simulation system (5100) applied to a Fulfillment Center scenario.

[0350] As seen in FIG. 4, the first user interface (5121) may be configured as an input screen for setting an environment layout. In this embodiment, the first user interface (5121) may receive environment setting information including a warehouse grid structure, shelf arrangement, locations of receiving and shipping stations, robot initial placement information, and operation parameters. The setting information may be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8410).

[0351] The left area of ​​FIG. 8e can illustrate an integrated logistics environment configured by the environment layout data (8410). The environment includes an outbound area for order picking and a stacking area for inbound goods stacking, and can be represented as a spatial structure in which multiple robots can perform picking and stacking operations in parallel. Accordingly, a complex operation scenario in which outbound and inbound operations occur simultaneously within the same environment can be implemented.

[0352] The right area of ​​FIG. 8e may display performance indicator data (8420) generated as a result of simulation execution. The performance indicator data (8420) may include average total completion time, average order picking time, average stowing time, picking completion rate, stowing completion rate, robot status distribution, etc. When multiple control strategy models are applied, the performance indicator data (8420) may be linked to strategy model identifiers and displayed separately by strategy, or displayed in the form of a parallel or overlapping graph for comparison. Accordingly, integrated operational performance by strategy can be intuitively compared under the same environment layout (8410) conditions.

[0353] FIG. 8f is a flowchart illustrating the process of a strategy model running in parallel in a time step-based closed-loop structure in the Fulfillment Center integrated environment shown in FIG. 8e.

[0354] Referring to FIG. 8f, the system (5100) can receive configuration information for an integrated logistics environment through a first user interface (5121) (S8410). The configuration information can be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8410). The generated environment layout data (8410) can be stored in a storage unit (5110) or provided to an environment state management unit (5142).

[0355] The control unit (5140) can generate an initial environment state S(0)(8411) based on the environment layout data (8410) (S8420). The initial environment state may be composed of an integrated state vector including the position and work status of multiple robots, order waiting queue information, product waiting for incoming goods information, and shelf occupancy information.

[0356] When the simulation is initiated, the control unit (5140) can manage time steps, and at each time step t, the environment state management unit (5142) can configure the current environment state S(t) (8412) (S8430). The environment state S(t) can be represented as structured data that includes both a picking operation state and a stacking operation state.

[0357] The model linkage unit (5143) can generate model input data (8413) by receiving the environment state S(t) (8412) as input and performing normalization and feature extraction (S8440). The model input data (8413) can be provided to a control strategy model.

[0358] When multiple control strategy models are selected, the control unit (5140) can provide the same environmental state S(t) (8412) to each strategy model in common (S8450). Each strategy model can independently calculate an action A_i(t) that considers picking and stacking operations simultaneously (S8460). The action may include assigning tasks to each robot, determining picking priority, selecting stacking target shelves, and determining movement paths.

[0359] The control unit (5140) can convert the action A_i(t) into executable control command data (8414) (S8470). The simulation execution unit (5144) can apply the control command data (8414) to perform a state transition model and generate state update data (8415) (S8480). Accordingly, the next environment state S_i(t+1) (8416) can be generated (S8490).

[0360] Additionally, the simulation execution unit (5144) can calculate compensation data (8417) by applying a strategy-specific compensation function R_i(t) (S8500). The compensation function can be defined by considering the total completion time, picking performance, stacking performance, whether a collision occurs, resource utilization efficiency, etc.

[0361] Subsequently, the control unit (5140) can calculate performance indicator data (8420) for each strategy (S8510), and the performance indicator data (8420) can be visually provided through the second user interface (5122) (S8520). The above process can be repeated until a termination condition is satisfied, and accordingly, a closed-loop simulation structure can be formed that can quantitatively compare and evaluate the overall operational performance of multiple strategies under the same integrated environment conditions.

[0362] Below, an Automated Production Line to which the present invention is applied is described by way of example. This embodiment illustrates a simulation structure for simulating the process of processing, transporting, and shipping products in a virtual environment by multiple robots and equipment collaborating in an automated production line environment, and for evaluating the performance of a control strategy model.

[0363] FIG. 8g is a conceptual diagram illustrating the conceptual configuration of a physical AI-based simulation system (5100) applied to an Automated Production Line scenario.

[0364] As seen in FIG. 4, the first user interface (5121) may be configured as an input screen for setting an environment layout. In this embodiment, the first user interface (5121) may receive environment setting information including a grid structure of a production line, a location for the placement of processing equipment, a transfer path, initial robot placement information, and operational parameters. The setting information may be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8610).

[0365] The left area of ​​FIG. 8g may illustrate an automated production line environment configured by the environment layout data (8610). The environment may include a plurality of processing equipment, a transfer conveyor, a robot, and a work area, and may include a structure in which products are moved and processed according to the process sequence.

[0366] More specifically, referring to FIG. 8g, based on production line environment setting information input through the first user interface (5121), the environment layout generation unit (5141) can generate environment layout data (8610) that defines the spatial structure of an automated production line. The environment layout data (8610) may include production equipment placement information, transfer path information, robot initial placement information, and operation parameter information. In the present invention, the production line environment configured based on the environment layout data (8610) can form an automated process structure including a plurality of processing equipment, transfer means, and robots. The environment can reflect a work flow in which products are moved and processed according to the process sequence.

[0367] Additionally, on the right side of FIG. 8g, performance indicator data (8620) calculated by applying a strategy model in the above environment may be illustrated. The performance indicator data (8620) may include throughput, total deliveries, robot status, and cumulative rewards. When multiple control strategy models are applied, the performance indicator data (8620) may be displayed separately according to a strategy model identifier.

[0368] FIG. 8h is a flowchart illustrating the process of performing a simulation in a time-step-based closed-loop structure in the Automated Production Line scenario illustrated in FIG. 8g. Referring to FIG. 8h, production line environment setting information can be input through a first user interface (5121) (S8610), and the setting information can be interpreted and structured by an environment layout generation unit (5141) to generate environment layout data (8610). A control unit (5140) can generate an initial environment state S(0) (8611) based on the environment layout data (8610) (S8620), and the initial environment state can be represented as a structured state vector including robot position, equipment operating status, quantity of products waiting in the process, and process sequence information.

[0369] At each time step t, the environment state management unit (5142) can configure the current environment state S(t) (8612) (S8630), and the environment state can be expressed as integrated state data including equipment state, product location, robot work state and process progress status. The model linkage unit (5143) can generate model input data (8613) by receiving the environment state S(t) (8612) and performing normalization and feature extraction (S8640), and the model input data can be provided to the control strategy model.

[0370] If multiple control strategy models exist, the control unit (5140) can provide the same environmental state S(t) (8612) to each strategy model in common (S8650), and each strategy model can independently calculate an action A_i(t) corresponding to the current state (S8660). The action may include production process assignment, robot path determination, work priority adjustment, etc. The control unit (5140) can convert the action A_i(t) into executable control command data (8614) (S8670), and the simulation execution unit (5144) can apply the control command data (8614) to perform a state transition operation and generate state update data (8615) (S8680). Accordingly, the next environmental state S_i(t+1) (8616) can be generated (S8690).

[0371] Additionally, the simulation execution unit (5144) can calculate compensation data (8617) by applying a strategy-specific compensation function R_i(t) (S8700), and the control unit (5140) can calculate performance indicator data (8620) based on the compensation data (8617) (S8710). The performance indicator data (8620) may include throughput, total delivery quantity, robot status, and cumulative compensation, and may be visually provided through the second user interface (5122) (S8720). The above process may be repeated until a termination condition is satisfied, and accordingly, a closed-loop simulation structure can be formed that allows for the quantitative comparison of the production efficiency of multiple strategies under the same production line environment conditions.

[0372] Meanwhile, in the present invention, scene data may be generated by at least one of an environment layout generation unit (5141), an environment state management unit (5142), a model linkage unit (5143), and a simulation execution unit (5144). The scene data may include data that visually represents changes in the environment state or represents an animation that is a simulation result.

[0373] According to one embodiment of the present invention, the scene data may be provided to an analysis module (400) illustrated in FIG. 9, and the analysis module (400) may be configured to take the scene data as input, separate and encode spatial information and temporal information, and perform a spatial-language mapping operation.

[0374] The above scene data may be static frame data corresponding to a single time step, or dynamic sequence data generated continuously over multiple time steps.

[0375] The above scene data may be defined as data that visually represents at least one of the object's location, movement path, occupied area, distance relationship between objects, collision status, or work progress status.

[0376] In the present invention, “scene data” may include at least one of (i) initial scene data generated immediately after object placement is set through the first user interface (5121), (ii) intermediate state scene data generated at time steps during the execution process of the control strategy model, and (iii) scene sequence data reconstructed based on the execution history of each strategy after the simulation ends.

[0377] The above scene data may be implemented in at least one form among pixel-based image data, object-unit graphic rendering data, or visual representation data reconstructible based on state vectors, and is not limited to a specific representation format.

[0378] The objects included in the above scene data may include at least one of a robot, an autonomous vehicle, a drone, equipment, a work object, or a spatially constrained structure, and the objects may refer to entities corresponding to the state management unit of the simulation.

[0379] Scene data defined as above may be provided as input to an analysis module (400), and the analysis module (400) may be configured to interpret spatiotemporal relationships by separating and encoding spatial information and time information included in the scene data.

[0380] The analysis module (400) according to the present invention may be named in various ways, such as a spatiotemporal analysis module, a scene analysis module, a spatiotemporal analysis module, a spatial-language analysis module, a simulation scene analysis module, etc.

[0381] The analysis results may be optionally provided to the strategy control module (5130), the simulation execution unit (5144), or the storage unit (5110).

[0382] According to one embodiment of the present invention, the scene data may be provided to an analysis module (400) illustrated in FIG. 9, and the analysis module (400) may be configured to take the scene data as input, separate and encode spatial information and temporal information, and perform a spatial-language mapping operation. According to one embodiment, the analysis module (400) may be included in a simulation system (5100) and configured to operate under the control of a control unit (5140).

[0383] FIG. 9 is a block diagram illustrating the configuration of an analysis module (400) according to one embodiment of the present invention. More specifically, FIG. 9 is a block diagram of an analysis module (400) implementing a scene data analysis function based on separate interpretation of spatial features and temporal features according to another embodiment. FIG. 10 illustrates an exemplary configuration of an analysis architecture that performs scene data analysis based on separate interpretation of spatial features and temporal features according to one embodiment. FIG. 11 is intended to explain a method for encoding location information for an input sequence of a large language model according to one embodiment. FIG. 12 is a block diagram illustrating the configuration of a spatial-language mapping module (17) according to one embodiment. FIG. 13 is intended to explain the function of a spatial-language mapping operation module (22) according to one embodiment.

[0384] Referring to FIG. 9, the simulation system (5100) may include a user interface unit (5120) and an analysis module (400), and the analysis module (400) may include at least one of a vision feature extraction module (10), a global token generation module (11), a spatial token generation module (12), a time token generation module (13), a text token generation module (14), an input sequence configuration module (15), a large language model processing module (16), a space-language mapping module (17), and a prompt generation module (18).

[0385] Additionally, referring to FIG. 10, the scene data analysis model framework used by the analysis module (400) according to one embodiment for scene data analysis may include the architecture of a multimodal vision-language model (VLM) that performs complex spatiotemporal inference by processing input scene data and text data in an integrated manner.

[0386] This architecture processes images using a vision encoder (e.g., CLIP) to extract visual features, which can then be separated into spatial and temporal tokens through processes such as spatial pooling and temporal pooling. The text tokens generated by passing these prepared visual tokens and text provided by the user through a tokenizer and a text embedding module can then be input into a large language model. Subsequently, positional information can be encoded by applying Harmonic-Rotary Position Embedding to the input sequence, and a specific data fusion operation (DST Attention) can be performed to separate spatiotemporal information.

[0387] The vision feature extraction module (10) can perform the role of extracting visual features necessary for subsequent analysis and language model training from input scene data. The vision feature extraction module (10) may include a pre-trained vision encoder and a projector that connects the feature vector generated by the vision encoder to the dimensions of a large language model.

[0388] The vision feature extraction module (10) can process input scene frames (or scene frames) for each time step to capture the content and meaning of the corresponding image. Referring to FIG. 10, the vision feature extraction module (10) can extract a high-dimensional feature vector (X) that captures meaningful information such as objects, actions, and scenes within the image by utilizing a model such as CLIP (Contrastive Language-Image Pre-training), for example.

[0389] The extracted high-dimensional feature vector (X) passes through a projector (or a trainable projection layer) and can be transformed into a token embedding (z) optimized for vision-language learning. The projector performs a key processing role in aligning the dimensions of the high-dimensional feature vector extracted from the vision encoder with the dimensions of text tokens in a massive language model (LLM), ensuring that visual meaning is precisely aligned within a linguistic context.

[0390] The vision features extracted by the vision feature extraction module (10) can have a high-dimensional vector form that captures meaningful information such as objects, actions, and scenes within the image, going beyond mere pixel information of the image. These features can then be transmitted to the spatial token generation module (12) and the temporal token generation module (13) to be separated and processed into tokens containing spatiotemporal information.

[0391] The global token generation module (11) can generate a frame-by-frame global token that summarizes the overall content of each time point based on the visual features of the image obtained from the vision feature extraction module (10).

[0392] The global token generation module (11) can be designed to represent the macroscopic context or key content that penetrates the entire sequence in a single compressed form, rather than being limited to the local features of each frame or patch of the image. Referring to FIG. 10, for example, the global token generation module (11) can generate Classification Tokens (CLS tokens) that can function as vectors containing comprehensive information about the entire image.

[0393] Global tokens can be generated through a pooling process of extracted visual features. This pooling process, including weight-based summing, maximum value extraction, or average calculation, condenses the most important information from numerous spatial and temporal tokens to produce a single or small number of global token vectors. The global tokens generated through the global token generation module (11) can then be combined with spatial and temporal tokens and input into a large language model.

[0394] Global tokens serve as representatives containing the high-level context of an image, encouraging the model to consider the overall situation when deriving answers to complex queries (e.g., prompts) or image analysis results. This enables the model to perform consistent inference within the overall context rather than relying solely on partial features, thereby playing a key role in improving the accuracy and reliability of the analysis.

[0395] The spatial token generation module (12) can perform the role of separating and encoding spatial information within each frame of input scene data based on visual features extracted through the vision feature extraction module (10). The spatial token generation module (12) can generate spatial tokens that can identify spatial correlations within the same point in time (frame).

[0396] Referring to FIG. 10, the spatial token generation module (12) can generate spatial tokens by applying spatial pooling to the extracted vision feature maps, and thus the spatial tokens can include details related to the local context within each frame and the shape and location of the object.

[0397] For example, the spatial token generation module (12) divides the token embedding (z) from the projector into multiple segments (e.g., 4) along the time axis. Temporal pooling is performed within each segment to condense the visual features of the corresponding interval, and then k×k spatial pooling (e.g., 2×2) is applied to the entire interval to produce the final spatial token. The spatial token generated through this series of spatial pooling processes can clearly capture not only the local context within the frame but also static scene features that penetrate the entire image.

[0398] When the generated spatial tokens are subsequently processed through a disentangled spatial and temporal attention mask in a large language model, they can be controlled to perform attention operations only with other spatial tokens within the same frame, without performing attention operations with tokens from other time frames. By processing spatial information separately from temporal information in this way, the model can prevent unnecessary data entanglement that occurs during the analysis of complex sequence data and clearly and efficiently interpret the spatial context within a single time point.

[0399] The time token generation module (13) can separate and encode temporal flow information of an input scene data sequence based on visual features obtained from the vision feature extraction module (10). The time token generation module (13) can generate temporal tokens capable of capturing movement or long-term dependencies over time at the same spatial location.

[0400] Referring to FIG. 10, the time token generation module (13) can generate time tokens by applying temporal pooling to the extracted features of the extracted vision feature map, thereby compressing and including information about the temporal relationship between each frame and changes in the object in the time tokens.

[0401] For example, the time token generation module (13) can generate time tokens by integrating three types of token embeddings based on token embeddings (z) from a projector.

[0402] First, the time token generation module (13) can maintain the macroscopic flow of the entire video through the CLS token, which is global summary information of each frame from the global token generation module (11).

[0403] Next, the time token generation module (13) can generate tokens summarizing the visual state of each individual time point by applying spatial average pooling to the feature map token embedding (z) of each frame.

[0404] Finally, the time token generation module (13) can generate frame-difference tokens by calculating the difference in feature values ​​between adjacent frames and then performing spatial average pooling again, thereby enabling encoding that emphasizes only the actual motion components of the object that contrast with the static background information.

[0405] The time token generation module (13) can finally generate a time token by concatenating these three types of tokens sequentially.

[0406] The time tokens generated in this way can help the vision language model understand the overall flow and detailed movements of the input scene data as a single long context.

[0407] For example, the time token generation module (13) according to one embodiment may be used to analyze not only short movements of a general video but also long-term change patterns between time-series images taken at significant time intervals.

[0408] When the generated time tokens are processed through a Disentangled Spatial & Temporal Attention Mask, they can be controlled to perform attention operations only with same-spatial location tokens in different time frames, rather than with other spatial tokens in the same time frame. By processing temporal information separately from spatial information in this way, the model can effectively grasp the progression of the action itself without interference from background information. Consequently, this resolves the problem of complex entanglement of visual features, thereby significantly mitigating the action-scene hallucination phenomenon that frequently occurs in vision language models.

[0409] The text token generation module (14) can generate text tokens by processing and encoding language features input to the simulation system (5100), namely user prompts or explanatory text. The text token generation module (14) can prepare the simulation system (5100) to fuse visual information and language information in a multi-modal environment to perform inference.

[0410] Referring to FIG. 10, for example, a text token generation module (14) can apply a tokenizer to input text to separate the sentence into individual words or subwords and convert them into text embeddings. Through this embedding process, a high-dimensional vector containing semantic information and grammatical context of the words can be generated. The generated text tokens are then input into a large language model and can be used in attention operations along with spatial and temporal tokens extracted from scene data. Through this operation, the model can accurately map which spatiotemporal elements of the image the user's question (language) is related to and generate complex inference results.

[0411] In addition, the text token generation module (14) according to one embodiment can receive and process various forms of text-based domain-specific data (e.g., operation logs, scenario descriptions, rule / constraint information, etc.) linked to image information to be analyzed, beyond simple natural language queries.

[0412] The input sequence configuration module (15) can combine and align various types of tokens extracted and generated within the simulation system (5100) into a single integrated sequence that fits the input format of the large language model. The input sequence configuration module (15) receives global tokens, spatial tokens, temporal tokens, and text tokens, determines the order among the tokens, and arranges them into a single long vector sequence. In this way, the input sequence configuration module (15) clearly configures the input sequence and transmits it to the large language model processing module (16) to enable efficient learning and inference.

[0413] The large language model processing module (16) may be configured to receive an input sequence during the operation phase and output an analysis result for the scene data. Additionally, if necessary, at least some parameters of the large language model processing module (16) may be further learned by the learning method illustrated in FIGS. 15 to 17.

[0414] For example, the input sequence configuration module (15) can form a final input sequence by placing a global token at the very front and then sequentially concatenating L text tokens, N space tokens, and M time tokens thereon.

[0415] The integrated sequence is passed to each block within the large language model and processed through DST-Attention and a specific positional encoding (e.g., Harmonic-RoPE) to ensure that spatiotemporal information does not become entangled while maintaining optimal correlations that match the text query. Through this, the model is not biased toward specific scenes in the video but analyzes the overall context and detailed actions in a balanced manner to derive accurate inference results.

[0416] However, the above global token may be included in the input sequence but may be excluded from the computation of position encoding and / or Disentangled Spatio-Temporal (DST) attention. That is, DST attention and harmonic rotation position embedding may be performed on spatial tokens, temporal tokens, and text tokens excluding the global token. Meanwhile, the global token is combined with the output vector generated from the attention computation result and input to a subsequent multi-layer perceptron (MLP), thereby contributing to the generation of an output that reflects the global context of the entire scene.

[0417] The large language model processing module (16) may include a pre-trained artificial intelligence model and, if necessary, may be further trained by the learning method shown in FIG. 15.

[0418] The large language model processing module (16) can sequentially perform structured position encoding and DST (Disentangled Spatio-Temporal) attention operations on global tokens, spatial tokens, time tokens, and text tokens received by the large language model from the input sequence configuration module (15).

[0419] The integrated input sequence transmitted to the large language model processing module (16) may first undergo a step of encoding position information. Through this process, structured spatiotemporal position information can be reflected for tokens based on Harmonic-Rotary Position Embedding.

[0420] For example, referring to Fig. 11, a Balance Position Id based on local order and a Distinct Position Id based on global order may be assigned to each token.

[0421] Here, local order refers to the relative order within the token type group to which the token belongs (time token group, space token group, etc.), and global order may refer to the absolute order of the token relative to the entire input sequence.

[0422] In this case, the last position of the local order within the time token group and the last position of the local order within the spatial token group can be set to be the same. Accordingly, the spatial token and the time token share the same location identifier range, which becomes a key mechanism that enables the model to recognize that the time token and the spatial token are substantially equidistant from the text token.

[0423] For example, referring to the following formula (1), when there are M time tokens, N space tokens, and L text tokens, when 1 to N balanced position identifiers are sequentially assigned to each space token, N+1 to N+L balanced position identifiers are sequentially assigned to each text token, and N is added to all of 1 to L, and M balanced position identifiers are sequentially assigned with N as the last number and N-(M-1) as the first number.

[0424] Equation (1):

[0425] For example, referring to Fig. 11, the first token generated within the time token group has a balanced position identifier of '1' according to local order, and the fourth spatial token generated throughout the input sequence may have a unique position identifier of '4' according to global order.

[0426] In this manner, the balanced position identifier and unique position identifier assigned to each token can serve as criteria for determining the rotation angle to effectively inject spatiotemporal position information into the token's embedding vector. The position information obtained from these two identifiers is combined to calculate the final rotation angle for each token, and this angle can be used to perform rotation operations based on Harmonic-Rotary Position Embedding. In this process, the balanced identifier provides a rotation component that contributes to relative and local relationships within the token group, while the unique identifier provides a rotation component that reflects global position information based on the entire sequence.

[0427] For example, when the large language model processing module (16) divides each token having D dimensions by component and applies a rotation operation, it may apply a balanced position identifier to even-indexed pairs and a unique position identifier to odd-indexed pairs.

[0428] For example, referring to the following equations (2) and (3), for each component of a token embedding, a rotation matrix operation can be performed based on the final rotation angle (pθk) calculated by multiplying a predetermined frequency component θk by each position identifier (p). In this case, an angle obtained by multiplying the balance identifier by θk can be applied to the even components of each token embedding, and an angle obtained by multiplying the unique identifier by θk can be applied to the odd components.

[0429] Equation (2):

[0430] Equation (3): (where D is the dimension of each token)

[0431] The actual rotation operation can be performed by applying a single rotation operation to the embedding vector of the corresponding token using a rotation angle determined by integrating information obtained from two identifiers. Through this operation, the token embedding can simultaneously include two different types of location information (local order and global order). This enables large language models to accurately recognize relationships between tokens that are complexly intertwined in space and time, and consequently, can mitigate hallucinations that occur during scene data analysis and dramatically improve the consistency and precision of inference.

[0432] Tokens encoded with location information can be processed through Disentangled Spatial & Temporal (DST) attention operations, thereby preventing unnecessary entanglement between spatiotemporal information and maximizing analysis efficiency.

[0433] DST attention operations can be performed based on a Disentangled Spatial & Temporal Attention Mask (DST attention mask).

[0434] The above DST attention mask can be designed to restrict direct attention between spatial tokens and temporal tokens. For example, the mask can be set so that mutual attention weights are not calculated by assigning a value of -∞ to combinations of spatial tokens and temporal tokens. The -∞ value can be added to the attention score of the step prior to Softmax so that the corresponding term becomes 0 in the Softmax result.

[0435] Meanwhile, text tokens may be allowed to reference space tokens and time tokens, respectively, and space tokens may be configured to reference space tokens and text tokens, and time tokens may be configured to reference time tokens and text tokens, respectively.

[0436] Furthermore, causal attention, which refers only to past time point tokens, can be applied to time tokens, while bidirectional attention, which refers to the entire context, can be allowed for spatial and text tokens. Attention operations can be performed by calculating an attention score through the inner product between the query and key of tokens that reflect location information. This score is applied to a Softmax function and converted into weights in the form of a probability distribution. Additionally, tokens included in the input sequence can be transformed into value (V) vectors through a separate linear transformation module (Value Projector). In this case, ground truth (GT) data can be provided in the form of correct labels or correct description text for the input scene data, and the final output of the large language model can be compared with this ground truth data to calculate the loss function.

[0437] The attention weights passed through the Softmax function are multiplied by the value (V) of the corresponding token and weightedly summed to generate the final output vector (Y), which can be input into a Multi-Layer Perceptron (MLP). In this case, the vector of global tokens can also be input into the MLP along with the final output vector (Y) for processing.

[0438] The MLP can refine the input feature vector through a non-linear transformation. The above-mentioned DST attention operation can be performed according to the following equations (4) to (7).

[0439] Equation (4): , (where M=MDST)

[0440] Equation (5):

[0441] Equation (6):

[0442] Equation (7):

[0443] During the backpropagation process, parameters can be adjusted to minimize the calculated loss. At this time, the parameters of pre-trained feature extractors, such as vision encoders (CLIP) and text embedding models, are frozen, while the parameters of modules responsible for spatiotemporal relationship inference—such as various projectors that generate spatial and temporal tokens from vision feature maps, DST attention blocks, and MLPs—can be updated.

[0444] Through these sophisticated DST attention operations and structured positional encoding processes, the large language model can clearly learn complex spatial relationships within images and associations between visual and language. This enables the model to infer precisely and error-free even for complex linguistic instructions that combine visual elements and spatial locations, and consequently serves as a foundation for providing high-reliability scene data analysis results that significantly reduce errors (hallucinations) in analytical inference.

[0445] The above equations (1) to (7) are merely examples, and position encoding and attention operations can be implemented in various modified forms.

[0446] The spatial-language mapping module (17) may use a Spatial-Language Mapping Neural Module (SLMM) that performs operations to fuse spatial meanings by integrating and aligning visual spatial information extracted from scene data and the user's linguistic instructions.

[0447] The space-language mapping module (17) can generate contextual space tokens that enable precise inference without error even for complex language instructions that combine visual changes and spatial locations, by going beyond simply connecting features of text and images, through the interaction of a space description token corresponding to information specifying space (e.g., bounding box information, coordinate information, etc.) and a language token containing language information describing a predetermined space.

[0448] The spatial-language mapping module (17) may include various internal modules for such complex mapping operations. For example, referring to FIG. 12, it may be configured around a spatial-language mapping operation module (22) that fuses the information, and includes a spatial information encoding module (20) and a language information encoding module (21) that process visual features and language features, respectively. Additionally, the spatial-language mapping module (17) may include a spatial information transformation module (23) and a language information transformation module (24) that improve the features of spatial information and language information so that the model can interpret them easily, thereby increasing the accuracy of the mapping operation.

[0449] Furthermore, the spatial-language mapping module (17) may include modules to support complex context and knowledge-based reasoning. For example, the spatial-language mapping module (17) may include a prompt reconstruction module (25) to increase the depth of reasoning by utilizing external knowledge, such as a spatial knowledge graph database (27) and a domain-specific spatial knowledge database (28), and to resolve ambiguity in the input prompt.

[0450] Finally, the space-language mapping module (17) integrates visual, linguistic, and knowledge information through the multimodal context encoding module (26) to complete contextual space tokens, thereby laying the foundation for the simulation system (5100) to finally output an accurate response.

[0451] The process of generating contextual space tokens (CSI) within the space-language mapping module (17) can be achieved through the organic interaction of the space information encoding module (20), the language information encoding module (21), and the space-language mapping operation module (22).

[0452] The space-language mapping module (17) can fuse the meaning of location information (spatial information), such as bounding boxes / coordinates specifying the space, and text (language information) describing the space through cross-attention.

[0453] Referring to FIG. 13, for example, a spatial information encoding module (20) can generate spatial description tokens (SI) based on spatial information provided by a user. Here, spatial information may refer to structured numerical information that specifies the location of an object included in scene data, such as bounding box coordinates, the center point of an object, and the size of an area. The spatial description tokens (SI) may correspond to a set of vectors that encode location information specifying such space.

[0454] Additionally, the language information encoding module (21) can process the user's prompt text to generate a language token (LI). The language token (LI) may correspond to a set of vectors that encode the meaning and context of the text describing the space.

[0455] The spatial-language mapping operation module (22) can perform a cross-attention operation that analyzes the interaction between spatial descriptive tokens (SI) and language tokens (LI) to generate contextual spatial tokens (CSI). This process fuses the meanings of spatial descriptive tokens (SI) corresponding to location information such as bounding boxes / coordinates that specify the space and language tokens (LI) corresponding to information regarding text indications, and provides this to the learning process of a large language model, thereby enabling the large language model to accurately identify the spatial meaning indicated by the text within the visual features.

[0456] In the cross-attention operation, spatial descriptive tokens (SIs) act as queries, while language tokens (LIs) can serve as keys and values. This corresponds to a process where spatial descriptive tokens (SIs) reference language tokens (LIs) to re-evaluate their importance and meaning. This operation is similar to the process of finding the answer to the question, "Which part of the image does the text describe?", and a high weight (attention score) can be assigned to areas where the meaning of the text matches spatial features.

[0457] Contextual Spatial Tokens (CSIs) generated as a result of the cross-attention operation correspond to a new set of tokens enhanced by the fusion of spatial information and the context of linguistic references. CSIs can contain vectors in which the importance and meaning of visual elements are reinterpreted according to specific text references. For example, through the text "red object in the top-left corner," CSIs can be used to improve inference accuracy by maximizing the importance of spatial tokens related to location information regarding the top-left corner area over all other spatial features.

[0458] The spatial information transformation module (23) is responsible for directly transforming the spatial information of an object appearing in an image, which can contribute to securing the robustness of the model. The spatial information transformation module (23) can generate transformed spatial information that can be recognized at various viewpoints or under various conditions by applying transformations such as rotation, enlargement (reduction), and movement based on input spatial information (bounding box coordinates, center point, etc.).

[0459] For example, by generating new spatial information by rotating or enlarging the coordinates of an object in an image, the model can be supported in learning to maintain consistency even in the deformed form of the object when it is subsequently passed to the spatial-language mapping operation module (22). The spatial information transformation module (23) can consequently contribute to increasing the accuracy of the mapping operation by improving spatial features so that the model can interpret them easily.

[0460] The language information transformation module (24) can perform the role of adjusting or refining the features of language tokens according to the transformation applied by the spatial information transformation module (23). This can contribute to preventing inference errors or hallucinations caused by image transformation and increasing the accuracy and consistency of spatial information processing in the large language model by ensuring that the model accurately reflects the impact of spatial information on language expression.

[0461] For example, when the spatial information of an object included in the input scene data undergoes transformations such as rotation, enlargement, or reduction, the language information transformation module (24) can adjust or refine the features of the language tokens to match the transformed spatial conditions. If the coordinates of "the box in the top-left corner" are changed by the transformation module (23) to indicate "the bottom-right corner," the language information transformation module (24) can automatically adjust the meaning of the original language token "top-left corner" to a text feature indicating "the bottom-right corner."

[0462] In this manner, the transformed spatial descriptive tokens (SI) and the adjusted language tokens (LI) can be provided as targets for cross-attention operations of the spatial-language mapping operation module (22). That is, the spatial-language mapping operation module (22) performs cross-attention operations using these transformed tokens as queries and key / value pairs, thereby generating a final contextual spatial token (CSI) with ensured robustness. This process ensures that the large language model accurately fuses linguistic instructions and spatial meanings without error, even with respect to transformed spatial information, and can ultimately be provided to the learning process of the large language model.

[0463] The multimodal context encoding module (26) can independently integrate and process visual information and linguistic information within the spatial-language mapping module (17) to complete an optimized multimodal representation that can be utilized by a large language model. This can complement or process the primary fusion results of the spatial-language mapping operation module (22) in parallel, thereby contributing to complex spatial context inference and improving the accuracy of the information. The multimodal context encoding module (26) can improve the accuracy and consistency of inference by going beyond simply combining features and deeply considering the influence of visual and linguistic information on each other.

[0464] The multimodal context encoding module (26) can receive visual tokens and language tokens separately from the space-language mapping operation module (22) and perform independent multimodal fusion. The multimodal context encoding module (26) can encode complex context information.

[0465] For example, a multimodal context encoding module (26) can generate a multimodal representation represented as a visual-language integrated embedding by integrating an image (visual) embedding and a text (language) embedding. This may include a process of aligning vectors through a linear transformation or an initial fusion operation so that features of different modalities exist in a common semantic space.

[0466] Additionally, the multimodal context encoding module (26) can perform attention operations on the integrated multimodal representation to identify the relationship between specific regions of the image and the text, and reflect the contextual meaning in spatial information processing. Through this process, visual features important to text instructions are emphasized, and the accuracy of inference can be improved.

[0467] Furthermore, the multimodal context encoding module (26) may include a network capable of inferring spatial relationships between objects included in the scene data and reflecting them in the final text representation. This can help interpret complex relationships, such as "A is to the left of B," beyond simple object detection.

[0468] The multimodal context encoding module (26) can add depth to reasoning by integrating external knowledge, such as a spatial knowledge graph database (27) and a domain-specific spatial knowledge database (28). By combining or referencing structured spatial relationships extracted from external knowledge into contextual spatial tokens (CSI), the multimodal context encoding module (26) can help prevent the large language model from making errors in reasoning that requires unstructured spatial relationships or expertise in a specific domain. This integration can contribute to solving problems that limit the use of large language models, particularly in applications where the accuracy of spatial information is critical, such as domain-specific simulation scene analysis, autonomous driving, and robot control.

[0469] The multimodal context encoding module (26) can contribute to increasing the accuracy and consistency of spatial information processing, beyond simply processing text and visual information in parallel in a large language model.

[0470] The spatial knowledge graph database (27) may include a structured external knowledge-based database that is utilized to process unstructured spatial relationships and improve the spatial interpretation capabilities of a large language model. The spatial knowledge graph database (27) may include a knowledge graph constructed by expressing spatial relationships and characteristics extracted from various sources, such as text, images, and sensor data, in the form of nodes and edges.

[0471] The spatial-language mapping module (17) can improve spatial accuracy and consistency in the inference of the large language model by inferring spatial relationships between objects included in the scene data by referring to this database and reflecting this in the language expression. This can help the large language model to understand complex spatial contexts contextually by utilizing structured spatial knowledge, rather than relying simply on visual information.

[0472] A domain-specific spatial knowledge database (28) can be constructed to effectively integrate specialized and highly specific spatial information of a specific application domain (e.g., robots, autonomous driving, etc.) into a large language model. This enables the processing of spatial information that takes into account domain-specific characteristics that are difficult to resolve with only general spatial relationships, and this can contribute to overcoming the spatial interpretation limitations of the large language model.

[0473] By utilizing a domain-specific spatial knowledge database (28), the simulation system (5100) can minimize spatial reasoning errors occurring in the domain and achieve accuracy that meets the requirements of actual application fields.

[0474] The prompt generation module (18) can analyze basic query or strategy condition information received from the user interface unit (5120) and reconstruct and expand it into a form suitable for the large language model processing module (16) to perform spatiotemporal inference on the scene data. For example, the prompt generation module (18) can not only simply transmit the user's query but can also generate an optimized expanded prompt through the following steps.

[0475] First, the prompt generation module (18) can perform prompt engineering. If the query entered by the user is abstract or the goal is unclear, the prompt generation module (18) can add instructions to specify the object, space, time interval, and relationship type that the analysis module (400) needs to check in the scene data.

[0476] For example, when a user asks, “Find the section with a high risk of robot collision,” the prompt generation module (18) can expand this into a prompt that includes analysis guidelines and supporting items, such as “In the time step t to t+k of the scene data, explain step by step the section with a high risk of collision and the cause based on the change in relative distance between robot objects, whether there is an intersection path, and whether a congestion section occurs.”

[0477] Second, the prompt generation module (18) can perform context integration. For example, the prompt generation module (18) can form a long context by combining at least one of the environment layout identifier, strategy model identifier, operational parameters, constraint information, or simulation execution history (state / behavior / reward history) stored in the storage unit (5110) in addition to the user's query. For example, by combining “user query + environment layout conditions + constraints (no-go / maximum speed / task priority) + execution history of a specific strategy model,” a composite prompt in the form of “analyze the cause of bottlenecks in Strategy Model A in the corresponding layout in terms of object occupancy area and path conflict, and suggest control alternatives for improvement under the same conditions” can be constructed.

[0478] Third, the prompt generation module (18) can perform output formatting. This may be a process of standardizing the result structure so that the output of the large language model processing module (16) can be immediately utilized for simulation operations and strategy evaluation. For example, the prompt generation module (18) may include an output format within the prompt such that “the answer should be described in the order of 1) target object and time interval, 2) observed spatial / temporal patterns (distance, occupancy, path intersection, etc.), 3) grounds for determining risk / bottleneck, and 4) recommended control commands or strategy modification points.” Accordingly, the large language model processing module (16) may be induced to generate analysis results in an explainable form.

[0479] The extended prompt generated by the prompt generation module (18) can be passed to the text token generation module (14). The text token generation module (14) tokenizes and embedding the extended prompt to convert it into a text token sequence, which can be combined with spatial tokens and temporal tokens generated from scene data and provided as input to the large language model processing module (16). In this way, the prompt generation module (18) can structure the user's intent into conditions and criteria necessary for scene data analysis, thereby supporting the derivation of high-reliability inference results based on the separation and interpretation of spatiotemporal features.

[0480] The prompt generation module (18) can reconfigure and expand the input prompt of the large language model processing module (16) to enable spatiotemporal inference of scene data based on query information, strategy condition information, or analysis request information received from the user interface unit (5120).

[0481] According to one embodiment of the present invention, the query or analysis request may be generated in different contexts depending on the operation stage of the simulation system (5100). For example, immediately after object placement is set through the first user interface (5121), an analysis request for scene data corresponding to an initial state defined by the environment layout generation unit (5141) may occur. Additionally, during the process of executing a control strategy by the strategy control module (5130), a dynamic analysis request for intermediate state scene data generated by the simulation execution unit (5144) at time steps may occur. Furthermore, after the simulation ends, a request for comparative analysis between multiple strategies may occur based on the execution history or scene sequence data for each strategy recorded in the storage unit (5110). The prompt generation module (18) may generate an extended prompt including the type of scene data to be analyzed, time interval, strategy condition, or constraint information in response to at least one of the above situations, thereby inducing the large language model processing module (16) to interpret the spatial and temporal features of the scene data separately.

[0482] According to one embodiment, for initial scene data generated immediately after object placement, the prompt generation module (18) may generate a prompt instructing the analysis of distance relationships between objects, whether occupied areas overlap, or potential collision possibilities. Accordingly, the analysis module (400) may derive spatial risk factors from scene data corresponding to the initial state S (0) and output the basis for judgment in a structured form.

[0483] Additionally, during the process in which a control strategy is executed by the strategy control module (5130), the prompt generation module (18) can generate an analysis instruction including state transition history, object movement path, or constraint information for a specific time interval. Accordingly, the analysis module (400) can infer bottleneck occurrence sections, path overlap patterns, or causes of efficiency degradation regarding scene data of multiple time steps generated by the simulation execution unit (5144).

[0484] Furthermore, based on the execution history of multiple strategies recorded in the storage unit (5110) after the simulation ends, the prompt generation module (18) can generate a prompt to compare the performance between strategies under the same layout conditions. Accordingly, the analysis module (400) can compare scene sequences by strategy and derive a superior strategy or suggest a direction for modifying the strategy based on differences in collision frequency, movement efficiency, or resource occupancy.

[0485] Scene data analysis using an artificial intelligence model (S100)

[0486] A scene data analysis method (S100) using an artificial intelligence model according to one embodiment can improve inference errors (hallucinations) and efficiency issues associated with existing complex image data analysis systems. The method (S100) goes beyond simply analyzing images and enables intelligent multimodal inference that interprets a user's linguistic instructions by accurately connecting them to the spatiotemporal characteristics of the image. For example, the method (S100) can increase the reliability of the analysis by applying a new encoding technique that helps the artificial intelligence more clearly recognize complex positional relationships or temporal changes within the image.

[0487] The method (S100) can separate and reconstruct spatial information extracted from scene data and user's linguistic information and provide them to an artificial intelligence model. Through this process, the artificial intelligence model can accurately learn information about where, when, and in what relationship a specific object is located within the overall flow of the video. For example, it can respond precisely without error even to complex linguistic instructions that combine time and location, such as "a red object moving in the top-left corner."

[0488] The method (S100) helps an artificial intelligence model deeply understand the context of data, thereby effectively reducing the illusion of inferring information that does not actually exist, and provides high scalability and flexibility that allows it to be stably applied to new types of data or long video sequences without additional learning, thus providing reliable analysis results and excellent utility value in various application fields such as domain-specific simulation scene analysis, autonomous driving, and robot control.

[0489] FIG. 14 is a flowchart of a scene data analysis method (S100) according to one embodiment. FIG. 15 is a flowchart of a learning method for at least one artificial intelligence model used in the method (S100) of FIG. 14. FIG. 16 is a flowchart for explaining steps that may be included in the step (S1059) of encoding location information of the method of FIG. 15. FIG. 17 is a flowchart for explaining steps for generating contextual spatial tokens through vision-language mapping that the method of FIG. 15 may further include.

[0490] Referring to FIG. 14, a method for analyzing scene data using an artificial intelligence model according to one embodiment (S100) may include the steps of receiving scene data and storing it in at least one memory (S101), loading the scene data from the at least one memory (S103), generating at least one analysis result for the scene data using at least one artificial intelligence model that takes the scene data as input by at least one processor (S105), and inputting the analysis result to at least one subsequent processing component (S107).

[0491] In one embodiment, the method (S100) may be performed by a processor included in a control unit (5140) included in a simulation system (5100). However, it is not limited thereto, and at least a portion of the method (S100) may be performed by a control unit (5140) included in a simulation system (5100), and another portion may be performed by one or more processors constituting an analysis module (400).

[0492] For convenience of explanation, the following description describes that a processor included in the control unit (5140) included in the simulation system (5100) performs the method (S100).

[0493] In step (S101), the control unit (5140) may acquire scene data generated by the simulation execution unit (5144) and store it in the storage unit (5110). The scene data may include a static scene frame corresponding to a single time step or a scene sequence spanning multiple time steps. This process is responsible for the initial acquisition of data for subsequent analysis and stable storage within the system, and the received scene data may be converted into visual features by the vision feature extraction module (10) and then provided to the large language model processing module (16).

[0494] The received scene data can be stored in memory (RAM, ROM, etc.) or a storage device (SSD, disk, etc.) to ensure data integrity, and can be prepared so that a processor included in the control unit (5140) can efficiently load the scene data in a subsequent step (S103). This step (S101) signifies the starting point of a preprocessing process for accurately extracting and analyzing spatiotemporal features of an image, and corresponds to an initial stage in which the data is converted to a state accessible to the computing device of the simulation system (5100).

[0495] In step (S103), the processor included in the control unit (5140) can load scene data stored in at least one memory for subsequent processing. Step (S103) may include loading data from a storage location to the workspace (e.g., cache or main memory) of the processor included in the control unit (5140) so that the processor included in the control unit (5140) can directly access and process the scene data.

[0496] In step (S105), the processor included in the control unit (5140) can perform a series of core processes to generate analysis results for scene data using an artificial intelligence model. Step (S105) can extract spatial and temporal visual features from the scene data, perform integrated analysis with the linguistic features of the text data through a Disentangled Spatio-Temporal (DST) attention operation, and finally generate an output analysis result.

[0497] In this case, the analysis result may include at least one of a natural language description including spatial-temporal relationships and context regarding the scene data and information regarding object coordinates.

[0498] FIG. 15 illustrates a separate training method for pre-training an artificial intelligence model used in step (S105).

[0499] Referring to FIG. 15, for example, at least one artificial intelligence model used in step (S105) comprises: a step of extracting a vision feature map from training scene data for pre-training (S1051); a step of generating a global token, a spatial token, and a temporal token for the training scene data using the vision feature map (S1053); a step of generating a text token from a text input (S1055); a step of constructing an input sequence including the global token, the spatial token, the temporal token, and the text token (S1057); a step of encoding position information by applying a harmonic rotation position embedding to each token of the input sequence (S1059); a step of performing an attention operation with the input sequence encoded with position information as input, applying the separation mask to control the interaction between the spatial token and the non-spatial token (S1061); a step of generating an output result by performing a non-linear transformation on the result of the attention operation (S1063); and updating the parameters of the at least one artificial intelligence model so as to minimize the loss based on the output result and the ground truth data. It can be pre-trained by a learning method including step (S1065).

[0500] In step (S1051), the processor included in the control unit (5140) can extract a Vision Feature Map from the training scene data. This process may include processing the input scene data using a Vision Encoder, compressing the image content (objects, shapes, scenes, etc.), and generating a feature map in the form of a high-dimensional vector containing meaning. This Vision Feature Map can be used as basic data for generating global tokens, spatial tokens, and temporal tokens in the next step (S1053).

[0501] In step (S1053), the processor included in the control unit (5140) can generate global tokens, spatial tokens, and temporal tokens based on the vision feature map extracted in step (S1051). This process may mean separating and encoding the features of the vision modality into independent token forms.

[0502] For example, global tokens contain macroscopic context or scene information for the entire image, while spatial tokens represent static spatial features such as the location, boundaries, and shapes of objects included in the scene data. Additionally, time tokens can encode dynamic temporal features, such as object movement or state changes within a video sequence.

[0503] By preparing each feature in the form of a separated token in this way, the interaction between spatial and temporal features is precisely controlled through the DST attention operation described later, thereby enabling the large language model to effectively infer complex spatiotemporal contexts.

[0504] In step (S1055), the processor included in the control unit (5140) can generate text tokens from the text input. This process may include separating the prompt or query text entered by the user into individual words or semantic units using a tokenizer and converting them into text embeddings. The generated text tokens are prepared in the form of high-dimensional vectors containing linguistic meaning and context, and can be used as input data for a DST attention operation along with visual tokens (global, spatial, and temporal tokens) in a subsequent step.

[0505] In step (S1057), the processor included in the control unit (5140) can combine all tokens generated in the previous steps (S1053 and S1055), respectively, to form a final input sequence for a Disentangled Spatio-Temporal (DST) attention operation.

[0506] This input sequence may consist of a Global token, a Spatial token, a Temporal token, and a Language token. A processor included in the control unit (5140) can arrange these separate tokens into a single integrated sequence.

[0507] Through step (S1057), data can be arranged in the most efficient form so that the DST operation can simultaneously reference spatial, temporal, and linguistic features and analyze their interaction. This lays the foundation for precisely processing complex multimodal information in separate dimensions.

[0508] In step (S1059), the processor included in the control unit (5140) can encode position information by applying a Harmonic-Rotary Position Embedding (RoPE) to each token of the input sequence. This process clearly reflects the relative positional relationships between tokens in a vector space, which helps the large language model accurately grasp the spatiotemporal context and reduce hallucinations.

[0509] Referring to FIG. 16, for example, step (S1059) may include: a step (S191) of assigning a balanced position identifier based on a local order and a unique position identifier based on a global order to each token in the input sequence; a step (S193) of determining a rotation angle for each token based on the balanced position identifier and the unique position identifier; and a step (S195) of encoding position information for each token by performing a rotation operation on each token in the input sequence using the rotation angle determined for each token.

[0510] In step (S191), the processor included in the control unit (5140) may assign a balanced location identifier based on local order and a unique location identifier based on global order to each token in the input sequence. The balanced location identifier indicates the relative order within the token type group (space, time, text) to which the token belongs and contributes to understanding the local context. The unique location identifier indicates the absolute order of the corresponding token relative to the entire input sequence and contributes to understanding the macroscopic context and long-term location.

[0511] These two identifiers define the spatiotemporal location of the token in a dual manner, and can subsequently be used as a criterion for determining the rotation angle.

[0512] In step (S193), the processor included in the control unit (5140) can determine the rotation angle for each token based on the balance position identifier and unique position identifier assigned to each token in step (S191). In this process, the information of the two identifiers can be integrated to calculate the final angle at which the token must rotate in the embedding space. This angle contains relative distance information regarding how far the token is from other tokens and is uniquely determined for each token, which can be used to increase the precision of spatiotemporal relationship inference.

[0513] In step (S195), the processor included in the control unit (5140) can finally encode position information by performing a rotary operation on each token in the input sequence using the rotation angle determined per token in step (S193). This rotary operation modifies the embedding vector of the token to effectively inject position information, thereby enabling accurate spatiotemporal relationship inference during subsequent attention operations and improving the accuracy of scene data analysis.

[0514] In step (S1061), the processor included in the control unit (5140) can perform a Disentangled Spatio-Temporal (DST) attention operation using an input sequence in which location information is encoded as input. This operation can apply a disentangling mask to control the interaction between spatial tokens and temporal tokens. The disentangling mask allows causal attention for temporal tokens and bidirectional attention for spatial tokens and text tokens, and can prevent unnecessary entanglement between spatial information and temporal information.

[0515] Through this, the model can efficiently process spatiotemporal information by separating it, mitigate long-term dependency issues, and improve the accuracy and consistency of the analysis.

[0516] In step (S1063), the processor included in the control unit (5140) can generate a final output result by performing a non-linear transformation on the result of the attention operation from the previous step (S1061). This non-linear transformation can be performed, for example, through a Multi-Layer Perceptron (MLP) module, and the MLP module can receive the feature vector enhanced through the attention operation, refine the features into a form suitable for the final inference purpose (e.g., classification, response generation, object detection), and transform them into a high dimension.

[0517] The output result generated in this step (S1063) may include information that the simulation system (5100) ultimately provides to the user (e.g., text answer, location / description of an object included in the scene data).

[0518] In step (S1065), the processor included in the control unit (5140) can update the parameters of the artificial intelligence model to minimize loss based on the generated output result and ground truth data.

[0519] This process is carried out through the backpropagation algorithm, and the model's internal weights (parameters) can be adjusted to reduce the error (loss) between the output results and the ground truth data. Parameter updates can be performed on modules responsible for spatiotemporal relationship inference, such as DST attention blocks, MLPs, and projectors, for example; through this, the model's performance can be progressively improved and its ability to mitigate hallucinations can be enhanced.

[0520] In step (S107), the processor included in the control unit (5140) may input the analysis result generated using the artificial intelligence model in step (S105) into at least one subsequent processing component. This analysis result is final output information obtained through inference of a large language model (LLM) and may include text descriptions of complex images, location and attribute information of objects, or inferred spatial relationships.

[0521] Step (S107) may correspond to a step of stably and efficiently transmitting the results to the next step of the simulation system (5100) so that the results can be utilized in actual application fields after the analysis of the artificial intelligence model is completed.

[0522] In this case, at least one subsequent processing component can use the analysis results to provide services related to autonomous driving, robot control, and domain-specific simulation scene analysis. The analysis results can be provided to the strategy control module (5130) and used for control strategy correction, state prediction, or subsequent simulation execution.

[0523] For example, spatial relationship inference information derived from analysis results can be utilized for path planning in autonomous driving systems or as direct data for workspace recognition and command execution in robot control systems.

[0524] Here, at least one subsequent processing component may include various types of systems or modules that receive an analysis result (e.g., spatiotemporal relationship inference result) generated through an artificial intelligence model in step (S105) and provide a final service or perform additional system tasks based thereon. The subsequent processing component may utilize the analysis result in accordance with the requirements of the actual application domain and may be implemented as a module inside the analysis module (400) or as an external application system depending on the system architecture.

[0525] If the follow-up processing component is implemented as a module within the analysis module (400), it can perform the role of delivering the analysis results to the end user or integrating them with system data. For example, it may include a module that manifests the analysis results through at least one user interface (UI) in the form of a text report, a warning message, or a visualized form. Additionally, it includes a database storage system for preserving the analyzed spatial relationships and inference results for a long period, and a communication / transmission module for securely transmitting data to other internal modules or external systems. These internal modules can contribute to converting the LLM analysis results into a user-friendly form and ensuring data integrity within the system.

[0526] The follow-up processing component may be an external application system that provides services for specific purposes by utilizing the analysis results. This can be applied in domains where the accuracy of spatial information is critical, such as autonomous driving, robot control, domain-specific simulation scene analysis, architectural design, and construction management. For example, autonomous driving systems utilize precise spatiotemporal relationship information derived from analysis results to improve the accuracy of collision risk assessment and path planning, while robot control systems execute commands without errors based on information regarding the location and movement of objects within the workspace. In this way, external application systems can use highly refined multimodal analysis results as the basis for final decisions and actions.

[0527] Additionally, at least one subsequent processing component can manifest the analysis results through at least one user interface (UI). This means displaying the analysis results on a screen so that the user can visually check and interact with them, or outputting them in the form of specific warning messages or reports. Through this, the final result of the simulation system (5100) can be delivered to the user, thereby maximizing its usefulness.

[0528] In this case, for example, the process of at least one subsequent processing component expressing the analysis results through at least one user interface may be based on a weight matrix calculated through a predetermined operation using a separation mask for spatial tokens and non-spatial tokens.

[0529] For example, at least one follow-up processing component can provide an image through a user interface with different visual effects applied to multiple regions of the image related to the analysis result.

[0530] In other words, by analyzing the interaction strength (weight matrix) between tokens during DST attention operations and applying specific emphasis effects (e.g., highlights, color changes, focusing, etc.) only to image regions highly relevant to the user's query (e.g., objects to be analyzed), the complex inference results of a large language model can be conveyed to the user intuitively and clearly.

[0531] Meanwhile, the method (S100) may further include steps for generating contextual space tokens through vision-language mapping.

[0532] For example, referring to FIG. 17, the method may include the step of generating a spatial description token corresponding to information specifying a predetermined space for the training scene data (S111), the step of generating a language token including language information describing the predetermined space (S113), and the step of generating a contextual spatial token by performing an operation to fuse spatial meaning through the interaction of the spatial description token and the language token (S115).

[0533] In step (S111), the processor included in the control unit (5140) can generate a spatial description token (SI) corresponding to information specifying a predetermined space for the training scene data. Here, the information specifying the space may include structured numerical information specifying the location of an object included in the scene data, such as bounding box coordinates, the center point of an object, and the size of an area. The spatial description token (SI) is a set of vectors that encodes this location information and can perform the role of a query in a spatial-language mapping operation.

[0534] In step (S113), the processor included in the control unit (5140) can generate a language token (LI) containing language information describing the predetermined space. This process may include separating words or phrases (e.g., "top left corner," "red object") containing spatial meaning from a user's prompt or text instruction through a tokenizer and converting them into text embeddings. The language token (LI) can serve as a key and value in a space-language mapping operation to provide linguistic context to the space description token (SI).

[0535] In step (S115), the processor included in the control unit (5140) can generate a contextual spatial token (CSI) by performing an operation that fuses spatial meaning through the interaction of a spatial descriptive token (SI) and a language token (LI). This operation can be performed in such a way that the spatial descriptive token (SI) updates itself by referencing the language token (LI) through cross-attention. The contextual spatial token (CSI) contains spatial information whose importance is reinterpreted by linguistic instructions, which can help a large language model accurately identify the spatial meaning indicated by the text within the visual features.

[0536] FIG. 18 is a conceptual diagram illustrating a structure for applying an analysis module (400) to initial scene data S(0) according to one embodiment.

[0537] According to the present embodiment, placement information, initial coordinates, direction information, attribute information, or operation condition information of a plurality of objects may be input through the user interface unit (5121) as described in FIG. 4. The objects may include at least one of a robot, an autonomous vehicle, a drone, equipment, a work object, or a spatial constraint structure.

[0538] The above input data can be transmitted to a control unit (5140), and the control unit (5140) can generate scene data corresponding to an initial environment state S(0) based on the object placement information. The scene data may be a static scene frame corresponding to a single time step, and may be visual representation data including at least one of object location information, occupied area information, distance relationship between objects, spatial placement state, or planned path information.

[0539] The generated initial scene data S(0) can be provided as input to the analysis module (400). The analysis module (400) may be implemented by a computing device including one or more processors, and according to one embodiment, may be included in a simulation system (5100) and configured to operate under the control of a control unit (5140). According to another embodiment, the analysis module (400) may be executed on an external server system connected via a network.

[0540] The analysis module (400) can extract visual feature data from the scene data through the vision feature extraction module (10). The extracted features can be converted into spatial token data by the spatial token generation module (12) and into time token data by the time token generation module (13). Even if the initial scene data is static data corresponding to a single time step, the time token generation module (13) can generate tokens in a form that reflects potential time axis information considering future state transitions.

[0541] Additionally, constraint information, strategy condition information, or analysis request information received through the user interface unit (5120) can be converted into text token data by the text token generation module (14).

[0542] The spatial token data, temporal token data, and text token data can be provided as inputs to a Disentangled Spatio-Temporal (DST) attention operation. The DST attention may include a separation mask that restricts direct attention between spatial tokens and temporal tokens, thereby preventing unnecessary entanglement between spatial information and temporal information, and allowing spatial relationships and potential state transitions to be structurally interpreted.

[0543] The spatial-language mapping module (17) can dynamically adjust the importance of analysis for a specific area or specific object through the interaction of spatial tokens and text tokens. Accordingly, spatial features related to the user request can be emphasized and contextual tokens can be generated.

[0544] Analysis result data may be generated from the above operation result, and the analysis result data may include at least one of distance relationships between objects, possibility of overlapping occupied areas, possibility of intersection between paths, or potential collision areas.

[0545] The above analysis result data is provided as input to the strategy control module (5130) and can be used to perform at least one of readjusting the initial interval between objects, recalculating the planned route, setting traffic restrictions for specific areas, adjusting control parameters, or changing execution conditions. Accordingly, potential risk factors can be eliminated in advance before the simulation execution unit (5144) initiates strategy execution.

[0546] Additionally, the analysis results are recorded in the storage unit (5110) and can be utilized in the iterative process of design-verification-redesign. For example, if the initial risk level exceeds a preset threshold, the control unit (5140) can modify the object placement or environment layout and generate new initial scene data S (0) corresponding to the modified placement and re-enter it into the analysis module (400). Accordingly, an iterative verification loop can be formed in the pre-execution stage.

[0547] Furthermore, the above initial analysis results can be used as reference data in the analysis during execution phase (Fig. 19) and the strategy comparison analysis phase (Fig. 20). For example, the prediction accuracy can be evaluated by comparing the actual bottleneck section that occurred during execution with the initial predicted risk area, and the correlation between the initial risk and execution performance can be analyzed when comparing multiple strategies.

[0548] In addition, the above analysis results can be utilized as training data in subsequent training stages. For example, inference accuracy can be improved by adjusting the parameters of the artificial intelligence model to reflect the difference between the initial risk prediction results and the actual execution results.

[0549] Accordingly, the initial scene data analysis according to Fig. 18 is not limited to a simple preliminary verification stage, but can function as a starting point for a full-cycle analysis flow leading to strategy correction, execution control, strategy comparison, and model improvement.

[0550] FIG. 19 is a conceptual diagram illustrating a structure for applying an analysis module (400) to scene sequence data generated during a strategy execution process according to one embodiment.

[0551] According to the present embodiment, when a control strategy is executed by the strategy control module (5130), the simulation execution unit (5144) can calculate an environment state S(t) according to a time step t and generate scene data S(t) corresponding thereto. The scene data may include a scene sequence that is continuously generated over multiple time steps in the form of S(t), S(t+1), …, S(t+n).

[0552] The above scene sequence may be visual representation data including at least one of the position coordinates of an object, an occupied area, a distance relationship between objects, a direction of movement, a velocity vector, whether a path intersects, or collision event information.

[0553] The above scene sequence data may be provided as input to the analysis module (400). The analysis module (400) may be implemented by a computing device including one or more processors, and according to one embodiment, may be included in a simulation system (5100) and configured to operate in real-time or near-real-time under the control of a control unit (5140).

[0554] The analysis module (400) can extract visual features from the scene frames of each time step through the vision feature extraction module (10). The extracted features can be expressed in the form of high-dimensional feature vectors and can subsequently be used as input for token-based encoding.

[0555] The spatial token generation module (12) can encode the above visual features into spatial tokens that reflect spatial relationships within the same time step. The spatial tokens may include information related to relative positions between objects, overlapping occupied areas, proximity status, or arrangement structures.

[0556] The time token generation module (13) can generate time tokens that reflect state changes and object movement patterns over multiple time steps. The time tokens can encode feature changes between frames, object movement trajectories, rates of change in distance, or the timing of event occurrences. Accordingly, the analysis module (400) can structurally interpret dynamic state transition patterns beyond single-point-time analysis.

[0557] The text token generation module (14) can generate text tokens by tokenizing strategy condition information, constraint information, or analysis request information received through the user interface unit (5120).

[0558] Subsequently, the spatial token, temporal token, and text token may be provided to a Disentangled Spatio-Temporal (DST) attention operation. The DST attention may include a separation mask that restricts direct attention between the spatial token and the temporal token. Accordingly, the operation can be controlled so that spatial information focuses on interpreting relationships within the same time point, and temporal information focuses on tracking changes between time points. Unidirectional attention to past time points may be applied to the temporal token, while bidirectional attention may be allowed for the spatial token and the text token.

[0559] Such a spatiotemporal separation interpretation structure is distinguished from simple coordinate comparison or critical distance determination methods. The present invention is differentiated in that it converts a scene sequence into a token-based high-dimensional representation and structurally infers spatiotemporal relationships through a separated attention structure.

[0560] The spatial-language mapping module (17) can dynamically adjust the importance of analysis for a specific object, a specific region, or a specific time interval through the interaction between text tokens and spatial tokens. For example, if a request to “identify a section with a high risk of collision” is input, the token weights of the time interval and spatial region associated with the request may be strengthened.

[0561] Based on the above calculation result, the analysis module (400) can calculate at least one of the bottleneck section occurring during the strategy execution process, the duration of the proximity state between objects, the path overlap pattern, the collision risk time period, or the efficiency degradation section.

[0562] The calculated analysis results are provided to the strategy control module (5130) and can be used to correct the strategy during execution. For example, if a high collision risk is calculated for a specific time interval, the strategy control module can adjust the speed of the object or recalculate the path. Additionally, if bottlenecks occur repeatedly, the priority policy or resource placement strategy can be modified.

[0563] When a control strategy is executed by the above-mentioned strategy control module (5130), the simulation execution unit (5144) can generate scene data S(t) using time step-by-time environment state data as input and continuously output it to form a scene sequence.

[0564] The generated scene sequence data can be provided as input data to the analysis module (400), and the analysis module (400) can load it into internal memory and then convert it into visual feature data through the vision feature extraction module (10). The visual feature data can be converted into spatial token data and time token data, respectively, through the spatial token generation module (12) and the time token generation module (13).

[0565] Additionally, strategy conditions or analysis request information received from the user interface unit (5120) can be converted into text token data through the text token generation module (14).

[0566] The above spatial token data, temporal token data, and text token data are provided as inputs to a DST attention operation, and analysis result data can be generated from the result of a spatiotemporal attention operation with a separation mask applied.

[0567] The above analysis result data can be provided as an input to the strategy control module (5130) and used as an input variable for strategy correction calculation, and at the same time, it can be recorded in the storage unit (5110) and used as input data for subsequent analysis or strategy comparison.

[0568] Accordingly, scene sequence data is converted into structured analysis result data via the analysis module (400), and a closed-loop data flow can be formed in which the result is fed back to the strategy control module (5130).

[0569] In addition, the above analysis results are recorded in the storage unit (5110) and can be used for performance comparison by strategy, generation of subsequent training data, or post-analysis.

[0570] Accordingly, the present invention provides a feedback loop structure that, rather than a structure that simply evaluates results after strategy execution, interprets scene sequences in real time during strategy execution and feeds the results back to strategy control.

[0571] With such a configuration, the present invention enables the early identification of potential risk factors during execution and the dynamic modification of strategies, thereby providing technical effects such as reduced risk of conflict, improved resource utilization efficiency, and increased system stability.

[0572] FIG. 19 is a conceptual diagram illustrating a structure for applying an analysis module (400) to scene sequence data generated during a strategy execution process according to one embodiment.

[0573] FIG. 20 is a conceptual diagram illustrating a structure for comparing and analyzing multiple strategies based on strategy execution history and environmental condition data according to one embodiment.

[0574] According to the present embodiment, the storage unit (5110) can store strategy execution history data and environmental condition data. The strategy execution history data may include at least one of a strategy identifier, a strategy model ID, a reward function type, a control parameter value, the number of collisions, an average travel distance, an average waiting time, total throughput, or a bottleneck occurrence frequency.

[0575] The above environmental condition data may include at least one of an environmental layout ID, a work demand pattern, a traffic restriction condition, a speed constraint condition, or an external constraint variable.

[0576] The above execution history data for multiple strategies (e.g., Strategy A, Strategy B, Strategy C) can be provided as input data to the analysis module (400).

[0577] The analysis module (400) can receive the strategy execution history data and environment condition data and perform a tokenization process. Specifically, the spatial token generation module (12) can convert environment layout information and object placement information into spatial tokens, and the time token generation module (13) can convert the temporal change pattern of the execution history into time tokens. Additionally, the text token generation module (14) can convert the strategy identifier, reward function type, or analysis request condition into text tokens.

[0578] The above token data can be provided to a DST separation attention operation, and the interrelationships between spatial conditions, temporal execution patterns, and strategy conditions can be interpreted as a separation structure. Subsequently, the spatial-language mapping module (17) can generate comparable analysis result data by contextually aligning performance characteristics by strategy.

[0579] The above analysis result data may include at least one of the following: a comparison of conflict risk by strategy, a comparison of bottleneck occurrence frequency, calculation of efficiency and stability indicators, a sensitivity analysis based on environmental conditions, or the result of deriving an optimal strategy.

[0580] For example, conflict risk indicators can be calculated for Strategies A, B, and C, and efficiency and stability scores for each strategy can be quantitatively evaluated. Additionally, the range of variation in strategy performance due to changes in specific environmental conditions can be analyzed to derive the environmental sensitivity of the strategies.

[0581] The above analysis results can be visually provided through the user interface unit (5120) and transmitted to the strategy control module (5130) to be used for selecting the next strategy or adjusting control parameters.

[0582] In addition, the above analysis results can be recorded in the storage unit (5110) and used as subsequent training data. For example, the accuracy of strategy recommendations by the artificial intelligence model can be improved by reflecting the difference between the optimal strategy and the non-optimal strategy as training data.

[0583] Accordingly, the structure according to Fig. 20 goes beyond simple statistical comparison and provides a system for comprehensively evaluating strategy performance by structuring the spatiotemporal interrelationships between strategy execution history and environmental conditions on a token basis.

[0584] With such a configuration, the present invention can quantitatively compare performance between strategies, derive an optimal strategy, and suggest directions for strategy improvement, thereby providing a technical effect that improves the long-term operational efficiency and stability of the system.

[0585] As seen above, according to the embodiments illustrated in FIGS. 18 to 20, the present invention can provide a full-cycle analysis system that analyzes scene data into a spatiotemporal separation analysis structure throughout the entire process of strategy execution.

[0586] Specifically, the analysis of the initial scene data S(0) according to FIG. 18 is performed in the stage prior to strategy execution, allowing potential risk factors based on object placement to be identified in advance and the initial strategy to be corrected. Accordingly, a structure for quantitatively verifying design suitability in the stage prior to execution can be provided.

[0587] Furthermore, the scene sequence analysis during strategy execution according to FIG. 19 can analyze dynamic state transitions over multiple time steps to calculate bottleneck sections, collision risk time periods, or efficiency degradation sections in real time, and feed the analysis results back to the strategy control module (5130) to dynamically correct the strategy during execution.

[0588] In addition, the comparative analysis based on strategy execution history and environmental conditions according to Fig. 20 allows for the quantitative comparison of performance among multiple strategies, analysis of sensitivity to environmental changes, and the derivation of optimal strategies and improvement directions.

[0589] Scene data or execution history data generated at each of the above steps is input into the analysis module (400), undergoes token-based spatiotemporal separation analysis, and is output as analysis result data to be provided to the strategy control module (5130) or storage unit (5110). Accordingly, a closed-loop data flow can be formed in which input data is converted into analysis results and reflected back into the control step.

[0590] The present invention is distinguished by providing a platform structure that goes beyond merely determining collisions or calculating statistical indicators to integrally structure and interpret data generated during the pre-, mid-, and post-strategy execution stages. In particular, by processing spatial and temporal information into separate attention structures, it is possible to implement a system that structurally interprets spatiotemporal relationships and effectively feeds them back into strategy control.

[0591] Accordingly, the present invention can simultaneously achieve improved stability in the strategy design phase, minimized risk in the execution phase, derived an optimal strategy in the post-comparison phase, and improved long-term model performance, and can provide technical effects that comprehensively enhance the efficiency and stability of a simulation-based strategy operation system.

[0592] Meanwhile, according to one embodiment of the present invention, a physical AI-based simulation system (5100) may utilize an attribute processing module (1000) illustrated in FIG. 21 to perform a simulation by reflecting changes in environmental conditions.

[0593] The attribute processing module (1000) may be a module configured to efficiently process problems occurring in various environments. More specifically, the attribute processing module (1000) may be a module configured to generate a prediction result by processing problems or data containing attribute information corresponding to each of a plurality of attributes.

[0594] Here, the problem may include an object or task that must be solved by considering various constraints. In this case, the attributes included in the problem may include elements that define various constraints.

[0595] According to one embodiment of the present invention, the problem may include an operational situation defined by at least one of an environment layout and operational parameters. In this case, the attributes included in the problem may include at least one of operational parameters (e.g., workflow, resource placement, time conditions, etc.) and environment configuration information (e.g., grid structure, shelf placement location, receiving station location, robot initial placement information, operational parameters, etc.).

[0596] In another embodiment, the “problem” may include a problem that achieves a specific goal by considering various constraints. As one embodiment, the problem may include a combinatorial optimization (CO) problem. A combinatorial optimization problem is a problem that seeks an optimal solution to minimize cost or maximize performance under various constraints, and is utilized in various industries and fields (e.g., path optimization, resource allocation, scheduling, graph-based selection problems, robot operation problems, etc.).

[0597] In this regard, the Vehicle Routing Problem (VRP) is a combinatorial optimization problem in logistics and transportation that seeks an optimal solution (i.e., a driving plan) that minimizes total costs (e.g., total travel distance or total transportation costs) while satisfying various constraints (e.g., vehicle load capacity, driving distance, driving time, etc.) when delivering materials from one or more depots to multiple customers (or demand centers).

[0598] For example, a variation of the vehicle routing optimization problem (or vehicle routing problem), including the Capacitated Vehicle Routing Problem (CVRP), may be defined by constraints corresponding to the activated attributes among the set of attributes described below. An instance of the Capacitated Vehicle Routing Optimization Problem (or Capacitated Vehicle Routing Problem) (see FIG. 36(a)) is defined on a graph (G) having a set of nodes (see FIG. 36(b)), and edge weights may be given by Euclidean distances (see FIG. 36(c)). Here, node 0 represents a depot (start and return point), and the remaining nodes may represent customers. Each node has coordinates (see FIG. 36(d)) and an attribute feature (see FIG. 36(e)) that defines constraints specific to the vehicle routing optimization problem. In this specification, “attribute feature” may also be expressed as “attribute characteristic.”

[0599] The goal of the vehicle path optimization problem is to find an optimal solution (see (f) in Fig. 36), where intermediate depot visits can divide the optimal solution into K paths. Nodes corresponding to every customer must be visited exactly once. In this case, the objective function may be to minimize the total travel distance (see (h) in Fig. 36) while satisfying constraints defined by all attribute features (or all attribute characteristics, see (g) in Fig. 36).

[0600] Additionally, variations of the vehicle path optimization problem may be extended by combining additional active attributes (or activated attributes) based on a capacity-constrained vehicle path optimization problem that includes at least one attribute (e.g., Linehaul, Capacity(Q)), etc. Each variation must satisfy the constraints of both the capacity and the active attributes. In one embodiment, the active attributes may include at least one of Backhaul(B), Mixed Backhaul(MB), Open(O), Time Window(TW), and Linehaul(L).

[0601] For an instance (see FIG. 37 (a)), the active attribute can be represented by a binary attribute indicator vector (see FIG. 37 (b)). The vector for each of the active attributes (or the corresponding vector) can be represented as in FIG. 37 (c). In this case, if attribute (A) is active, the attribute may be included in the problem. On the other hand, if the attribute is not active, the attribute may not be included in the problem. For example, a problem involving a time window (e.g., Open Vehicle Routing Problem, OVRPTW) involves an open and time window, so the vector for it can be represented as “(0, 0, 1, 1, 0)”.

[0602] Furthermore, the present specification may represent the vehicle path optimization problem as a sequential decision process within a Markov Decision Process (MDP) framework. In this case, the solution may be constructed autoregressively. In this Markov decision process formulation, the state at time t (see Fig. 37 (d)) may consist of an instance (x) and a partial solution (see Fig. 37 (e)). The agent may select the next node (see Fig. 37 (f)) within the range satisfying the constraints. In this case, the reward may be defined as in Fig. 37 (g).

[0603] Here, “configured autoregressively” can also be understood as using an autoregressive policy composed of an encoder and a decoder. Such a policy defines conditional probabilities (see FIG. 38 (a)), and the probability of a solution being generated can be represented as in FIG. 38 (b). In this case, the goal may be to maximize the expected reward (see FIG. 38 (c)). To this end, at least one algorithm (e.g., the REINFORCE algorithm) or at least one technique (e.g., POMO (Policy Optimization with Multiple Optima) technique, variation-specific reward normalization technique, etc.) may be used or applied in this specification.

[0604] Meanwhile, the attribute processing module (1000) can be universally applied to various problems and can be usefully utilized to solve various problems. By flexibly responding to various problems, this attribute processing module (1000) can efficiently solve various problems.

[0605] In one embodiment, the attribute processing module (1000) may be a module included in at least one system (e.g., an artificial intelligence inference system) linked with the physical AI-based simulation system (5100). In this case, the attribute processing module (1000) may also be understood as “the attribute processing module (1000) of the at least one system” or “the attribute processing module (1000) linked with the physical AI-based simulation system (5100).”

[0606] The attribute processing module (1000) may be configured to perform the role of solving various problems through inference. In the present invention, the attribute processing module (1000) may be a module that is universally applicable to various problems and capable of inferencing even new problems that have not been learned. In this specification, the “attribute processing module (1000)” may also be expressed as an “attribute processing model,” an “artificial intelligence model,” or a “prediction model.”

[0607] Specifically, the attribute processing module (1000) may be configured to generate a prediction result through inference on at least one data (or input data) corresponding to (or included in) the prediction request. In this case, the data may be data containing attributes (or attribute information) corresponding to each of a plurality of attributes. Alternatively, the data may include a problem containing (or composed of) at least one attribute.

[0608] In one embodiment, in a variation of the vehicle path optimization problem, attributes may have at least one of a first attribute (e.g., Intrinsic Semantics for Individual Attributes) and a second attribute (e.g., Contextual Cross-Attribute Interactions).

[0609] For example, a single attribute has an inherent and intrinsic meaning and can maintain the same constraint definition across all combinations of attributes. That is, a single attribute can have the same intrinsic meaning regardless of the combination. In this specification, such a characteristic of an attribute may be referred to as the “first characteristic of the attribute.” For example, the influence of a specific attribute (e.g., Linehaul(L)) can enforce the same maximum path length limit in both VRPL and OVRPL, which have different problem structures.

[0610] For example, an attribute has a first characteristic, but a combination of attributes can generate contextual interactions that go beyond the effects of the individual attributes. That is, attributes maintain a first characteristic that has inherent meaning, but additional contextual interactions may occur when combined with one another. In this specification, such characteristics of an attribute may be referred to as “second characteristics of the attribute.” For example, the influence of a specific attribute (e.g., Linehaul(L)) can be significantly weakened when present with an open route attribute (e.g., Open(O)). This is because length constraints are significantly relaxed as the return to the depot in OVRPL is removed, thereby reducing the influence of the said specific attribute compared to its role in VRPL.

[0611] In this regard, the encoder in the present invention may decompose the final representation (see (d) of FIG. 38) into at least one component to encode the characteristics of the attribute (first characteristic and second characteristic). The component may include at least one of an intrinsic attribute embedding (IAE) and a contextual interaction embedding (CIE), as shown in FIG. 38 (e) and (f).

[0612] Ideological attribute embeddings (IAEs) can be understood as embeddings that express the unique and unchanging inherent meaning of each attribute. Through these Ideological attribute embeddings, the same attribute can be learned to maintain a consistent (or identical) meaning regardless of the problem combination in which it is included. To this end, the present invention encodes analogy relationships into the embedding space so that semantic changes resulting from the addition of attributes appear identically even in different problem contexts. Through these Ideological attribute embeddings, the inherent meaning of an attribute is expressed separately from other attributes, and the model is trained to use a reusable representation across various problem combinations. In this case, the Ideological attribute embedding can be understood as including (or reflecting) a first characteristic.

[0613] Contextual interaction embeddings can be understood as embeddings that express combination-specific interaction effects occurring when multiple attributes are activated together. Based on the inherent meanings of attributes provided by implicit attribute embeddings, they can model interactions within a problem context. In other words, they can model contextual interactions added to the inherent meanings of each attribute. These contextual interaction embeddings reflect the interaction effects that appear when the same attribute is combined with other attributes, and enable the learning of expressions that vary depending on the combination, even for the same attribute. In this case, contextual interaction embeddings can be understood as including (or reflecting) a second characteristic.

[0614] More specifically, in the present invention, the attribute processing module (1000) is enabled to learn an analogy for an attribute (or an analogous characteristic for an attribute) in order to separate the learned attribute into the intrinsic meaning of the attribute and the interaction between attributes.

[0615] In one embodiment, the analogy “Attribute A is for Attribute B, and Attribute C is for Attribute D (which can be expressed as A : B :: C : D)” captures the relationship between two pairs and may imply that a transformation from Attribute A to Attribute B is similar to a transformation from Attribute C to Attribute D. Let us assume that there exist two vehicle path optimization problems (e.g., Problem 1(x), Problem 2(y), etc.) in which Attribute A is not activated. In this case, variations of the vehicle path optimization problem in which Attribute A is activated can be represented as shown in FIG. 39 (a), respectively.

[0616] At this time, based on the first characteristic examined above, the intrinsic meaning of attribute A can be expressed by an analogy as shown in Fig. 39 (b). For example, assuming x is CVRP, y is OVRP, and A is the length constraint (L), this can be expressed as “VRPL : CVRP :: OVRPL : OVRP”.

[0617] These analogical characteristics can be represented (or expressed) in the embedding space as shown in (c) of FIG. 39.

[0618] Furthermore, in the present invention, at least one pre-configured loss function may be used to enable the attribute processing module (1000) to encode the analogy relationship between attribute combinations into an embedding space. For example, the loss function may include an objective function used in contrastive learning (e.g., InfoNCE loss). The objective function may induce the attribute processing module (1000) to learn a representation that preserves the intrinsic meaning of each attribute for the intrinsic meaning of the attribute. Additionally, the loss may also be understood as a combinational loss (see (h) in FIG. 39), and such a combinational loss may be added to a reward function (see (i) in FIG. 39). This may be represented as shown in (j) in FIG. 39.

[0619] In this regard, the calculation of combination loss can also be represented by the algorithm of FIG. 49. To describe the algorithm illustrated in FIG. 49, first, all activated attributes for each instance included in the batch are iterated through, and for each attribute (A), an attribute vector representing the meaning of that attribute is calculated. This may be a process of extracting the implicit meaning of attribute A as a difference vector in the embedding space. Subsequently, all acquired attribute vectors are stored in an attribute pool (P). Then, for each attribute vector included in the pool, other vectors with the same attribute type are selected as positive samples, and vectors corresponding to different types are formed as a negative set. Furthermore, a contrast loss in the form of InfoNCE is calculated to train the model to increase similarity between identical attributes and decrease similarity between different attributes. Finally, the average loss is returned by dividing by the total attribute pool size.

[0620] In one embodiment, let us assume that a function (see FIG. 48 (a)) representing a set of attributes activated in a problem instance (x) is given in a situation where a batch contains instances of various problem types. In this case, for any non-empty subset (see FIG. 48 (b)), an attribute vector can be extracted as shown in FIG. 48 (c). Here, a Node Embedder can be represented as in FIG. 48 (d), and a masking function that removes features of at least one attribute (e.g., Attribute A) from an instance can be represented as in FIG. 48 (e). Furthermore, in the present invention, all attribute vectors (see FIG. 48 (g)) can be collected from all instances in the batch to form an attribute pool (see FIG. 48 (f)). For the attribute vectors extracted from the above attribute pool, vectors corresponding to the same attribute type can be classified into a positive class, and vectors corresponding to different types can be classified into a negative class. One attribute vector is sampled from the positive class. In this case, when the magnitude of the negative class for a specific attribute is B, the combinatorial attribute loss can be calculated through the formula shown in (h) of FIG. 48. Additionally, the temperature parameter can be represented as shown in (i) of FIG. 48.

[0621] Here, “encoding analogical relationships between attribute combinations into an embedding space” can also be understood as encoding analogical consistency for intrinsic attribute embeddings into an embedding space. This indicates that the intrinsic meaning of a specific attribute (e.g., Attribute A) must be identifiable and consistent regardless of the base instance or other active attributes. In this case, the intrinsic meaning of the specific attribute can be represented as shown in (d) of FIG. 39. That is, in the present invention, analogical consistency can be expressed in an embedding space by distinguishing between identical attributes and different attributes.

[0622] For example, in the present invention, attribute vectors can be extracted by masking the features of the corresponding attribute in the same instance. In this case, for a specific attribute (e.g., Attribute A), a vector obtained from another instance (see FIG. 39 (e)) (see FIG. 39 (f)) can be used as a positive sample. On the other hand, vectors corresponding to different attributes (see FIG. 39 (g)) can constitute negative samples.

[0623] Meanwhile, in the present invention, the attribute processing module (1000) may be configured to include an encoder-decoder structure. At this time, in the present invention, the encoder may be configured to further include a specific module.

[0624] For example, as illustrated in FIG. 22 (a) and (b), the attribute processing module (1000) may be configured to include at least one of an encoder (610) and a decoder (620). In this case, the encoder (610) may be configured to further include at least one of a specific module (e.g., “ARC Module”, 611) and a node embedder (611a). In this case, the encoder (610) may be represented as in FIG. 43 (a), and the node embedder may be represented as in FIG. 43 (b).

[0625] The encoder (610) can process information of different attributes (or categories, variables, etc.). In one embodiment, the encoder (610) can process at least one of global attributes and node-specific attributes. Global attributes may include at least one of a global binary variable indicating whether a specific attribute (e.g., Open(O)) is enabled (see Fig. 43 (c)), a run time limit (see Fig. 43 (d)), and a mixed backhaul (see Fig. 43 (e)), which may represent problem-level constraints. On the other hand, node-level constraints such as linehaul demand (see Fig. 43 (f)), backhaul demand (see Fig. 43 (g)), and time windows (e.g., start time (e), end time (l), service time (s), etc.) may be associated with individual nodes.

[0626] In this case, global attribute information is integrated into the depot representation, and node attribute information (see (h) in FIG. 43) is integrated into each customer node to generate node embeddings (or initial node embeddings). These node embeddings can be represented as in FIG. 44 (a). Here, the learnable linear layer (or linear layer) can be represented as in FIG. 44 (b), and the embedding dimension can be represented as in FIG. 44 (c). Additionally, the number of global attributes and node-level attributes can be represented as in FIG. 44 (d), and the time window end value of the depot can be represented as in FIG. 44 (e).

[0627] Subsequently, the above node embedding can be input into a node embedding (611a) composed of N transformer-based blocks (e.g., NodeEmbBlock). The structure of such a node embedding (611a) can be represented as (f) in FIG. 44. Each block constituting the node embedding (611a) may be composed of at least one sublayer.

[0628] For example, the sublayer may be configured to include at least one of a first sublayer (e.g., Multi-Head Attention (MHA)) layer and a second sublayer (e.g., Feed Forward ParallelGatedMLP layer). The first sublayer captures dependencies between different locations of the input sequence, and the second sublayer may apply non-linear transformations to features. This can be represented as in (g) of FIG. 44. Here, the input of the l-th block can be represented as in (a) of FIG. 45, and the first sublayer can be represented as in (b) of FIG. 45. In this case, a is used as a query, and b can provide a key and a value. Additionally, RMSNorm may mean normalization.

[0629] In addition, the ParallelGatedMLP function can be represented as shown in Fig. 45 (c). Here, the element-wise multiplication can be represented as shown in Fig. 45 (d), and the SiLU can represent the Sigmoid Linear Unit (Swish) activation function. Additionally, the trainable linear layer can be represented as shown in Fig. 45 (e).

[0630] Next, a specific module (611) may be a module that performs the role of separating (or decomposing) an attribute representation into an implicit attribute embedding and a contextual interaction embedding. The specific module (611) may utilize at least one of the final output of the node embedding (611a) (i.e., the implicit attribute embedding), an attribute indicator, and a global attribute feature as input.

[0631] For example, given an input having an attribute indicator (see (f) in Fig. 45), if attribute (A) is activated, said attribute may be included in the problem. On the other hand, if the attribute is not activated, said attribute may not be included in the problem. In this case, the initial contextual interaction embedding can be represented as in (g) of Fig. 45. Here, the trainable linear layer can be represented as in (h) of Fig. 45, where Concat means concatenation of feature dimensions and LayerNorm means layer normalization.

[0632] A specific module (611) may be composed of N blocks (e.g., MixerBlock). The l-th block receives a global embedding (see (i) in FIG. 45) and an implicit attribute embedding (see (j) in FIG. 45) as inputs. The N blocks constituting this specific module (611) may be represented as shown in FIG. 46 (a). Here, the trainable linear layer may be represented as shown in FIG. 46 (b).

[0633] In addition, GlobalModule(a, b), which receives a main input (a) and a sub-input (b) and generates an output (see (c) in FIG. 46), can be calculated through the formula shown in (d) in FIG. 46.

[0634] Finally, the final output (or final output embedding) of the encoder (610) can be represented as shown in (e) of FIG. 46. In this case, the final output of the encoder (610) can be calculated by combining (or summing) the intrinsic attribute embedding and the contextual interaction embedding. This can be represented as shown in (f) of FIG. 46.

[0635] Next, the decoder (620) can output action probabilities for each node at each time point t based on the encoded node embeddings. The decoder (620) can calculate context embeddings (see (a) in FIG. 47) using the embeddings of the previously selected node and attribute feature values ​​at time point t. Here, the remaining capacity of the vehicle for line-haul and back-haul can be represented as shown in (b) in FIG. 47, and the current time, the remaining length of the partial solution, and whether there is an open path can be represented as shown in (c) in FIG. 47.

[0636] The decoder (620) can calculate the probability of selecting each node by using the context embedding as the query and the previously encoded node values ​​as the key and value. In this case, to generate a solution that satisfies the constraints, nodes that cannot be visited in each decoding process (or step) according to the activated constraints can be masked. In this case, the probability for the action can be calculated through the formula shown in (d) of FIG. 47. Here, the set of possible nodes at time t can be represented as in (e) of FIG. 47, and the clipping hyperparameter can be represented as in (f) of FIG. 47. Additionally, the dimension of the query can be represented as in (g) of FIG. 47, and the action probability can be calculated by applying a softmax to the probability value.

[0637] Meanwhile, based on the description of the attribute processing module (1000) examined above, the structure and learning flow of the attribute processing module (1000) according to the present invention will be briefly examined below.

[0638] Referring again to FIG. 22 (a) and (b), the encoder (610) of the attribute processing module (1000) can receive data containing attribute information corresponding to each of a plurality of attributes. The encoder (610) can be configured to receive the received data as input and perform the role of generating at least one embedding vector. For example, the encoder (610) can generate a final output (or final embedding, final output embedding, etc.) by using (i.e., combining) an intrinsic attribute embedding (e.g., “IAE”, 24a) corresponding to the final output of a node embedding (611a) and a contextual interaction embedding (e.g., “CIE”, 21b) corresponding to the output of a specific module (611).

[0639] Subsequently, the final output of the encoder (610) can be processed as an input to the decoder (620). Based on the final output, the decoder (620) can output a solution (25, or solution) for the data as a predicted result. In this case, the attribute processing module (1000) can generate first embedding data (25a) based on the solution (25).

[0640] Furthermore, the attribute processing module (1000) can process the data (26a, 27a, 28a) from which at least one attribute has been removed from the data, respectively, to generate embedding data (e.g., second embedding data (26b), third embedding data (27b), fourth embedding data (28b), etc.). And, the attribute processing module (1000) can learn to generate a representation vector corresponding to at least one attribute based on the difference between at least one of the embedding data (26b, 27b, 28b) and the first embedding data (25a).

[0641] As a specific example, the difference between the first embedding data and the second embedding data can be calculated by a vector difference operation. However, the present invention is not limited thereto, and the difference between the first embedding data and the second embedding data can be calculated by a distance function between vectors, a normalization operation, a projection, a linear transformation, a non-linear transformation, or a combination thereof.

[0642] In addition, a normalization operation may be performed on the result of the vector difference operation. For example, L2 normalization, scaling, or a transformation operation including a linear transformation or a non-linear activation function may be additionally performed.

[0643] In this case, the attribute processing module (1000) can be understood to be trained using at least one pre-set loss function (630) to perform training based on the difference between different embedding data. More specific details regarding this will be explained later.

[0644] In another embodiment of the present invention, the attribute processing module (1000) may also be configured with the structure shown in FIG. 23. The encoder (610) of the attribute processing module (1000) may receive data (31, 32, 33) containing attribute information corresponding to each of a plurality of attributes. For example, the encoder (610) may generate a final output embedding by using an attribute mixing embedding (e.g., “Attribute Mixer”, 612) corresponding to the final output of a first module (e.g., “Attribute Mixer”, 612) and a node embedding (32b) corresponding to the output of a second module (e.g., “Node Embedder”, 613).

[0645] Subsequently, the final output of the encoder (610) can be processed as the input of the decoder (620). Based on the final output, the decoder (620) can output a solution (35) for the data (31, 32, 33) as a prediction result. In this case, the attribute processing module (1000) can generate a first embedding data (35a) based on the solution (35).

[0646] Furthermore, the attribute processing module (1000) can process the data (36a, 37a, 38a) from which at least one attribute has been removed from the data (31, 32, 33), respectively, to generate embedding data (e.g., second embedding data (36b), third embedding data (37b), fourth embedding data (38b), etc.). Then, the attribute processing module (1000) can learn to form a representation vector corresponding to at least one attribute based on the difference between at least one of the embedding data (36b, 37b, 38b) and the first embedding data (35a). In this case, the attribute processing module (1000) can perform learning using at least one loss function (630) that has been set in advance.

[0647] Meanwhile, through the learning process described above, the attribute processing module (1000) can be learned so that representation vectors corresponding to the same attribute included in different attribute combinations have similar representations in vector space.

[0648] For example, similarity (or similarity) can be defined based on cosine similarity. To this end, a loss function can be constructed to minimize the distance between representation vectors corresponding to the same attribute and to maximize the distance between representation vectors corresponding to different attributes. Additionally, the learning can be performed using a triplet loss or contrastive learning-based loss function.

[0649] As illustrated in FIG. 24 (a) to (c), the attribute processing module (1000) can be trained such that, among the attributes (41a, 41b, 41c, 41d, 41e, 42a, 42b, 42c, 42d, 42e) included in the first combination in which the first attribute and the second attribute are combined, the representation vectors (41a, 41b, 41c, 41d, 41e) corresponding to the first attribute have similar representations in a vector space. Additionally, the attribute processing module (1000) can be trained such that, among the attributes (41a, 41b, 41c, 41d, 41e, 42a, 42b, 42c, 42d, 42e) included in the first combination, the representation vectors (42a, 42b, 42c, 42d, 42e) corresponding to the second attribute have similar representations in a vector space.

[0650] In addition, the attribute processing module (1000) can be trained such that, among the attributes (43a, 43b, 43c, 43d, 43e, 44a, 44b, 44c, 44d, 44e) included in the fourth combination in which the third attribute and the fourth attribute are combined, the representation vectors (43a, 43b, 43c, 43d, 43e) corresponding to the third attribute have similar representations in a vector space. Additionally, the attribute processing module (1000) can be trained such that, among the attributes (43a, 43b, 43c, 43d, 43e, 44a, 44b, 44c, 44d, 44e) included in the fourth combination, the representation vectors (44a, 44b, 44c, 44d, 44e) corresponding to the fourth attribute have similar representations in a vector space.

[0651] As such, with reference to FIG. 24 (c), it can be seen that the present invention forms well-separated clusters in response to variations of various problems. Additionally, it can be seen that the attribute processing module (1000) according to the present invention effectively separates intrinsic components and contextual components.

[0652] Meanwhile, the attributes used in the learning or inference process of the attribute processing module (1000) in the present invention may vary.

[0653] In this specification, the solution is described using sequential notation (see FIG. 40(a)), in which depot visits divide the sequence into K vehicle paths. For example, the solution in this specification may be represented as shown in FIG. 40(b), which may consist of K paths. Each path (see FIG. 40(c)) starts at a depot and ends at a depot (i.e., see FIG. 40(d)). All customer nodes are included in exactly one path, and the total cost may be represented as in FIG. 40(e).

[0654] Below, based on the notation examined above, we will explain the set of attributes that define variations of different problems (see (f) in Fig. 40).

[0655] In the capacity-constrained vehicle path optimization problem, customer nodes have non-negative demand (see Fig. 40 (g)), which may refer to linehaul services such as delivery. In this case, the attribute corresponding to customer node i may consist of a single demand (or demand value) (see Fig. 40 (h)). Vehicles have the same capacity (see Fig. 41 (a)). The capacity attribute requires that for each path, the sum of customer demand (see Fig. 41 (b)) must not exceed the number of customers (see Fig. 41 (c)).

[0656] In one embodiment, as illustrated in FIG. 25(e), the vehicle does not need to return to the depot (51e) after visiting all customers (54l, 54n, 54m, 54o, 54p) (i.e., after visiting the last customer). This can be represented as in FIG. 41(d), where FIG. 25(e) represents a visual example of a problem involving an Open(O) property.

[0657] In addition, unlike capacity-constrained vehicle route optimization problems where only linehaul customers exist, Backhaul (B) or Mixed Backhaul (MB) variations include backhaul customers (collection), which may mean that goods must be transported to a depot. Each route must satisfy one of two mutually exclusive properties.

[0658] In one embodiment, as illustrated in FIG. 25 (a), the backhaul must start from the depot (51a), visit all linehaul customers (52a, 52b, 52c) first, and then visit the backhaul customers (53a, 53b). On the other hand, as illustrated in FIG. 25 (b), the mixed backhaul may start from the depot (51b) and visit linehaul customers (52d, 52e, 52f) or backhaul customers (53c, 53d) without restriction on the order of visit. These attributes may be represented as in FIG. 41 (e) and may be inc...

Claims

Claim 1 A physical AI-based simulation method characterized by comprising: a step of providing a first user interface for generating an environment layout related to robot operation, wherein the method is performed by a computer; a step of interoperating at least one control strategy model with at least one environment layout generated by the first user interface; and a step of executing a robot simulation for robot operation based on the interoperated environment layout and the control strategy model, and outputting the execution result of the robot simulation in a second user interface. Claim 2 A physical AI-based simulation method according to claim 1, further comprising the steps of: generating an environment layout based on the first user interface; and saving the generated environment layout in a file format. Claim 3 A physical AI-based simulation method according to paragraph 2, wherein the generating step comprises: a step of setting a grid for a space related to robot operation; a step of placing at least one of a robot and a component related to robot operation on the space where the grid is set; and a step of setting operation parameters related to robot operation. Claim 4 In paragraph 3, the above-mentioned operational parameters include a physical AI-based simulation method comprising a parameter defining at least one of a workflow, resource allocation, and temporal operational conditions. Claim 5 A physical AI-based simulation method according to claim 3, wherein the first user interface is provided with a dropdown menu, and the dropdown menu is configured to select at least one of the robot and component and place it in the space as a dropdown. Claim 6 A physical AI-based simulation method according to claim 1, wherein the robot operation includes the operation of an autonomous vehicle, and the environment layout defines an operation environment including at least one of spatial structure information and object placement information related to a plurality of autonomous vehicles. Claim 7 A physical AI-based simulation method according to claim 1, wherein the step of mutual interoperability comprises: a step of selecting one of a plurality of control strategy models; and a step of interoperating the selected control strategy model based on at least one environment layout and standard interface generated by the first user interface. Claim 8 A physical AI-based simulation method according to claim 7, wherein the second user interface is provided with a selection area for selecting any one of the plurality of control strategy models. Claim 9 A physical AI-based simulation method according to claim 1, wherein the second user interface comprises an animation display area that displays the movement state of the robot in an animation form. Claim 10 A physical AI-based simulation method according to claim 9, wherein the second user interface further comprises an indicator display area for displaying performance indicators related to robot operation. Claim 11 A physical AI-based simulation method according to claim 1, wherein the control strategy model comprises at least one of a rule-based algorithm, a reinforcement learning-based policy model, and a language model-based planner. Claim 12 A physical AI-based simulation method according to claim 1, further comprising the step of executing a driving decision of at least one robot based on the execution result of the robot simulation. Claim 13 A physical AI-based simulation method according to claim 12, wherein the step of executing a driving decision of at least one robot comprises: a step of generating a plurality of candidate data in a search mode based on the execution result of the robot simulation; and a step of determining whether to terminate the search mode by comparing the candidate data with a preset transition specification and activating a decision mode to execute a driving decision of the robot. Claim 14 A physical AI-based simulation method according to claim 1, wherein the control strategy model includes a spatial token generation module and a time token generation module that generate spatial features and time features by separating them from data. Claim 15 A physical AI-based simulation method according to claim 14, wherein the control strategy model performs an attention operation that applies a separate spatiotemporal attention mask to the space token and the time token. Claim 16 A physical AI-based simulation method according to claim 14, wherein the control strategy model performs rotation-based position encoding based on the local and global order of tokens. Claim 17 A physical AI-based simulation method according to claim 1, characterized in that the control strategy model includes a mapping module that generates contextual spatial tokens by fusing structured location information and language information that specify spatial information. Claim 18 A physical AI-based simulation method according to claim 17, wherein the mapping module performs a cross-attention operation between a spatial token encoded with the position information and a language token encoded with the language information. Claim 19 A physical AI-based simulation method characterized by comprising: a step of generating an environment layout related to robot operation in a method performed by a computer; a step of linking at least one control strategy model with said environment layout; a step of separating and generating spatial features and temporal features from environment state data generated by said control strategy model based on said environment layout; a step of interpreting the spatiotemporal relationship between a robot and an object based on said spatial features and said temporal features; and a step of executing a robot simulation based on said interpretation results and outputting the execution results of said robot simulation in a user interface. Claim 20 A physical AI-based simulation system comprising: one or more processors; and a memory for storing instructions executed by the one or more processors, wherein the one or more processors provide a first user interface for generating an environment layout related to robot operation by executing the instructions, at least one control strategy model interacts with at least one environment layout generated by the first user interface, executes a robot simulation for robot operation based on the interacted environment layout and the control strategy model, and outputs the execution result of the robot simulation through a second user interface.

Images