A multi-agent cooperation method based on space perception and multi-modal AI fusion

Abstract

The application relates to a multi-agent cooperation method based on space perception and multi-modal AI fusion, and belongs to the field of information technology. The method comprises the following steps: through synchronous acquisition of multi-source sensor data, fusion of multi-modal features, establishment of virtual-real system bidirectional mapping and closed-loop control, construction of a digital twin model of a physical entity and generation of an optimal control strategy; through cross-modal feature fusion space perception action, construction of a dynamic knowledge graph, triggering of a semantic rule engine for logical deduction, acquisition of an optimized strategy set and state prediction data; based on the optimized strategy set, an enhanced strategy set is constructed, a two-stage game is adopted to realize Pareto optimization to obtain a consensus strategy, and a reward shaping technology is used to quantize indicators to correct KPI and feed back to the space perception action. Through multi-modal AI fusion and multi-agent collaborative optimization, the application realizes accurate multi-source data acquisition and fusion, virtual-real state synchronization and dynamic closed-loop control, and improves the collaborative efficiency of the physical entity and the digital twin model.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a multi-agent collaboration method based on the fusion of spatial perception and multimodal AI. Background Technology

[0002] With the advancement of industrialized construction, standardized design has become a key means to improve the manufacturability and assemblability of products. By breaking down the building system into independent yet interconnected standard units, industrialized construction has achieved efficient development. At the same time, digital and intelligent technologies, through virtual simulation, digital twins, and other methods, promote the collaborative optimization of design, production, equipment, and management, realizing data-driven integrated operation of factories and equipment.

[0003] However, traditional technologies still rely primarily on two-dimensional drawings and static graphics, which are insufficient to meet the multi-dimensional needs of complex projects, and the value of data is not fully realized. They typically depend on single-modal data for state perception, leading to fragmented environmental understanding; multi-source heterogeneous sensor data lacks hardware-level synchronization and semantic-level fusion, resulting in spatiotemporal misalignment and information silos; local control strategies often aim for optimal performance of a single entity, lacking multi-agent collaborative mechanisms from a global business perspective, leading to system-level resource conflicts and efficiency bottlenecks. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides a multi-agent collaboration method based on spatial perception and multimodal AI fusion. Through multimodal AI fusion and multi-agent collaborative optimization, it achieves accurate multi-source data acquisition and fusion, virtual-real state synchronization, and dynamic closed-loop control, thereby improving the collaborative efficiency between physical entities and digital twin models.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] This disclosure provides a multi-agent collaboration method based on the fusion of spatial perception and multimodal AI, including the following steps:

[0007] S1. Spatial Perception Action: By synchronously collecting data from multiple sensor sources, fusing multimodal features, establishing a two-way mapping and closed-loop control between virtual and real systems, a digital twin model of the physical entity is constructed and the optimal control strategy is generated.

[0008] S2, Multimodal AI Fusion: By fusing spatially perceived actions through cross-modal feature fusion, a dynamic knowledge graph is constructed to trigger a semantic rule engine for logical deduction, generating the optimal action sequence and predictive analysis results, and obtaining optimization strategy sets and state prediction data;

[0009] S3. Multi-agent collaboration: Based on the optimization policy set, an enhanced policy set is constructed. A two-stage game is used to achieve Pareto optimality and obtain consensus strategy. Reward shaping technology is used to quantify indicators, correct KPIs, and feed them back to spatial perception actions.

[0010] Furthermore, the spatial perception action includes the following steps:

[0011] S11, Global Information Perception and Acquisition: Data is collected by synchronously triggering multi-source heterogeneous sensors, processed by coordinate system one and semantic parsing, and encapsulated into spatiotemporally aligned multimodal structured data packets;

[0012] S12. Data Fusion and Refinement: By fusing images and point cloud data of physical entities, a multimodal query vector is generated, and correlation matching and incremental updates are performed based on spatial and feature similarity to output a global feature set representing the current scene state.

[0013] S13. Virtual-real mapping synchronization: By comparing virtual and real states and synchronizing real-time data, a two-way closed-loop control mechanism is established to realize the dynamic mapping and collaborative optimization of digital twins and physical entities under hard real-time conditions.

[0014] S14. Analysis, Prediction and Optimization: Aggregate real-time and historical data and domain knowledge rules, realize multi-dimensional situational awareness and simulation through digital twins, and generate the optimal control strategy based on model predictive control, forming a self-learning optimization closed loop through closed-loop verification.

[0015] Furthermore, the comprehensive information perception and acquisition includes the following steps:

[0016] Synchronous Triggered Sampling: RGB-D cameras and LiDAR are deployed and synchronously triggered by a hardware synchronization generator to acquire color images, depth images, and point cloud data at the same time; at the same time, the data acquisition card synchronously reads the parameters of the physical sensors and adds time stamps to the synchronously sampled data sets;

[0017] Data group spatial alignment: The acquired RGB images are corrected using the intrinsic parameters of each camera to eliminate lens distortion; then the calculated extrinsic parameters are used to transform all sensor data into the same unified global coordinate system.

[0018] Semantic information extraction: The image segmentation model is used to analyze the RGB image in real time and automatically divide the image into different physical entity regions; at the same time, the feature extraction model is used to extract the visual semantic information of each region.

[0019] Structured encapsulation: All acquired data is used to create structured data packets, which include timestamps, RGB images, depth images, point cloud data, physical sensor readings, segmentation masks, physical entity bounding boxes, and visual semantic information.

[0020] Furthermore, the data fusion and refinement includes the following steps:

[0021] Generate query vector: Read the segmentation mask of the physical entity in the data packet, calculate its minimum bounding rectangle, and crop the corresponding image region from the RGB image in the data packet; at the same time, locate the corresponding pixel region in the depth image, backproject the image coordinates and depth value of each pixel in the region to the three-dimensional space, and gather the calculated points into a three-dimensional point cloud; calculate the axial bounding box of the three-dimensional point cloud, traverse and filter the point cloud data in the data packet, and construct a subset of the point cloud;

[0022] The image region of the physical entity, the 3D point cloud, and the subset of the point cloud are input into a lightweight neural network for deep fusion and dimensionality reduction, and a fixed-length high-dimensional feature vector is output to obtain the query vector of the physical entity.

[0023] Incremental update: All generated query vectors are matched with historical query vectors. For successfully matched physical entities, a smooth update strategy is adopted to merge the new query vectors into the existing global query vectors. For any newly appearing physical entities that cannot be matched, a new query vector entry is created in the global memory. Finally, the set of global query vectors for the latest scene state is output.

[0024] Furthermore, the virtual-real mapping synchronization includes the following steps:

[0025] Virtual-real state comparison: Call the historical state snapshot of the corresponding physical entity in the digital twin model library, compare it with the global query vector set generated by real-time perception, and quantify the position and attitude deviation by calculating the Euclidean distance between feature points and generate state difference signals.

[0026] Real-time data synchronization: The simulation cycle and the controller scanning cycle are kept in sync through a hardware synchronization mechanism; in the virtual to real direction, the control instructions generated based on the state differential signal are serialized and written into the controller storage address; in the real to virtual direction, the original state of the sensor and the feedback data of the device are periodically read.

[0027] Two-way closed-loop control: A digital twin model is established and driven through a dual-mode mechanism. In monitoring mode, real-time data from physical sensors is used to drive the state of virtual model components, realizing real-time reproduction of virtual and real states. In control mode, control commands are generated based on state differential signals and issued for execution.

[0028] Furthermore, the analysis and prediction optimization includes the following steps:

[0029] Situational understanding: Aggregates the latest state data from the digital twin model, time-series information from the historical operation database, and rules and knowledge bases from domain experts; utilizes data mining and machine learning to extract situational data from massive amounts of data, including multi-dimensional state vectors, quantitative evaluation results of key performance indicators, and identified abnormal pattern labels;

[0030] Simulation prediction: Using a multi-dimensional state vector as the initial state, a forward inference calculation is carried out by embedding a digital twin simulation model; at the same time, key performance indicators are transformed into multi-objective functions and constraints of optimization algorithms, and simulation scenarios are set according to the identified abnormal mode labels; by executing model predictive control and multi-objective optimization algorithms, the optimal control strategy that satisfies the performance objectives and constraints is solved in the digital twin space.

[0031] Closed-loop verification: The optimal control strategy is converted into executable instructions for the device and sent to the physical controller; after the instructions are executed, the deviation between the actual output and the expected target is compared by continuously monitoring sensor data, the control effect is quantified and a performance evaluation report is generated.

[0032] Furthermore, the multimodal AI fusion includes the following steps:

[0033] S21. Adaptive cross-modal feature fusion: Align digital twin models, optimal control strategies, and external domain knowledge based on physical entities, map data into high-dimensional feature vectors through an encoder, calculate the correlation weights between different modal features through a cross-attention mechanism, perform weighted fusion of features, and generate a multimodal feature tensor.

[0034] S22. Knowledge Graph Construction and Semantic Reasoning: Using an attribute classifier network, the input multimodal feature tensor is decoded into discrete state attributes corresponding to entities in the knowledge graph; the decoded semantic attributes are associated with the corresponding entity nodes in the knowledge graph to update the current state attributes of the entities;

[0035] The inference engine continuously monitors the updates of entity attributes. When the attribute value meets the preconditions of the preset rules, the predefined domain rules are automatically triggered. The inference conclusion dynamically activates the semantic relationships between entities in the knowledge graph, thereby generating a dynamic knowledge graph that includes the current system status, the impact of faults, and handling suggestions.

[0036] Furthermore, the multimodal AI fusion also includes the following steps:

[0037] S23. Strategy Generation and Predictive Analysis: Based on dynamic knowledge graphs, state assessment and decision generation are performed. A lightweight classifier is used to quickly determine the system state. For abnormal states, a reinforcement learning decision module is triggered to generate the optimal action sequence. At the same time, a time series prediction model is used to analyze historical data and the current state to predict key indicators such as equipment lifespan and task completion time, and output an optimized strategy set and state prediction data.

[0038] Furthermore, the multi-agent collaboration includes the following steps:

[0039] S31, Policy Space Enhancement: Treat the optimized policy set as the basic action space, and use the built-in prior knowledge base and K-nearest neighbor algorithm to generate policies that meet the correlation threshold and construct the enhanced policy set;

[0040] S32. Multi-stage game: The first stage adopts a weight-independent neural network architecture, which does not adjust parameters but only evolves the structure to enable each agent to quickly evaluate the impact of the enhanced policy on local utility. The second stage switches to a differential game framework, where each agent calculates the marginal contribution of the policy change based on the counterfactual baseline, and iteratively updates the policy through a distributed Nash equilibrium search algorithm, eventually converging to the Pareto optimal consensus policy.

[0041] Furthermore, the multi-agent collaboration also includes the following steps:

[0042] S33. Reconstruct and optimize objectives: Analyze the strategic intent of the consensus strategy, apply reward shaping to transform the strategic objectives into quantitative indicators, generate efficiency-type KPIs based on path length rewards, generate balanced KPIs based on path repetition negative rewards, and output the corrected key performance indicators to feed back to spatial perception actions.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention first employs simultaneous acquisition of data from multiple heterogeneous sensors, utilizing coordinate transformation and semantic information extraction techniques to achieve comprehensive information perception and precise data fusion. This process eliminates the time synchronization and spatial alignment problems inherent in traditional solutions, and through the fusion of multimodal data, achieves efficient physical entity modeling and dynamic monitoring. Furthermore, by introducing multi-layer neural networks and deep feature extraction techniques, the generated query vector deeply integrates the visual, geometric, and other features of the physical entity, providing richer contextual information for subsequent semantic reasoning and decision-making.

[0045] In terms of virtual-physical mapping synchronization, a two-way closed-loop control mechanism is established between the digital twin model and the physical entity through precise state comparison and real-time data synchronization, ensuring dynamic mapping and collaborative optimization of the virtual and physical systems under hard real-time conditions. This mechanism not only solves the problem of information silos but also achieves efficient process optimization and task execution through precise control commands and feedback mechanisms. Furthermore, through analysis and predictive optimization, combined with historical data and domain knowledge, real-time situational awareness and simulation inference are realized, providing intelligent support for equipment health prediction and task optimization.

[0046] In the multi-agent collaboration component, by introducing policy space enhancement, game theory models, and differential game frameworks, each agent can make collaborative decisions based on the globally optimal strategy in a dynamic environment. This strategy not only improves the overall efficiency and security of the system but also continuously optimizes key performance indicators through feedback mechanisms, ultimately achieving intelligent collaboration oriented towards global optimization. Attached Figure Description

[0047] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0048] Figure 1 This is a schematic diagram illustrating the steps of a multi-agent collaboration method based on the fusion of spatial perception and multimodal AI, provided in an embodiment of the present invention.

[0049] Figure 2 A schematic diagram illustrating the steps of spatial perception actions provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram illustrating the steps of multimodal AI fusion provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the steps of multi-agent collaboration provided in an embodiment of the present invention. Detailed Implementation

[0052] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0053] This embodiment provides a multi-agent collaboration method based on the fusion of spatial perception and multimodal AI, such as... Figure 1 As shown, it includes the following steps:

[0054] S1. Spatial Awareness and Action: By synchronously acquiring data from multiple sensor sources, fusing multimodal features, establishing a two-way mapping and closed-loop control between the virtual and real systems, a digital twin model of the physical entity is constructed, and the optimal control strategy is generated, such as... Figure 2 As shown, it includes the following steps:

[0055] S11. Global Information Perception and Acquisition: Data is collected by synchronously triggering multi-source heterogeneous sensors, processed by coordinate system one and semantic parsing, and finally encapsulated into a spatiotemporally aligned multimodal structured data packet, including the following steps:

[0056] Synchronous Triggered Sampling: Deploy RGB-D cameras (such as Intel RealSense, Kinect) and LiDAR, and trigger synchronously through a hardware synchronization generator to acquire color images, depth images, and point cloud data at the same time; at the same time, the data acquisition card synchronously reads the parameters of physical sensors (such as the readings of deployed vibration, temperature, and pressure sensors) and adds time stamps to the synchronously sampled data sets.

[0057] Data group spatial alignment: The acquired RGB images are corrected using the intrinsic parameters (focal length, optical center, distortion coefficient) of each camera to eliminate lens distortion; then, the calculated extrinsic parameters (a 4x4 transformation matrix that includes rotation and translation) are used to transform all sensor data into the same unified global coordinate system.

[0058] Sensor data conversion includes:

[0059] Each pixel in the depth image is converted to three-dimensional world coordinates based on its depth value and extrinsic parameters; the point cloud coordinates of each LiDAR are transformed to the same world coordinates through its extrinsic parameter matrix; the installation position of the physical sensor is also recorded in the coordinate system.

[0060] At this point, the same physical point in space achieves a precise one-to-one correspondence between its pixels in the RGB image, its points in the depth map, and its coordinates in the laser point cloud.

[0061] Semantic information extraction: The image segmentation model (such as SAM) is used to parse the RGB image in real time and automatically divide the image into different physical entity regions; at the same time, the feature extraction model (such as DINO) is used to extract the visual semantic information (high-dimensional feature vector) of each region.

[0062] Structured encapsulation: All acquired data is used to create structured data packets, which include timestamps, RGB images, depth images, point cloud data (LiDAR), physical sensor readings, segmentation masks, physical entity bounding boxes, and visual semantic information.

[0063] It should be noted that in the full-domain information perception and acquisition, the time difference is first eliminated at the source by forcing synchronous sampling through hardware signals. Then, based on pre-calibration and coordinate transformation, the accurate fusion of multi-source data in three-dimensional space is achieved. This allows subsequent processing to be carried out based on a unified three-dimensional space, rather than isolated processing in their respective two-dimensional images or point clouds. At the same time, by extracting semantic information, visual semantic information is embedded into the data packet, realizing semantic parsing at the data acquisition end, i.e., perception acquisition, which reduces the burden of subsequent data interpretation.

[0064] S12. Data Fusion and Refinement: By fusing images and point cloud data of physical entities, a multimodal query vector is generated. Based on spatial and feature similarity, association matching and incremental updates are performed to output a global feature set representing the current scene state. This includes the following steps:

[0065] Generate query vector: Read the segmentation mask of the physical entity in the data packet, calculate its minimum bounding rectangle, and crop the corresponding image region from the original RGB image in the data packet (i.e., the output result is the image block of the physical entity); at the same time, locate the corresponding pixel region in the depth image, backproject the image coordinates and depth value of each pixel in the region to the three-dimensional space, and gather the calculated points into a three-dimensional point cloud; calculate the axial bounding box of the three-dimensional point cloud, traverse and filter the point cloud data in the data packet, and construct a subset of the point cloud;

[0066] The image region, 3D point cloud, and subset of point cloud of a physical entity are input into a lightweight neural network (such as a multilayer perceptron, MLP). The appearance texture features and 3D geometric features of the physical entity are deeply fused and dimensionality reduced, and a fixed-length high-dimensional feature vector is output. This vector is the "query vector" representing the physical entity. It condenses the multimodal information of the physical entity and gives it a mathematical representation that can be used for comparison and retrieval.

[0067] Incremental update: All generated query vectors are matched with historical query vectors; for successfully matched physical entities, a smooth update strategy (such as exponential moving average) is used to merge the new query vectors into the existing global query vectors; for any newly appearing physical entity that cannot be matched, a new query vector entry is created for it in the global memory; finally, the global query vector set of the latest scene state is output.

[0068] The basis for association matching is to calculate the intersection-union ratio of the bounding boxes of the physical entities represented by the new and old query vectors in three-dimensional space, and to make a comprehensive judgment by combining the cosine similarity of their feature vectors.

[0069] Understandably, during the data fusion and extraction process, the visual (RGB), geometric (depth / point cloud), and other multimodal information of the segmented physical entity regions are fused into a unified, high-dimensional mathematical representation (query vector), creating a digital identity for each physical entity that can be retrieved, matched, and reasoned about later. Then, the physical entities perceived in the current frame are associated with historical information to achieve continuous tracking and state updates of physical entities, maintaining a dynamically evolving rather than static global scene representation. Finally, the updated global memory state is encapsulated into a structured, lightweight scene semantic description, providing a directly usable interface for subsequent advanced tasks such as virtual-real synchronization and simulation analysis.

[0070] S13. Virtual-Real Mapping Synchronization: By comparing virtual and real states and synchronizing real-time data, a two-way closed-loop control mechanism is established to realize the dynamic mapping and collaborative optimization of digital twins and physical entities under hard real-time conditions, including the following steps:

[0071] Virtual-to-real state comparison: Historical state snapshots of the corresponding physical entities in the digital twin model library are retrieved and compared with the global query vector set generated by real-time perception. By calculating the Euclidean distance between feature points, position and attitude deviations are quantified and a state difference signal is generated. This process realizes the transformation from perceived data to cognitive judgment, providing accurate spatial deviation parameters for subsequent decision-making.

[0072] Real-time data synchronization: An industrial real-time Ethernet protocol is used to establish a session with the programmable logic controller (PLC). A hardware synchronization mechanism ensures that the simulation cycle and the controller scan cycle are synchronized. In the virtual-to-real direction, control commands (such as target position, motion trajectory, and equipment start / stop commands) generated based on state differential signals are serialized and written to the controller's storage address. In the real-to-virtual direction, the original sensor states and equipment feedback data are periodically read. This process solves the problem of information silos, ensuring that command transmission and state feedback are completed within a defined time window, meeting the hard real-time requirements of industrial control.

[0073] Two-way closed-loop control: A dual-mode mechanism is used to establish and drive the digital twin model. In monitoring mode, real-time data from physical sensors drives the state of virtual model components, achieving real-time reproduction of virtual and real states. In control mode, control commands generated based on state differential signals are issued and executed. By continuously correcting the virtual model state and controlling the physical entity based on simulation results, a complete closed loop of perception-analysis-decision-execution is formed, ultimately achieving the goal of optimizing the physical system with a virtual model.

[0074] S14. Analysis, Prediction, and Optimization: Aggregating real-time and historical data and domain knowledge rules, multi-dimensional situational awareness and simulation are achieved through digital twins. Based on model predictive control, the optimal control strategy is generated, and a self-learning optimization closed loop is formed through closed-loop verification. This includes the following steps:

[0075] Situational understanding: Aggregates the latest state data from the digital twin model, time-series information from the historical operation database, and rules and knowledge bases from domain experts (such as equipment maintenance specifications and process quality standards); utilizes data mining and machine learning (such as time-series pattern recognition and cluster analysis) to obtain situational data from massive amounts of data, including multi-dimensional state vectors, quantitative evaluation results of key performance indicators (KPIs), and identified abnormal pattern labels;

[0076] Simulation prediction: Using a multi-dimensional state vector as the initial state, a forward extrapolation calculation is performed by embedding a digital twin simulation model; at the same time, key performance indicators (such as equipment health score and current efficiency value) are transformed into multi-objective functions and constraints of the optimization algorithm, and simulation scenarios are set in a targeted manner based on the identified abnormal mode labels (such as focusing on analyzing the vibration response under different working conditions for abnormal vibration modes); by executing model predictive control and multi-objective optimization algorithms, the optimal control strategy that satisfies the performance objectives and constraints is solved in the digital twin space.

[0077] Closed-loop verification: The optimal control strategy is converted into executable instructions and sent to the physical controller. After the instructions are executed, the deviation between the actual output and the expected target is compared by continuously monitoring sensor data to quantify the control effect and generate a performance evaluation report. The verification results are simultaneously used to update the parameters of the digital twin model and the weights of the optimization algorithm, forming a perception-decision-execution closed loop with self-learning capabilities, ultimately achieving the goal of continuous system optimization.

[0078] It should be noted that the spatial perception action constructs a digital twin model of the physical entity and generates the optimal control strategy. The optimal control strategy is the physical entity action generated based on the initial state of the multi-dimensional state vector, key performance indicators, and abnormal mode labels. It completes the most basic autonomous action based on key performance indicators. In the subsequent process, the key performance indicators and abnormal mode labels will be continuously corrected based on global multi-agent scheduling and cooperation to achieve the final multi-agent collaborative control action.

[0079] S2. Multimodal AI Fusion: By fusing spatially perceived actions through cross-modal feature fusion, a dynamic knowledge graph is constructed to trigger a semantic rule engine for logical deduction, generating optimal action sequences and predictive analysis results, and obtaining optimization strategy sets and state prediction data, such as... Figure 3 As shown, it includes the following steps:

[0080] S21. Adaptive cross-modal feature fusion: Align digital twin models, optimal control strategies, and external domain knowledge based on physical entities, map data into high-dimensional feature vectors through an encoder, calculate the correlation weights between different modal features through a cross-attention mechanism, perform weighted fusion of features, and generate a multimodal feature tensor.

[0081] The data in the digital twin model consists of geometric attributes, physical attributes, rule attributes, and behavioral attributes; the optimal control strategy includes a basic set of operating instructions, which is the smallest functional unit of the control system; external domain knowledge mainly includes process specifications and standards, product knowledge graphs, business and production plans, historical and experience data (such as historical performance data, faults, etc.) and environmental context information (such as weather forecasts, supply chain status, real-time energy prices, etc.).

[0082] It should be noted that the multimodal feature tensor is generated using text work order instructions as the query and visual and point cloud features as the key and value. The attention distribution is calculated through scaled dot-product attention, ultimately generating a dynamically weighted fused feature tensor. This method can adaptively highlight the visual spatial features most relevant to the current semantic context, overcoming the limitations of traditional methods that assume fixed weights between modalities.

[0083] S22. Knowledge Graph Construction and Semantic Reasoning: An attribute classifier network decodes the input multimodal feature tensor into discrete state attributes corresponding to entities in the knowledge graph. This network maps high-dimensional feature vectors to a predefined set of semantic labels (e.g., state: anomaly). This classifier is trained based on historical data and domain expert annotations.

[0084] The decoded semantic attributes are associated with the corresponding entity nodes in the knowledge graph, and the current state attributes of the entity are updated; this process dynamically refreshes the knowledge graph's representation of the physical world state.

[0085] The inference engine continuously monitors updates to entity attributes. When an attribute value meets the preconditions of a predefined rule, the predefined domain rule is automatically triggered. The engine performs logical deduction based on the description logic to deduce implicit conclusions and generate recommended actions.

[0086] The reasoning conclusions dynamically activate the semantic relationships between entities in the knowledge graph, thereby generating a dynamic knowledge graph that includes the current system state, the impact of faults, and handling suggestions.

[0087] S23. Strategy Generation and Predictive Analysis: Based on dynamic knowledge graphs, state assessment and decision generation are performed. A lightweight classifier is used to quickly determine the system state. For abnormal states, a reinforcement learning decision module is triggered to generate the optimal action sequence. At the same time, a time series prediction model is used to analyze historical data and the current state to predict key indicators such as equipment lifespan and task completion time, and output an optimized strategy set and state prediction data.

[0088] Specifically, based on deep situational awareness derived from knowledge graph inference, executable optimization strategies are generated, and future system states are predicted, providing forward-looking basis for decision-making. First, a lightweight model (such as DELM) is used to quickly analyze the multimodal feature tensor to determine the overall system state (normal / abnormal). If the state is "abnormal," the problem is transferred to a more complex model. This model can be a reinforcement learning agent, whose state space is an embedded representation of the knowledge graph, and whose action space is a combination of optimal control strategies. By calculating the long-term value (Q-value) of performing different "actions" in the current "state," higher-order optimization strategies are generated (e.g., "To ensure the overall project schedule, it is recommended that the transport vehicle detour and that maintenance robot R02 be dispatched to equipment A").

[0089] It should be noted that the underlying perceptual features output by the digital twin model are deeply integrated with external domain knowledge (including process specifications, business objectives, historical experience, etc.). Through semantic reasoning of knowledge graphs and dynamic relationship construction, semantic situational awareness and optimization strategies for global collaboration are generated. Its core value lies in elevating the system from a local passive command response to a global proactive collaborative decision-making level. By continuously refining the objective function and constraints of the underlying control logic, it drives the multi-agent system to achieve business-oriented adaptive optimization in a dynamic environment, ultimately achieving overall optimization of efficiency, safety, and resource utilization.

[0090] S3. Multi-agent collaboration: An enhanced policy set is constructed based on an optimized policy set. A two-stage game is used to achieve Pareto optimality and obtain a consensus strategy. Reward shaping techniques are used to quantify indicators, correct KPIs, and feed them back to spatial perception actions, such as... Figure 4 As shown, it includes the following steps:

[0091] S31, Policy Space Enhancement: Treat the optimized policy set as the basic action space, and use the built-in prior knowledge base and K-nearest neighbor algorithm to generate policies that meet the correlation threshold and construct the enhanced policy set;

[0092] S32. Multi-stage game: The first stage adopts a weight-independent neural network (WANN) architecture, which does not adjust parameters but only evolves the structure to enable each agent to quickly evaluate the impact of the enhanced policy concentration strategy on local utility; the second stage switches to a differential game framework, where each agent calculates the marginal contribution of the policy change based on the counterfactual baseline, and iteratively updates the policy through a distributed Nash equilibrium search algorithm, eventually converging to the Pareto optimal consensus policy.

[0093] It should be noted that, unlike traditional neural networks that learn by adjusting weights through gradient descent, the core of WANN is to find high-performance network topologies. Agents can quickly evaluate policies through WANN because it eliminates the need for time-consuming weight training; it finds a computational model that effectively "understands" the impact of policies on their local utility simply through the evolutionary structure. Differential game theory applies the analytical framework of game theory to continuous-time dynamic systems described by differential equations, providing a rigorous mathematical model for complex, interdependent policy interactions among multiple agents in a continuous state space. It forms the theoretical foundation for analyzing equilibrium and convergence. Counterfactual baselines, in multi-agent collaboration, are used to evaluate the contribution of an individual agent's actions to the overall team outcome, thus addressing the "credit allocation" problem. The first stage aims to enable each agent to efficiently "understand" the implications of all strategies for its own benefit, addressing the inefficient training and slow convergence issues caused by the sensitivity to initial parameters in traditional methods. The second stage, to accurately measure the marginal contribution of each agent's decisions to the global picture and solve the credit allocation problem, introduces a counterfactual baseline for calculation. This allows agents to assess "the additional global benefit of taking a certain action compared to maintaining the default action," thus incentivizing them to make more cooperative choices that benefit the overall system. Through this multi-stage, quantitative game interaction, the agents ultimately converge to a commonly accepted, Pareto-optimal consensus strategy.

[0094] S33. Reconstruct and optimize objectives: Analyze the strategic intent of the consensus strategy, apply reward shaping technology to transform the strategic objectives into quantitative indicators, generate efficiency-type KPIs (such as shortening the process) based on path length rewards, generate balanced-type KPIs (such as avoiding congestion) based on path repetition negative rewards, and output the corrected key performance indicators (KPIs) to feed back to the spatial perception action.

[0095] It should be noted that multi-agent collaboration enhances the possibility of collaborative local actions through policy space enhancement and achieves Nash equilibrium convergence of policies among agents based on a multi-stage game framework. Ultimately, the Pareto optimal consensus policy generated by the game is reconstructed into a system-level optimization objective. Its core value lies in transforming the local interest conflicts of multiple agents into global collaborative constraints through distributed decision-making and dynamic game mechanisms, driving continuous iterative optimization of key performance indicators, thereby achieving a leap from single-entity autonomous control to multi-agent collaborative autonomy, ensuring the overall efficiency, security, and resource balance of the system in dynamic environments. At the same time, the revised KPIs are fed back to spatial perception actions to achieve closed-loop optimization, thereby changing the optimal control strategy executed by the physical entity.

[0096] For example, in spatial perception actions, the KPI is set as "shortest path for a single AGV (transport vehicle)". Each AGV agent plans its own shortest path, resulting in congestion at the intersection and low overall efficiency. The traffic coordination agent detects the congestion and, based on multimodal AI fusion and global situational analysis (knowledge graph shows this is a critical path), makes a collaborative decision, generating "In the next 5 minutes, the overall throughput of this intersection has a higher priority than the path length of a single AGV". This decision is fed back to the agents controlling the AGVs in spatial perception actions, changing their objective function. Thus, the revised KPI is generated as "maximize the throughput of the intersection while ensuring safety".

[0097] This invention first constructs a hard real-time synchronized multi-source sensor data acquisition and fusion system through spatial perception actions. It then utilizes hardware synchronization triggering and coordinate unification processing to generate spatiotemporally semantically aligned multimodal structured data packets. Furthermore, it generates an optimal control strategy based on multi-dimensional state vectors through a closed-loop mechanism of virtual-real mapping in a digital twin model, providing a high-precision, low-latency perception and control foundation for the system. Next, through multimodal AI fusion, it overcomes the limitations of traditional single-modal analysis by achieving dynamic weighted fusion of visual, point cloud, and textual data through a cross-attention mechanism. Finally, it leverages knowledge graph semantic reasoning to integrate domain knowledge (process specifications, fault databases, business objectives, etc.). By deeply binding the target with the real-time state, an optimized strategy set that is both data-driven and business-interpretable is generated, thereby elevating the system from a passive response to an active cognitive decision-making level, providing global situational awareness for multi-agent collaboration. Finally, through policy space enhancement and multi-stage game theory mechanisms, multi-agent collaboration transforms local policy conflicts into Pareto optimal consensus strategies, and uses reward shaping technology to reconstruct the game results into efficiency and equilibrium key performance indicators, which are dynamically fed back to the underlying control loop. Ultimately, this achieves a leap from single-entity autonomous control to multi-agent collaborative autonomy, comprehensively improving the system's overall efficiency, security, and resource balance in dynamic environments.

[0098] This invention first employs simultaneous acquisition of data from multiple heterogeneous sensors, utilizing coordinate transformation and semantic information extraction techniques to achieve comprehensive information perception and precise data fusion. This process eliminates the time synchronization and spatial alignment problems inherent in traditional solutions, and through the fusion of multimodal data, achieves efficient physical entity modeling and dynamic monitoring. Furthermore, by introducing multi-layer neural networks and deep feature extraction techniques, the generated query vector deeply integrates the visual, geometric, and other features of the physical entity, providing richer contextual information for subsequent semantic reasoning and decision-making.

[0099] In terms of virtual-physical mapping synchronization, a two-way closed-loop control mechanism is established between the digital twin model and the physical entity through precise state comparison and real-time data synchronization, ensuring dynamic mapping and collaborative optimization of the virtual and physical systems under hard real-time conditions. This mechanism not only solves the problem of information silos but also achieves efficient process optimization and task execution through precise control commands and feedback mechanisms. Furthermore, through analysis and predictive optimization, combined with historical data and domain knowledge, real-time situational awareness and simulation inference are realized, providing intelligent support for equipment health prediction and task optimization.

[0100] In the multi-agent collaboration component, by introducing policy space enhancement, game theory models, and differential game frameworks, each agent can make collaborative decisions based on the globally optimal strategy in a dynamic environment. This strategy not only improves the overall efficiency and security of the system but also continuously optimizes key performance indicators through feedback mechanisms, ultimately achieving intelligent collaboration oriented towards global optimization.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A multi-agent collaborative method based on the fusion of spatial perception and multimodal AI, characterized in that: Includes the following steps: Spatial perception action: By synchronously collecting data from multiple sources of sensors, fusing multimodal features, establishing a two-way mapping and closed-loop control between virtual and real systems, a digital twin model of the physical entity is constructed and the optimal control strategy is generated; Multimodal AI fusion: By fusing spatially perceived actions through cross-modal feature fusion, a dynamic knowledge graph is constructed to trigger a semantic rule engine for logical deduction, generating the optimal action sequence and predictive analysis results, and obtaining optimization strategy sets and state prediction data; Multi-agent collaboration: An enhanced policy set is constructed based on an optimized policy set. A two-stage game is used to achieve Pareto optimality and obtain a consensus policy. Reward shaping technology is used to quantify indicators, correct KPIs, and feed them back to spatial perception actions.

2. The multi-agent collaboration method based on spatial perception and multimodal AI fusion according to claim 1, characterized in that: The spatial perception action includes the following steps: Full-domain information perception and acquisition: Data is collected by synchronously triggering multi-source heterogeneous sensors, processed by coordinate system and semantic parsing, and encapsulated into spatiotemporally aligned multimodal structured data packets; Data fusion and refinement: By fusing images and point cloud data of physical entities, multimodal query vectors are generated, and correlation matching and incremental updates are performed based on spatial and feature similarity to output a global feature set representing the current scene state; Virtual-real mapping synchronization: By comparing virtual and real states and synchronizing real-time data, a two-way closed-loop control mechanism is established to realize the dynamic mapping and collaborative optimization of digital twins and physical entities under hard real-time conditions. Analysis, prediction, and optimization: By aggregating real-time and historical data and domain knowledge rules, multi-dimensional situational awareness and simulation are achieved through digital twins. Based on model predictive control, the optimal control strategy is generated and a self-learning optimization closed loop is formed through closed-loop verification.

3. The multi-agent collaboration method based on spatial perception and multimodal AI fusion according to claim 2, characterized in that: The comprehensive information perception and acquisition includes the following steps: Synchronous Triggered Sampling: RGB-D cameras and LiDAR are deployed and synchronously triggered by a hardware synchronization generator to acquire color images, depth images, and point cloud data at the same time; at the same time, the data acquisition card synchronously reads the parameters of the physical sensors and adds time stamps to the synchronously sampled data sets; Data group spatial alignment: The acquired RGB images are corrected using the intrinsic parameters of each camera to eliminate lens distortion; then the calculated extrinsic parameters are used to transform all sensor data into the same unified global coordinate system. Semantic information extraction: The image segmentation model is used to analyze the RGB image in real time and automatically divide the image into different physical entity regions; at the same time, the feature extraction model is used to extract the visual semantic information of each region. Structured encapsulation: All acquired data is used to create structured data packets, which include timestamps, RGB images, depth images, point cloud data, physical sensor readings, segmentation masks, physical entity bounding boxes, and visual semantic information.

4. The multi-agent collaboration method based on spatial perception and multimodal AI fusion according to claim 3, characterized in that: The data fusion and extraction includes the following steps: Generate query vector: Read the segmentation mask of the physical entity in the data packet, calculate its minimum bounding rectangle, and crop the corresponding image region from the RGB image in the data packet; at the same time, locate the corresponding pixel region in the depth image, backproject the image coordinates and depth value of each pixel in the region to the three-dimensional space, and gather the calculated points into a three-dimensional point cloud; calculate the axial bounding box of the three-dimensional point cloud, traverse and filter the point cloud data in the data packet, and construct a subset of the point cloud; The image region of the physical entity, the 3D point cloud, and the subset of the point cloud are input into a lightweight neural network for deep fusion and dimensionality reduction, and a fixed-length high-dimensional feature vector is output to obtain the query vector of the physical entity. Incremental update: All generated query vectors are matched with historical query vectors. For successfully matched physical entities, a smooth update strategy is adopted to merge the new query vectors into the existing global query vectors. For any newly appearing physical entities that cannot be matched, a new query vector entry is created in the global memory. Finally, the set of global query vectors for the latest scene state is output.

5. A multi-agent collaboration method based on spatial perception and multimodal AI fusion as described in claim 4, characterized in that: The virtual-real mapping synchronization includes the following steps: Virtual-real state comparison: Call the historical state snapshot of the corresponding physical entity in the digital twin model library, compare it with the global query vector set generated by real-time perception, and quantify the position and attitude deviation by calculating the Euclidean distance between feature points and generate state difference signals. Real-time data synchronization: The simulation cycle and the controller scanning cycle are kept in sync through a hardware synchronization mechanism; in the virtual to real direction, the control instructions generated based on the state differential signal are serialized and written into the controller storage address; in the real to virtual direction, the original state of the sensor and the feedback data of the device are periodically read. Two-way closed-loop control: A digital twin model is established and driven through a dual-mode mechanism. In monitoring mode, real-time data from physical sensors is used to drive the state of virtual model components, realizing real-time reproduction of virtual and real states. In control mode, control commands are generated based on state differential signals and issued for execution.

6. The multi-agent collaboration method based on spatial perception and multimodal AI fusion according to claim 5, characterized in that: The analysis and prediction optimization includes the following steps: Situational understanding: Aggregates the latest state data from the digital twin model, time-series information from the historical operation database, and rules and knowledge bases from domain experts; utilizes data mining and machine learning to extract situational data from massive amounts of data, including multi-dimensional state vectors, quantitative evaluation results of key performance indicators, and identified abnormal pattern labels; Simulation prediction: Using a multi-dimensional state vector as the initial state, a forward inference calculation is carried out by embedding a digital twin simulation model; at the same time, key performance indicators are transformed into multi-objective functions and constraints of optimization algorithms, and simulation scenarios are set according to the identified abnormal mode labels; by executing model predictive control and multi-objective optimization algorithms, the optimal control strategy that satisfies the performance objectives and constraints is solved in the digital twin space. Closed-loop verification: The optimal control strategy is converted into executable instructions for the device and sent to the physical controller; after the instructions are executed, the deviation between the actual output and the expected target is compared by continuously monitoring sensor data, the control effect is quantified and a performance evaluation report is generated.

7. The multi-agent collaboration method based on spatial perception and multimodal AI fusion according to claim 1, characterized in that: The multimodal AI fusion includes the following steps: Adaptive cross-modal feature fusion: Align digital twin models, optimal control strategies, and external domain knowledge based on physical entities, map data into high-dimensional feature vectors through an encoder, calculate the correlation weights between features of different modalities through a cross-attention mechanism, perform weighted fusion of features, and generate a multimodal feature tensor; Knowledge graph construction and semantic reasoning: Using an attribute classifier network, the input multimodal feature tensor is decoded into discrete state attributes corresponding to entities in the knowledge graph; the decoded semantic attributes are associated with the corresponding entity nodes in the knowledge graph to update the current state attributes of the entities; The inference engine continuously monitors the updates of entity attributes. When the attribute value meets the preconditions of the preset rules, the predefined domain rules are automatically triggered. The inference conclusion dynamically activates the semantic relationships between entities in the knowledge graph, thereby generating a dynamic knowledge graph that includes the current system status, the impact of faults, and handling suggestions.

8. A multi-agent collaboration method based on spatial perception and multimodal AI fusion as described in claim 7, characterized in that: The multimodal AI fusion also includes the following steps: Strategy generation and predictive analysis: Based on dynamic knowledge graphs, state assessment and decision generation are performed. A lightweight classifier is used to quickly determine the system state. For abnormal states, a reinforcement learning decision module is triggered to generate the optimal action sequence. At the same time, a time series prediction model is used to analyze historical data and the current state to predict key indicators such as equipment lifespan and task completion time, and output an optimized strategy set and state prediction data.

9. A multi-agent collaboration method based on spatial perception and multimodal AI fusion as described in claim 1, characterized in that: The multi-agent collaboration includes the following steps: Policy Space Enhancement: Treating the optimized policy set as the basic action space, and using the built-in prior knowledge base and K-nearest neighbor algorithm, policies that meet the correlation threshold are generated to construct the enhanced policy set; Multi-stage game: The first stage adopts a weight-independent neural network architecture, which does not adjust parameters but only evolves the structure to enable each agent to quickly evaluate the impact of the enhanced policy on local utility. The second stage switches to a differential game framework, where each agent calculates the marginal contribution of policy change based on the counterfactual baseline, and iteratively updates the policy through a distributed Nash equilibrium search algorithm, eventually converging to the Pareto optimal consensus policy.

10. A multi-agent collaboration method based on spatial perception and multimodal AI fusion as described in claim 9, characterized in that: The multi-agent collaboration also includes the following steps: Reconstruct and optimize objectives: Analyze the strategic intent of the consensus strategy, apply reward shaping to transform the strategic objectives into quantitative indicators, generate efficiency KPIs based on path length rewards, generate balanced KPIs based on path repetition negative rewards, and output the revised key performance indicators to feed back to spatial perception actions.

Images