A dual closed-loop collaborative evolution control method and device for full heterogeneous subjects and a medium

Abstract

The application provides a double closed loop collaborative evolution control method, device and medium for full heterogeneous subjects, aiming at the calculation asynchrony and model synchronization error problem of full heterogeneous subjects in a dynamic environment, an asynchronous interaction "execution-evolution" double closed loop architecture is constructed: the subject side uses a time sequence belief network to execute local control flow and realizes asynchronous calculation and synchronization reporting of experience trajectory; the collaborative controller uses a global feature cross attention mechanism to execute strategy evolution flow and triggers a global model synchronization instruction when monitoring an environmental performance deficit. The core innovation is: the evolution timing is triggered accurately by actively determining the environmental drift, and a time and space semantic review mechanism is introduced to forcibly verify the semantic coherence of the parameters issued before the subject hot update and the current physical characteristics. The application solves the timing logic conflict and semantic mismatch risk in asynchronous evolution of large-scale heterogeneous systems, and realizes dynamic calculation synchronization and stable model synchronization among full heterogeneous subjects.

Description

Technical Field

[0001] This invention relates to the field of cyber-physical systems technology, and in particular to a dual-closed-loop collaborative evolution control method, device, and medium for heterogeneous entities. Background Technology

[0002] With the deep integration of artificial intelligence and Internet of Things (IoT) technologies, cyber-physical systems (CPS), represented by intelligent connected vehicles and robot swarms, have become the core of smart transportation and industrial automation. To achieve efficient collaboration among massive heterogeneous physical agents in complex environments, the industry currently widely adopts a multi-agent reinforcement learning paradigm based on "centralized training - decentralized execution." This paradigm deploys a value assessment network with a global perspective in the cloud for centralized training, while a lightweight policy network is deployed on various intelligent agent terminals for distributed execution, and model updates are achieved through periodic parameter distribution.

[0003] In existing engineering practices, cloud-based collaborative controllers iterate model parameters by aggregating interaction trajectory data reported by various terminals and use Over-The-Air (OTA) technology to distribute updated policy weights to physical terminals to cover the old model, thereby promoting the evolution of multi-agent collaborative strategies.

[0004] However, in real-world physical scenarios with high speed, dynamics, and stringent safety requirements, the above paradigm suffers from the following technical drawbacks: First, there is a misalignment in computational synchronization. Due to the asymmetry between the computing power and communication bandwidth of the heterogeneous entities, the experience trajectories aggregated in the cloud are severely fragmented and have phase deviations in temporal logic, making it difficult for centralized training to capture real cross-domain causal relationships. Second, model synchronization instability. There is an inherent time lag between the historical benchmark on which the cloud model evolution is based and the instantaneous physical environment at the time of terminal execution. Existing OTA technology can only complete "static transmission synchronization" at the binary level and cannot achieve "dynamic semantic synchronization" at the physical logic level.

[0005] When structural drift occurs in the environment, blindly issuing and forcibly loading "outdated models" that are mismatched with the current physical characteristics will cause asynchronous control instability or even safety accidents, becoming a core engineering bottleneck that hinders the safe evolution of large-scale heterogeneous systems. Summary of the Invention

[0006] This invention provides a dual-closed-loop collaborative evolution control method, device, and medium for heterogeneous entities, which solves the technical defects in the prior art, where the asynchronous computation of heterogeneous entities and the inaccuracy of model synchronization lead to serious deviations between control decisions and physical reality and uncontrollable evolution processes in environmental drift scenarios.

[0007] This invention provides a dual-closed-loop cooperative evolution control method for heterogeneous agents. The method is applied to a cyber-physical system, which includes multiple heterogeneous intelligent agents deployed across physical and information spaces, and a cooperative controller deployed in the information space. The heterogeneous intelligent agents possess asymmetric observation dimensions and unequal action spaces. The method includes: Each of the heterogeneous intelligent agents is controlled by a local policy network with embedded time-series memory and gating update modules. Based on real-time collected local time-series observation data, a belief state vector is constructed, and real-time control actions are output independently. The experience trajectory, including state, action and feedback reward, is asynchronously reported to the collaborative controller to execute a fast business flow. The collaborative controller aggregates the experience trajectories reported by each of the heterogeneous intelligent entities, performs global feature mining and value assessment based on a centralized value assessment network with an embedded global feature cross-attention mechanism, and generates updated policy model parameters to execute the slow management flow. The collaborative controller monitors the distribution status of environmental characteristics of the cyber-physical system. When it is determined that the preset environmental drift triggering condition is met, it triggers the action of sending the updated strategy model parameters to each of the heterogeneous intelligent agents. The system controls the heterogeneous intelligent agent to receive the updated policy model parameters. Through a built-in spatiotemporal semantic security audit mechanism, it verifies the semantic coherence between the historical benchmark features on which the policy model parameters were generated and the current instantaneous physical features of the heterogeneous intelligent agent. If coherence is determined, an online hot update is performed. If incoherence is determined, the parameter is rejected and the preset security baseline policy is maintained.

[0008] According to the method provided by the present invention, the process of controlling each of the heterogeneous intelligent agents based on a local policy network with embedded time-series memory and gating update modules, constructing a belief state vector based on real-time acquired local time-series observation data, and independently outputting real-time control actions accordingly includes: The current local observation data sequence collected by the heterogeneous intelligent agent is input into the local policy network; the temporal memory and gating update module is used to filter out some observable temporal noise in the physical environment, and the discrete and delayed local observation data is mapped into a continuous belief state vector. Based on the belief state vector, the real-time control action is output in a resource-constrained environment.

[0009] According to the method provided by the present invention, the process of global feature mining and value assessment based on a centralized value assessment network with an embedded global feature cross-attention mechanism includes: Map the asymmetric state features reported by different heterogeneous intelligent agents into query vectors, key vectors, and value vectors; The global feature cross-attention mechanism is used to calculate the attention weights between state features of different intelligent agents, mine the temporal dependency and cross-domain causal relationship of the global state, and generate a context feature vector with global dynamic insight capability. Based on the context feature vector, a long-term value assessment is performed on the joint actions in the experience trajectory to guide the generation of the updated policy model parameters.

[0010] According to the method provided by the present invention, the determination that the preset environmental drift triggering condition is met specifically includes any of the following determination methods: Method 1: Obtain the average joint reward actually acquired by the cyber-physical system within a preset long observation window, and calculate the cumulative performance deviation integral between the average joint reward and the expected optimal value. When the cumulative performance deviation integral exceeds a preset performance deficit threshold, it is determined that the environmental drift triggering condition is met. Method 2: Calculate the relative entropy or cosine similarity between the state space distribution in the current experience trajectory and the historical state space distribution when the policy model parameters were issued last time; when the relative entropy or cosine similarity is greater than the preset structural mutation threshold, it is determined that the environmental drift triggering condition is met.

[0011] According to the method provided by the present invention, the process of verifying the semantic coherence between the historical benchmark features on which the strategy model parameters are based and the current instantaneous physical features of the heterogeneous intelligent agent through a built-in spatiotemporal semantic security audit mechanism includes: Extract the timestamp tags and macro-semantic feature vectors carried in the updated strategy model parameters sent from the cloud; Obtain the real-time microscopic features captured by the current sensors of the heterogeneous intelligent subject and map them into the current semantic feature vector; Calculate the similarity between the macro-semantic feature vector and the current semantic feature vector; If the timestamp tag does not exceed the set physical security validity period and the similarity is greater than the preset security coherence threshold, it is determined to be semantically coherent, and the updated policy model parameters are hot-loaded into the local policy network for online switching; otherwise, it is determined to be semantically incoherent, the update instruction is intercepted and the preset security baseline policy is rolled back to block the control risks caused by asynchronous latency.

[0012] This invention also provides an evolved cooperative controller, deployed on the cyberspace core side of a cyber-physical system, comprising: The global state aggregation module is used to receive the experience trajectories asynchronously reported by various heterogeneous intelligent agents; The global value assessment module is used to map the asymmetric state features reported by different heterogeneous intelligent agents into query vectors, key vectors, and value vectors; calculate the attention weights between the state features of different intelligent agents using the global feature cross-attention mechanism, mine the temporal dependencies and cross-domain causal relationships of the global state, and generate a context feature vector with global dynamic insight capabilities; based on the context feature vector, perform long-term value assessment on the joint actions in the experience trajectory to guide the generation of the updated policy model parameters; The version triggering and distribution module is used to obtain the average joint reward actually obtained by the cyber-physical system within a preset long observation window, and calculate the cumulative performance deviation integral between the average joint reward and the expected optimal value. When the cumulative performance deviation integral exceeds the preset performance deficit threshold, it is determined that the environmental drift triggering condition is met, and the updated strategy model parameters are distributed to each of the heterogeneous intelligent agents.

[0013] This invention also provides a fully heterogeneous execution device with a security audit mechanism, deployed across the edge layer of physical space or information space, comprising: The temporal belief policy module is used to input the local temporal observation data into the gated loop unit embedded in the local policy network, filter temporal noise based on the gated update mechanism, map the discrete local observation data into a continuous belief state vector, and output control actions based on the belief state vector. A real-time control module is used to execute the control actions in both physical and information spaces and asynchronously report the experience trajectory; The logic review unit, connected in series between the external receiving port and the time-series belief strategy module, is used to extract the timestamp tag and macro-semantic feature vector carried in the updated strategy model parameters sent from the cloud; obtain the real-time micro-features captured by the current sensors of the heterogeneous intelligent agent and map them into the current semantic feature vector; and calculate the similarity between the macro-semantic feature vector and the current semantic feature vector. The model hot update unit is used to confirm semantic coherence and hot-load the updated policy model parameters into the local policy network for online switching when the logic review unit determines that the timestamp tag has not exceeded the set physical security validity period and the similarity is greater than the preset security coherence threshold.

[0014] This invention also provides a dual-closed-loop collaborative evolution computing platform for heterogeneous intelligent agents, the platform comprising: At least one evolved collaborative controller, and multiple heterogeneous execution devices deployed across different spatial levels, each of the heterogeneous execution devices serving as the physical carrier of the heterogeneous intelligent entity; As a specific physical scenario application, the fully heterogeneous intelligent entity includes a data synchronization entity deployed in the physical space for processing local state observation, and a computing orchestration entity deployed at the edge layer of the information space for processing computing resource scheduling; The evolved collaborative controller is connected to each of the heterogeneous execution devices via a network to enable asynchronous interaction between business flows and management flows.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the dual closed-loop cooperative evolution control method for heterogeneous entities as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the dual-closed-loop cooperative evolution control method for heterogeneous entities as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the dual-closed-loop cooperative evolution control method for heterogeneous entities as described above.

[0018] The technical solution provided by this invention has the following beneficial effects: 1. Achieving a deep balance between computational overhead and evolutionary efficiency: By constructing a fast and slow dual-loop architecture, high-frequency local control and low-frequency global evolution are decoupled. The terminal performs deterministic control locally based on a temporal belief network, effectively combating physical noise and communication latency; the cloud achieves cross-domain causal mining through a global feature cross-attention mechanism, ensuring the quality of computational synchronization of heterogeneous data in an asynchronous state.

[0019] 2. A precise triggering mechanism for model synchronization has been implemented: By monitoring both performance deficit integral and state distribution entropy, "blind periodic updates" are transformed into "on-demand proactive synchronization," which significantly reduces signaling load while ensuring the system's sensitive response to environmental drift, thereby reducing invalid model distribution from the source.

[0020] 3. A physical-level secure synchronization lock was constructed: Through a spatiotemporal semantic security audit mechanism, the coherence between cloud-based macroscopic features and local instantaneous features is forcibly verified before the model is hot-updated. This step logically blocks the risk of semantic mismatch caused by asynchronous evolution, ensures the steady state of model synchronization, and enables the system to have the engineering survivability of "evolving safely while running".

[0021] 4. Significantly improves the efficiency of joint scheduling of global resources and engineering scalability: This invention decouples control complexity from node scale. Through asynchronous security verification of fast terminal execution and slow cloud evolution, it achieves long-term and robust evolution of ultra-large-scale heterogeneous systems while ensuring physical execution security. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is one of the flowcharts of the dual-closed-loop cooperative evolution control method for heterogeneous subjects provided by the present invention.

[0024] Figure 2 This is the second flowchart of the dual-closed-loop cooperative evolution control method for heterogeneous subjects provided by the present invention.

[0025] Figure 3 This is the third flowchart of the dual-closed-loop cooperative evolution control method for heterogeneous subjects provided by the present invention.

[0026] Figure 4 This is the fourth flowchart of the dual-closed-loop cooperative evolution control method for heterogeneous subjects provided by the present invention.

[0027] Figure 5 This invention provides a fast and slow dual-closed-loop collaborative evolution computing platform for heterogeneous entities.

[0028] Figure 6 This is a schematic diagram of the structure of the evolved cooperative controller provided by the present invention.

[0029] Figure 7 This is a schematic diagram of the structure of the heterogeneous execution device with a security audit mechanism provided by the present invention.

[0030] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] Before describing the specific embodiments of the present invention in detail, we will first conduct a further in-depth analysis of the underlying technical defects existing in the prior art in conjunction with actual application scenarios.

[0033] While existing collaborative control architectures can achieve multi-agent policy evolution in ideal simulation tests, their underlying network structure and synchronization mechanisms still have significant technical limitations when deployed in fully heterogeneous, highly dynamic real physical systems. First, real physical environments exhibit strong partial observability. Traditional policy decision-making and value assessment networks often employ feedforward neural network structures, lacking temporal dynamic processing capabilities. This structure prevents terminal devices from effectively integrating historical observation trajectories, and cloud devices struggle to extract cross-domain causal relationships between globally independent events, making the final output control decisions highly susceptible to jitter and failure in non-stationary environmental changes.

[0034] Secondly, existing collaborative frameworks typically employ high-frequency parameter synchronization mechanisms with fixed periods or fixed training rounds. In deep neural network applications involving massive numbers of parameters, high-frequency model weight distribution not only consumes a large amount of limited wireless communication bandwidth resources, causing severe communication congestion, but also frequently rewriting the policy parameters at the underlying execution end directly disrupts the smoothness of continuous control of physical entities, resulting in severe policy oscillations and seriously affecting the stability of physical entity operation.

[0035] Finally, the existing architecture's model update process is passive and mechanical, lacking the ability to dynamically perceive macroscopic operating conditions. The system cannot proactively identify whether the state space distribution of the current physical environment has undergone substantial changes. When the environment is stable, continuous ineffective model updates and parameter overlays not only cause a serious waste of cloud computing power and communication resources, but also increase the uncertainty risk of system operation. Based on the above-mentioned insufficient underlying network feature extraction capabilities and the resource and stability defects caused by the mechanical synchronization mechanism, this invention proposes a novel dual-closed-loop architecture collaborative evolution control method and computing platform.

[0036] Before describing the technical solutions of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained illustratively.

[0037] Cyber-physical systems refer to complex systems that achieve deep integration of physical space and information space through computing, communication and control technologies. Entities in the physical space can perceive the environmental status in real time and perform computing, decision-making and collaborative control through the information space, including intelligent connected vehicles, robot swarms, intelligent transportation systems, etc.

[0038] Heterogeneous intelligent agents: refers to multiple types of intelligent entities deployed in cyber-physical systems, with asymmetric observation dimensions and unequal action spaces. Each agent differs in perception capabilities, computing resources, and execution functions, and together they constitute the distributed execution units of the system.

[0039] Collaborative Controller: The core computing unit deployed in the information space, responsible for aggregating the experience trajectories reported by various heterogeneous intelligent agents, training and evaluating them through a centralized value assessment network, and deciding when to issue updated policy model parameters to each agent.

[0040] Local policy network: A lightweight neural network model deployed across various heterogeneous intelligent agents. It outputs control actions based on real-time collected local observation data to achieve distributed real-time decision-making. It also has embedded time-series memory and gating update modules.

[0041] Centralized value assessment network: Deployed on the neural network model of the collaborative controller, it has a global feature cross-attention mechanism to evaluate the value of the aggregated global experience trajectory, explore the temporal dependence and causal relationship between the state features of different intelligent agents, and guide the update of the policy model.

[0042] Temporal Memory and Gated Update Module: A neural network component embedded in the local policy network, implemented based on a gated recurrent unit or its variant, used to memorize and filter local temporal observation data, transforming discrete observation sequences into continuous belief state vectors.

[0043] Environmental drift triggering conditions: The criteria for the collaborative controller to determine whether to issue an update strategy model include situations such as the cumulative performance deficit exceeding a preset threshold or structural changes in macro-operating conditions, which are used to achieve on-demand issuance rather than periodic synchronization.

[0044] Hot update: refers to the process of dynamically loading the received updated policy model parameters into the local policy network and making them effective during the operation of a heterogeneous intelligent entity, without restarting the system or interrupting business execution.

[0045] Spatiotemporal semantic security audit mechanism: a mandatory verification logic deployed on all heterogeneous intelligent agents, used to compare the semantic coherence between the historical benchmark features on which the model was generated and the current instantaneous physical features before hot update, and to decide whether to perform the update.

[0046] Security baseline strategy: When spatiotemporal semantic security audit determines that the system is irrelevant, the fully heterogeneous intelligent agent maintains the execution of the preset basic control strategy to ensure the basic operational security of the system after the model update is rejected.

[0047] Local observation data: Environmental state information collected in real time by the heterogeneous intelligent agent through its own sensors, which has the characteristics of locality, temporality and partial observability.

[0048] Contextual Feature Vector: A high-dimensional feature representation generated by the centralized value assessment network through a global feature cross-attention mechanism, which integrates the state features of each intelligent agent and their temporal dependencies, and is used as input for value assessment.

[0049] Joint reward mean: The statistical average of the reward values ​​obtained by all intelligent agents in the system within a preset observation window, used to evaluate the overall performance level of the system.

[0050] Cumulative performance deviation integral: The cumulative amount of deviation between the mean of the joint reward and the expected optimal value over time, used to determine whether the system performance continues to be lower than expected.

[0051] Policy model parameters: The set of weight parameters of the local policy network, which are generated by the collaborative controller and distributed to each heterogeneous intelligent agent to update local decision-making capabilities.

[0052] Real-time micro-features: The instantaneous environmental state information captured by the sensors of the heterogeneous intelligent subject at the current moment is used for semantic coherence comparison with historical benchmark features after feature extraction.

[0053] Physical security validity period: The time difference threshold from the time the model was generated to the current time. If the time difference exceeds this threshold, the model is considered to have lost its semantic coherence with the current physical environment due to excessive time delay.

[0054] The cooperative evolution control method for heterogeneous agents proposed in this invention is mainly applied to cyber-physical systems. In this system, heterogeneous intelligent agents are deployed across physical and information spaces, and their heterogeneity manifests in two dimensions: first, observation asymmetry, meaning that different agents acquire different dimensions and characteristics of observation data based on their sensor types or deployment locations; second, unequal action spaces, for example, some intelligent agents perform discrete data synchronization decisions, while others perform continuous or complex resource orchestration and scheduling decisions.

[0055] The system is logically divided into a fast business flow and a slow management flow. The fast business flow corresponds to the fast execution module, the core of which is a local policy network deployed on physical entities (such as vehicle terminals and edge nodes). This network embeds a gated recurrent unit for temporal modeling of historical local observation data, thereby mapping discrete and delayed local observation data into continuous belief state vectors to support millisecond-level real-time decision-making. The slow management flow corresponds to the slow training module, the core of which is a centralized value assessment network deployed on the cloud information virtual layer. This network embeds a self-attention mechanism, which calculates the cross-attention weights of global features to mine temporal dependencies and cross-domain causal relationships among multiple entities, generating context feature vectors with global dynamic insight capabilities.

[0056] In the dual-loop collaborative process, the system utilizes an environmental drift triggering mechanism to replace traditional periodic parameter synchronization. This mechanism determines whether a structural change has occurred in the physical environment by monitoring the cumulative performance deviation integral between the system's actual gains and the expected optimal value, or by calculating the relative entropy between the current empirical trajectory and the historical distribution. When the triggered model is deployed, the system does not immediately perform overwrite; instead, it intercepts and verifies the data through a spatiotemporal semantic security audit mechanism. This mechanism calculates cosine similarity between the cloud-based model and the current physical spatiotemporal context by extracting the macroscopic semantic feature vector associated with the model's generation and the real-time microscopic features captured by the terminal, thereby quantifying the semantic coherence between the cloud-based model and the current physical spatiotemporal context and ensuring the security of model hot updates.

[0057] The above architecture achieves the decoupling of high real-time response in the physical space from the intellectual evolution in the information space over time. In the embodiments, the specific interactions and parameter evolution processes of each module will be based on the above logic.

[0058] Figure 1 This is one of the flowcharts of the dual-closed-loop cooperative evolution control method for heterogeneous entities provided by the present invention. The method includes the following: Step 101: Control each of the heterogeneous intelligent subjects to construct a belief state vector based on the local policy network with embedded time-series memory and gating update modules, and independently output real-time control actions based on the real-time collected local time-series observation data. The experience trajectory including state, action and feedback reward is asynchronously reported to the collaborative controller to execute a fast business flow.

[0059] In this embodiment, multiple heterogeneous intelligent agents are first deployed in the physical space and at the edge layers of each information space. Each agent serves as an execution unit for a rapid business flow and is equipped with a local policy network that embeds a temporal memory and a gating update module. This local policy network is a lightweight neural network model, the core of which is a gating recurrent unit or a variant thereof, used to process local temporal observation data with strong partially observable characteristics.

[0060] Specifically, each heterogeneous intelligent agent collects local observation data of its surrounding environment in real time through its own sensors. Due to the dynamic and uncertain nature of the physical environment, the collected observation data is often a discrete, noisy time-series sequence. After receiving the local observation data at the current moment, the local policy network inputs it into the embedded temporal memory and gating update module. This module uses a gating mechanism to selectively memorize and forget historical observation sequences, filtering out temporal noise, thereby transforming the discrete observation data stream into a continuous belief state vector. This belief state vector is a compressed representation of the current hidden state of the environment, effectively reflecting the agent's cognition and inference about the environment.

[0061] Based on this belief state vector, the local policy network further generates real-time control actions through mapping at the fully connected layer or output layer, such as vehicle speed adjustment, steering commands, or computing resource scheduling decisions. Each heterogeneous intelligent agent independently executes the above decision-making process without relying on cloud commands in real time, thereby maintaining millisecond-level rapid business response in highly dynamic environments.

[0062] Simultaneously, each heterogeneous intelligent agent encapsulates the experience trajectory data generated during execution. This experience trajectory includes the current state information, the actions performed, and the immediate reward feedback obtained from the environment after execution. This experience trajectory data is asynchronously reported via a wireless network to a collaborative controller deployed in the information space, serving as input data for model training and evaluation in the subsequent slow management flow. The asynchronous reporting mechanism allows each agent to flexibly send data according to its own computing and communication resources, avoiding network congestion and resource contention caused by synchronous reporting.

[0063] Through the above steps, each heterogeneous intelligent agent achieves independent real-time decision-making based on temporal memory in a fast business flow, and provides rich global experience data for the cloud-based collaborative controller, laying the foundation for subsequent model evolution and collaborative optimization.

[0064] Step 102: Control the collaborative controller to aggregate the experience trajectories reported by each of the heterogeneous intelligent entities, perform global feature mining and value assessment based on a centralized value assessment network with an embedded global feature cross-attention mechanism, and generate updated policy model parameters to execute the slow management flow.

[0065] In this embodiment, the collaborative controller is deployed at the core of the information space, serving as the execution entity for the slow management flow. It is responsible for receiving and aggregating the experience trajectories asynchronously reported by various heterogeneous intelligent entities. The experience trajectories include state sequences, action sequences, and corresponding instant reward feedback generated by each entity during the execution of fast business flows. These data reflect the local environmental state and decision-making behavior of each node in the physical space at different times.

[0066] The collaborative controller inputs the aggregated global experience trajectory into a centralized value evaluation network embedded with a global feature cross-attention mechanism. This network, centered on the self-attention mechanism of the Transformer architecture, first maps the asymmetric state features of different heterogeneous intelligent agents into query vectors, key vectors, and value vectors, respectively. By calculating the attention weights between query vectors and key vectors, the network can automatically learn and quantify the degree of mutual influence between the state features of different agents, thereby uncovering global state dependencies and causal relationships across agents and time. For example, in the intelligent connected vehicle scenario, the network can capture the temporal influence relationship between the actions of the preceding vehicle and the state of the following vehicle; in the computing resource scheduling scenario, the network can identify the contribution of load changes of different computing nodes to the overall task completion efficiency.

[0067] Based on the aforementioned attention weights, the value vector is weighted and summed to generate a contextual feature vector with global dynamic insight capabilities. This vector integrates the key state information and temporal evolution characteristics of all heterogeneous entities, providing a high-dimensional compressed representation of the overall macroscopic state of the system. Building upon this, the network combines global collaborative reward signals to perform long-term, unbiased value assessments of joint actions in historical experience trajectories, i.e., calculating the expected cumulative reward that entities can obtain by jointly executing a set of actions under a specific global state.

[0068] The centralized value assessment network continuously optimizes its network parameters through the aforementioned forward computation and backpropagation training, and generates updated policy model parameters based on the assessment results. These updated policy model parameters correspond to the weighted updated version of the local policy network, aiming to enable all heterogeneous intelligent agents to make better collaborative decisions in subsequent execution cycles. The above training and parameter generation process is conducted on hourly or daily timescales, representing a slow management flow that does not interfere with the real-time response in the physical space.

[0069] Through the above steps, the collaborative controller utilizes a global feature cross-attention mechanism to achieve efficient mining and accurate evaluation of massive heterogeneous experience data, providing each heterogeneous intelligent agent with a strategy evolution direction that has global collaborative optimality, while maintaining asynchronous decoupling from the fast business flow, ensuring that real-time control of the physical space is not disturbed by the cloud training process.

[0070] Step 103: Control the collaborative controller to monitor the environmental feature distribution status of the cyber-physical system. When it is determined that the preset environmental drift triggering condition is met, trigger the action of sending the updated strategy model parameters to each of the heterogeneous intelligent agents.

[0071] In this embodiment, after the collaborative controller completes the training of the centralized value assessment network and generates updated policy model parameters, it does not immediately execute the distribution operation. Instead, it continuously monitors the distribution status of the macroscopic environmental characteristics of the cyber-physical system and decides whether to trigger model distribution based on preset environmental drift triggering conditions. This mechanism aims to replace the traditional periodic synchronization method, associating model updates with the real needs of the physical environment, and avoiding the waste of communication resources and control interference caused by ineffective distribution during stable environmental periods.

[0072] Specifically, the collaborative controller maintains a long observation window to statistically analyze the average joint reward reported by each heterogeneous intelligent agent and calculate the cumulative performance deviation integral between this average and the expected Pareto optimal value. When the cumulative performance deviation integral exceeds a preset performance deficit threshold, the collaborative controller determines that the overall collaborative efficiency of the system has significantly fallen below the expected level, indicating a risk due to outdated strategies, and triggers model deployment.

[0073] Meanwhile, the collaborative controller also monitors structural changes in the macroscopic operating conditions. Specifically, the collaborative controller extracts the state-space distribution features from the current empirical trajectory and compares them with the historical state-space distribution at the time of the last model deployment, calculating the KL divergence or cosine similarity between the two. When the calculated KL divergence is greater than a preset structural mutation threshold, or the cosine similarity is lower than a preset threshold, it indicates that the physical environment has fundamentally changed, and the original strategy model is no longer applicable to the new environmental modality. The collaborative controller then triggers a model deployment.

[0074] When any of the above triggering conditions are met, the collaborative controller encapsulates the updated policy model parameters into a data packet and initiates the distribution action to each heterogeneous intelligent agent via the wireless network. In addition to the model weight parameters, the data packet also includes a timestamp label and the historical macro-semantic feature vector on which the model was trained, for the receiving end to perform subsequent spatiotemporal semantic security verification.

[0075] Through the above steps, the collaborative controller implements an on-demand triggering mechanism for model distribution, binding model updates to the actual evolution of the physical environment, reducing unnecessary communication overhead from the source, and ensuring that the policy model update process is only initiated when there are substantial changes in the environment or a significant decline in system performance, laying the foundation for subsequent security audits and conditional hot updates.

[0076] Step 104: Control the heterogeneous intelligent agent to receive the updated strategy model parameters. Through the built-in spatiotemporal semantic security audit mechanism, verify the semantic coherence between the historical benchmark features on which the strategy model parameters were generated and the current instantaneous physical features of the heterogeneous intelligent agent. If coherent, perform online hot update; if incoherent, reject the parameter and maintain the preset security baseline strategy.

[0077] In this embodiment, each heterogeneous intelligent agent receives updated policy model parameters from the collaborative controller via a wireless network. Unlike the existing technology that directly performs hot updates, this embodiment sets up a logical audit unit between the receiving port and the local policy network. This unit has a built-in spatiotemporal semantic security audit mechanism to perform mandatory verification before the model is loaded.

[0078] Specifically, the collaborative controller includes two types of key information in the data packets it sends out: first, the timestamp label of the historical reference time on which the model is based during training; and second, the macroscopic semantic feature vector extracted from the global experience trajectory during the training phase of the model, which represents the overall environmental distribution state of the cyber-physical system at the time of model generation.

[0079] Upon receiving the aforementioned data packet, the logical review unit of the heterogeneous intelligent agent first extracts the timestamp tag and the macroscopic semantic feature vector. Simultaneously, the logical review unit calls upon the real-time microscopic data collected by the sensors at the current moment and maps it into a current semantic feature vector through a preset feature extraction network. This vector represents the instantaneous state of the physical environment in which the agent is currently located.

[0080] Subsequently, the logic auditing unit performs a coherence determination: First, it calculates the time difference between the current moment and the timestamp tag to determine whether it exceeds the preset physical security validity period, which is dynamically set based on the system's maximum tolerable delay; second, it calculates the cosine similarity between the macroscopic semantic feature vector and the current semantic feature vector to determine whether the similarity is greater than a preset security coherence threshold. The logic auditing unit determines that the model is semantically coherent with the current physical environment if and only if the time difference does not exceed the security validity period and the cosine similarity is greater than the security coherence threshold.

[0081] If semantic coherence is determined, the logical review unit sends a release command to the model hot update unit. The model hot update unit then loads the received updated policy model parameters into the local policy network, completing the online hot switch. The switched local policy network continues to execute real-time decisions in the fast business flow based on the new parameters.

[0082] If the model is determined to be semantically incoherent—that is, the time difference exceeds the safe validity period or the cosine similarity is below the safe coherence threshold—it indicates that the model has failed due to significant environmental changes or excessive delays between the generation time and the current time. In this case, the logical review unit forcibly rejects the model parameters, blocking their loading process into the local policy network, and simultaneously triggers the system to revert to a preset safe baseline policy. This safe baseline policy is a pre-stored basic control policy or hard-coded security rules that ensure the basic operational security of the heterogeneous intelligent agents after the model update is rejected, until the next valid model is released.

[0083] Through the above steps, the heterogeneous intelligent agent adds a mandatory spatiotemporal semantic verification step before the model is hot-updated. This fundamentally filters out outdated models caused by cloud training delays or environmental changes, and avoids control instability and security incidents caused by decisions that are semantically misaligned with the current physical environment. This provides the final guarantee for the secure collaborative evolution of cyber-physical systems.

[0084] The dual-closed-loop collaborative evolution control method for heterogeneous agents provided in this invention monitors the distribution of environmental features by controlling the collaborative controller and issuing update parameters when a preset environmental drift trigger condition is met, thus associating model issuance with changes in the physical environment and reducing invalid updates at the source. After receiving update parameters, the heterogeneous intelligent agents verify the semantic coherence between the historical baseline features used in model generation and the current instantaneous physical features through a built-in spatiotemporal semantic security audit mechanism. Online hot updates are only performed when coherence is determined; otherwise, the update is rejected and a safe baseline strategy is maintained. This adds a mandatory verification step before model loading, directly preventing the risk of expired models being executed due to time misalignment. Simultaneously, by controlling each intelligent agent to construct a belief state vector based on a local policy network with embedded temporal memory and gating update modules and independently output real-time control actions, the terminal can maintain smooth decision-making even when the cloud model is not updated or there is a communication delay. Finally, by controlling the collaborative controller to perform global feature mining and value evaluation based on a centralized value evaluation network with an embedded global feature cross-attention mechanism, the update strategy generated in the cloud is ensured to have global collaborative optimality. This invention achieves long-term collaborative evolution of the information space model by asynchronous interaction and security verification between rapid execution on the terminal and slow training in the cloud, while ensuring the security of real-time response in the physical space.

[0085] Further, see Figure 2 Step 101 includes: Step 201: Input the current local observation data sequence collected by the heterogeneous intelligent agent into the local policy network.

[0086] In this embodiment, each heterogeneous intelligent agent continuously collects local observation data of the surrounding environment through its onboard sensors during operation. Due to the dynamic and uncertain nature of the physical environment, the collected data typically presents as a discrete observation sequence with temporal dependencies. At each decision time step, the heterogeneous intelligent agent combines the local observation data from the current moment and several previous moments into a local observation data sequence, which serves as the input to the local policy network. This sequence encompasses the agent's continuous perception of the historical state of the environment, providing the raw data foundation for subsequent temporal feature extraction.

[0087] Step 202: Utilize the temporal memory and gating update module to filter out some observable temporal noise in the physical environment, map discrete and delayed local observation data into a continuous belief state vector, and output the real-time control action based on the belief state vector in a resource-constrained environment.

[0088] After receiving the local observation data sequence, the local policy network passes it to the embedded temporal memory and gating update module. This module, with a gating loop unit as its core structure, selectively remembers and forgets historical information through the coordinated operation of its update and reset gates. Specifically, the update gate controls how much of the hidden state from the previous time step can be transmitted to the current time step, and the reset gate controls how new observation information is fused with historical information. Through this gating mechanism, the module can effectively filter out observation noise and redundant information in the physical environment, while retaining key historical features relevant to decision-making, progressively compressing and mapping the discrete and potentially delayed local observation data sequence into a continuous belief state vector.

[0089] This belief state vector is a low-dimensional continuous representation of the hidden state of the current environment, reflecting the subject's internal cognition and inference about its own environmental state, overcoming the partial observability and temporal noise problems present in the original observation data. The local policy network further inputs this belief state vector into the fully connected output layer, generating real-time control actions through activation function mapping. Since the belief state vector already incorporates historical temporal information, the local policy network can achieve millisecond-level fast decision output in resource-constrained embedded environments or edge computing devices without relying on complex input preprocessing or external memory units. The output real-time control actions vary depending on the specific application scenario, including but not limited to speed adjustment, steering commands, data synchronization task triggering, or resource allocation decisions for computing nodes in intelligent connected vehicles.

[0090] Through the above steps, each heterogeneous intelligent agent can independently complete the construction of a continuous belief state from time-series observations locally, and realize high-frequency, smooth real-time control output based on the belief state. Even in the event of communication interruption or cloud command delay, it can still maintain a stable and fast business flow execution capability.

[0091] Further, see Figure 3 Step 102 includes: Step 301: Map the asymmetric state features reported by different heterogeneous intelligent agents into query vectors, key vectors, and value vectors.

[0092] In this embodiment, after aggregating the experience trajectories asynchronously reported by various heterogeneous intelligent agents, the collaborative controller first preprocesses the trajectory data to extract the state feature information contained therein. Due to differences in perception capabilities, computing resources, and functional positioning among the agents, their reported state features are asymmetric, meaning that the state dimensions, representation methods, and physical meanings differ between agents. The collaborative controller maps these asymmetric state features to the same feature space through a unified feature embedding layer, and further generates corresponding query vectors, key vectors, and value vectors for each agent's state features through linear transformations. The query vector represents the current agent's attention needs to other agents, the key vector represents the feature identifier of the agent's own state being followed by other agents, and the value vector carries the specific information content of the agent's state. This mapping process lays the data foundation for subsequent global feature cross-attention calculations.

[0093] Step 302: Calculate the attention weights between state features of different intelligent agents using the global feature cross-attention mechanism, mine the temporal dependency and cross-domain causal relationship of the global state, and generate a context feature vector with global dynamic insight capability.

[0094] After obtaining the query vectors, key vectors, and value vectors of each subject, the centralized value evaluation network initiates the computation process of a global feature cross-attention mechanism. Specifically, the network first calculates the dot product similarity between each subject's query vector and the key vectors of all other subjects, and then normalizes this similarity using the Softmax function to obtain an attention weight matrix. Each element in this weight matrix represents the degree of influence or attention intensity of one subject's state features on the state features of another subject during the global state evolution process.

[0095] By calculating the attention weights described above, the network can automatically uncover the implicit temporal dependencies in the global state, such as the impact of an agent's action in a previous time step on the state of other agents in subsequent time steps. It can also capture cross-domain causal relationships, such as the coupling between the motion state of a vehicle in physical space and the resource load of computing nodes in information space. Based on the calculated attention weights, the network performs a weighted summation of the value vectors of all agents, fusing them to generate a context feature vector with global dynamic insight. This vector not only contains the key state information of each agent but also encodes the complex interaction relationships and influence weights between agents, providing a high-dimensional compressed representation of the overall macroscopic state of the system.

[0096] Step 303: Based on the context feature vector, perform a long-term value assessment on the joint actions in the experience trajectory to guide the generation of the updated policy model parameters.

[0097] The centralized value assessment network takes the context feature vector as input and combines it with the joint action information recorded in the experience trajectory and the global collaborative reward signal from the environment to calculate the value function. Specifically, the network fuses the context feature vector and the joint action features through a fully connected layer, and outputs an estimate of the expected cumulative reward that each agent can obtain by jointly performing this set of actions in the current state, i.e., the Q value.

[0098] The network undergoes backpropagation training by minimizing the temporal difference error between the Q-value and the actual cumulative reward, continuously optimizing its parameters. After training convergence, the centralized value evaluation network can provide an accurate value assessment for any given combination of global state and joint actions. This assessment result is further used to guide the update of the policy network: through policy gradients or other reinforcement learning optimization algorithms, the collaborative controller utilizes the gradient information provided by the value evaluation network to generate updated policy model parameters. The updated policy model parameters aim to enable all heterogeneous intelligent agents to make collaborative decisions that yield higher expected returns in subsequent execution cycles. The above training and parameter generation process is carried out on hourly or dayly timescales, belonging to a slow management flow, which does not interfere with the real-time response in the physical space, while providing all heterogeneous intelligent agents with a policy evolution direction that has global collaborative optimality.

[0099] Furthermore, step 103 determines whether the preset environmental drift triggering conditions are met, specifically including any of the following determination methods: Method 1: Obtain the average joint reward actually acquired by the cyber-physical system within a preset long observation window, and calculate the cumulative performance deviation integral between the average joint reward and the expected optimal value. When the cumulative performance deviation integral exceeds the preset performance deficit threshold, it is determined that the environmental drift triggering condition is met.

[0100] In one implementation of this embodiment, the collaborative controller determines whether an updated policy model needs to be issued by monitoring the system's long-term cumulative performance. Specifically, the collaborative controller maintains a long observation window of a preset length, which covers several complete experience trajectory acquisition cycles. At each evaluation moment, the collaborative controller counts the instantaneous reward values ​​reported by all heterogeneous intelligent agents within this window and calculates their arithmetic mean as the joint reward mean.

[0101] Meanwhile, the cooperative controller has a pre-set expected optimal value, which can be calculated based on historical best performance records, simulation environment calibration, or theoretical Pareto boundary conditions. This value represents the theoretically optimal cooperative performance that the system can achieve under the current environmental mode. The cooperative controller calculates the difference between the joint reward mean and the expected optimal value, and integrates this difference over time to obtain the cumulative performance deviation integral. This integral reflects the cumulative degree to which the overall cooperative performance of the system deviates from the optimal level within the observation window.

[0102] When the cumulative performance deviation integral exceeds a preset performance deficit threshold, the collaborative controller determines that the system has fallen into a sustained period of low collaborative efficiency, and the existing strategy model is no longer suitable for the current environmental requirements. Without intervention, this will lead to long-term performance loss. At this point, the collaborative controller triggers the action of sending updated strategy model parameters to each heterogeneous intelligent agent, aiming to reverse the performance decline trend through strategy evolution.

[0103] Method 2: Calculate the relative entropy or cosine similarity between the state space distribution in the current experience trajectory and the historical state space distribution when the policy model parameters were issued last time; when the relative entropy or cosine similarity is greater than the preset structural mutation threshold, it is determined that the environmental drift triggering condition is met.

[0104] In another implementation of this embodiment, the cooperative controller determines whether an updated policy model needs to be issued by monitoring structural changes in the physical environment. Specifically, the cooperative controller aggregates the state features from the current experience trajectories reported by each heterogeneous intelligent agent to construct the state space distribution function at the current moment. Simultaneously, the cooperative controller retrieves the historical state space distribution corresponding to the last issuance of policy model parameters from the storage unit.

[0105] The collaborative controller uses relative entropy or cosine similarity as metrics to calculate the degree of difference between the current state space distribution and the historical state space distribution. Relative entropy can quantify the information loss between two probability distributions and is suitable for sensitive detection of distribution differences; cosine similarity measures the similarity of distribution features from the perspective of vector space and is suitable for fast comparison of high-dimensional sparse features.

[0106] When the calculated relative entropy exceeds a preset structural mutation threshold, or the cosine similarity falls below a preset similarity threshold, the collaborative controller determines that a fundamental mode shift has occurred in the physical environment. For example, a road traffic scenario may abruptly change from free-flow to congested, or the computing cluster load pattern may shift from balanced distribution to hotspot aggregation. At this point, even if the system's cumulative performance has not yet significantly decreased, the original strategy model is no longer applicable to the new environmental mode, and continued use may lead to sudden control failure.

[0107] In this case, the collaborative controller also triggers the action of sending updated policy model parameters to each heterogeneous intelligent agent to ensure that each agent can quickly adapt to environmental changes and maintain collaborative performance and security in the new environmental modality.

[0108] By combining or using one of the two judgment methods mentioned above, the collaborative controller realizes an on-demand triggering mechanism for model distribution. This mechanism can respond to gradual performance degradation and provide early warning of sudden environmental changes, thereby fundamentally avoiding invalid distribution and waste of communication resources caused by periodic synchronization.

[0109] Further, see Figure 4 Step 104 includes: Step 401: Extract the timestamp tag and macro-semantic feature vector carried in the updated strategy model parameters sent from the cloud.

[0110] In this embodiment, each heterogeneous intelligent agent receives updated policy model parameters from the collaborative controller via a wireless network. Unlike conventional model distribution, the collaborative controller appends two types of metadata to the distributed data packets: timestamp tags and macro-semantic feature vectors. The timestamp tags record the specific moment when the model parameters were trained and generated in the cloud, used to subsequently determine the model's timeliness. The macro-semantic feature vectors are high-dimensional feature representations extracted from the global experience trajectory by a centralized value evaluation network during the model training phase. This vector encodes the overall environmental distribution state of the cyber-physical system at the time of model generation, including key information such as the state patterns, interaction strengths, and macro-operational modes of each agent. Upon receiving the data packets, the built-in logical review unit of each heterogeneous intelligent agent first parses and extracts the above two types of metadata, temporarily storing them in a local review buffer for subsequent coherence determination.

[0111] Step 402: Obtain the real-time microscopic features captured by the current sensors of the heterogeneous intelligent subject and map them into the current semantic feature vector.

[0112] Simultaneously, the logic review unit actively invokes the sensor data acquisition interface of the heterogeneous intelligent agent at the current moment to obtain the real-time microscopic features of the physical environment in which the agent is located. These real-time microscopic features include, but are not limited to, raw sensor data such as the agent's position, velocity, distribution of surrounding obstacles, communication signal strength, and computational load status. Due to the high dimensionality and heterogeneity of this raw data, the logic review unit internally employs a lightweight feature extraction network, which maintains structural consistency with the feature encoder used during cloud training. The real-time microscopic feature data is forward-propagated through this feature extraction network and mapped into a current semantic feature vector. This vector is a high-dimensional compressed representation of the instantaneous state of the agent's current physical environment, reflecting the environmental modality and contextual information at the current moment.

[0113] Step 403: Calculate the similarity between the macro-semantic feature vector and the current semantic feature vector.

[0114] After obtaining the macroscopic semantic feature vector and the current semantic feature vector, the logic review unit initiates the similarity calculation process. This embodiment preferably uses cosine similarity as the metric, which calculates the cosine of the angle between two vectors in high-dimensional space. The cosine similarity value ranges from -1 to 1. A value closer to 1 indicates that the directions of the two vectors are more consistent, meaning that the macroscopic environment distribution during model generation is semantically closer to the current instantaneous environment state; a value closer to -1 indicates that the semantic directions are opposite, and the environment has fundamentally changed. The logic review unit temporarily stores the calculated cosine similarity value for subsequent coherence determination. Besides cosine similarity, this embodiment can also use other similarity metrics, such as Euclidean distance, Pearson correlation coefficient, or semantic matching scores based on neural networks, depending on the actual scenario requirements.

[0115] Step 404: If the timestamp tag does not exceed the set physical security validity period and the similarity is greater than the preset security coherence threshold, it is determined to be semantically coherent, and the updated policy model parameters are hot-loaded into the local policy network for online switching; otherwise, it is determined to be semantically incoherent, the update instruction is intercepted and the preset security baseline policy is rolled back to block the control risks caused by asynchronous latency.

[0116] The logic review unit performs the final coherence determination based on the aforementioned calculation results. The determination logic includes two conditions that must be met simultaneously: The first condition is timeliness, meaning the time difference between the current moment and the timestamp tag does not exceed the preset physical security validity period. This period is dynamically set based on the maximum tolerable latency of the cyber-physical system. For example, for high-speed intelligent connected vehicles, the security validity period might be set in milliseconds; for computing resource scheduling scenarios, it might be set in seconds or minutes. If the time difference exceeds this period, even if the environmental distribution is similar, the model generation time will be too far in the past to guarantee a match with the current hard real-time requirements.

[0117] The second condition is the semantic consistency condition, which requires the cosine similarity to be greater than a preset safety coherence threshold. This threshold is determined through offline calibration or online adaptive algorithms and reflects the minimum semantic matching required for the safe execution of the model. When the cosine similarity is lower than this threshold, it indicates that there is a significant difference between the environmental distribution at the time of model generation and the current environmental state, and executing the model may lead to a misalignment between the decision and physical reality.

[0118] The logical review unit determines that the issued model is semantically coherent with the current physical environment and sends a release command to the model hot update unit only when both of the above conditions are met. The model hot update unit then performs an online hot-switching operation, loading the received updated policy model parameters into the local policy network. The new model parameters take effect immediately, and each heterogeneous intelligent agent continues to execute real-time decisions in the fast business flow based on the updated policy.

[0119] If any condition is not met—that is, the time difference exceeds the safe validity period or the cosine similarity is lower than the safe coherence threshold—the logical review unit determines that the issued model is semantically irrelevant, immediately intercepts the model parameters, and blocks its loading process to the local policy network. Simultaneously, the logical review unit triggers the system rollback mechanism, switching the all-heterogeneous intelligent agent to a preset safe baseline policy. This safe baseline policy is a pre-stored basic control policy or hard-coded security rules that ensure the basic operational security of the agent after a model update is rejected, until the next valid model issuance or manual intervention.

[0120] Through the above steps, the heterogeneous intelligent agent adds a mandatory spatiotemporal semantic verification step before the model is hot-updated. This fundamentally filters out outdated models caused by cloud training delays or environmental changes, and avoids control instability and security incidents caused by decisions that are semantically misaligned with the current physical environment. This provides the last line of defense for the secure and collaborative evolution of cyber-physical systems.

[0121] like Figure 5 As shown, this embodiment provides a fast and slow dual-closed-loop collaborative evolution computing platform for all heterogeneous entities. Its system architecture spans physical space and information space, and adopts a layered deployment method to achieve asynchronous collaboration between fast business flow and slow management flow.

[0122] (1) Physical space layer: physical entity layer.

[0123] Multiple heterogeneous intelligent entities are deployed in the physical space, including but not limited to intelligent connected vehicles (vehicle 1, vehicle 2, ..., vehicle N). Each vehicle terminal acts as a data synchronization entity, equipped with a local policy network with an embedded gating loop unit (in this embodiment of the invention, it is a vehicle-side data synchronization actor). This layer is the physical entity layer, responsible for the online real-time execution and perception of the intelligent entities, forming the execution end of the rapid business flow.

[0124] Each vehicle terminal collects local observation data in real time through its own sensors (such as...) Figure 5 As shown in the local observation state report), the local policy network with embedded GRU maps discrete time-series observations to continuous belief state vectors, and outputs local actions independently based on these vectors (such as...). Figure 5The local action (points to the real physical environment) is executed within that environment. The environmental feedback obtained after execution (such as...) Figure 5 As shown in the environmental feedback, it serves as an immediate reward, which, together with the status and actions, constitutes an experience trajectory and is asynchronously reported to the information space layer via a wireless network.

[0125] (2) Information space layer: edge cloud.

[0126] In this embodiment of the invention, the information space uses edge cloud as its carrier and is internally divided into two logical layers: the middleware layer and the information virtual layer.

[0127] 1) Middleware layer (online / real-time) The middleware layer, located beneath the edge cloud, serves as a real-time data processing and model / decision delivery layer, undertaking online / real-time middleware functions. This layer includes a data and computing collaboration module, a spatiotemporal semantic security audit unit, a model repository, and a version controller (triggered by environment drift).

[0128] The local observation status reported by each vehicle terminal is first aggregated into the data and computing collaboration module (e.g., Figure 5 (As shown by arrow 2, "Aggregate Global State"). This module performs preliminary organization and alignment of heterogeneous data, and then transmits the aggregated global state to the cloud computing orchestration actor (e.g., ...). Figure 5 (As indicated by arrow 3, "Cloud Decision"). The cloud computing orchestration actor also embeds a GRU, which is responsible for making real-time decisions on the information space side based on the global state.

[0129] The version controller continuously monitors the distribution of environmental characteristics. When the preset environmental drift trigger conditions are met, it triggers the model to issue instructions (such as...). Figure 5 (As shown by arrow 4, "Joint Action Command Issuance"). The issued model parameters are first transmitted to the spatiotemporal semantic safety review unit. This review unit is connected in series between the external receiving port and the local policy model. It performs spatiotemporal semantic safety review on the received updated model parameters, verifying the semantic coherence between the historical baseline features at the time of model generation and the vehicle's current instantaneous physical features. After the review is passed, the model parameters are updated to the vehicle (e.g., ...). Figure 5 The middle arrow 10, "Update Model," points to the vehicle side.

[0130] 2) Virtual information layer (offline / non-real-time).

[0131] The information virtual layer, located above the edge cloud, serves as the model's training and evolution layer, handling offline / non-real-time slow management flow. This layer includes the central critic (Critic), the MAPPO training module, and the simulation environment.

[0132] The global experience data aggregated by the middleware layer is processed and then uploaded to the information virtual layer (such as...). Figure 5As indicated by arrow 6 (“Collecting online data for offline training”), this serves as the data source for offline training. The central critic, Critic, incorporates a self-attention mechanism to dynamically weight the global state sequence, accurately capturing the temporal dependencies and causal relationships between heterogeneous subjects. With the aid of simulation environment verification, the MAPPO training module performs centralized value evaluation based on the collected online data, calculating the joint action effect and reward (e.g., ...). Figure 5 (As shown by arrow 7 "Joint Action Effect & Reward"), generate global value and advantage functions (such as...) Figure 5 (As shown by the middle arrow 8, "Global Value & Advantage Function"), and based on this, iteratively generate new Actor model parameters (such as... Figure 5 (As indicated by arrow 9, "Generate new Actor model parameters"). The generated new model parameters are sent to the model repository in the middleware layer, awaiting release by the version controller.

[0133] (3) Asynchronous interaction mechanism with fast and slow double closed loops.

[0134] This embodiment achieves asynchronous collaboration with both fast and slow closed loops through the aforementioned layered architecture: Rapid closed-loop (business flow): Vehicle terminals in the physical space make millisecond-level real-time decisions based on a local policy network with embedded GRUs, independently output control actions, and receive environmental feedback, forming a rapid closed loop from "local observation → belief state → local action → environmental feedback". This closed loop does not rely on real-time intervention from the cloud, ensuring high real-time responsiveness in the physical space.

[0135] Slow closed loop (management flow): The central cloud of the information space performs long-term training and evolution on an hourly basis based on aggregated global experience data, generating updated policy model parameters, and then distributing them to physical terminals on demand through the version controller (environment drift triggering mechanism) of the edge cloud. Before distribution, the model undergoes mandatory verification by the spatiotemporal semantic security audit unit to ensure the semantic coherence of the model with the current physical environment, forming a slow management closed loop from "global experience aggregation → centralized evaluation → model generation → on-demand distribution → security audit → condition update".

[0136] The two closed loops are completely decoupled in terms of time scale: the fast closed loop ensures real-time security of the physical space, while the slow closed loop promotes the long-term collaborative evolution of intelligent agents. The two are asynchronously connected through the middleware layer of the edge cloud, which not only ensures the continuity of physical execution but also enables the continuous evolution of the information space model.

[0137] Combination Figure 5 The system architecture and fast / slow dual closed-loop physical deployment diagram are shown below. The overall workflow of this embodiment includes the following 10 steps, and the specific explanation of each step is as follows: Step 1: Report the local observation status.

[0138] Each vehicle-mounted data synchronization Actor (i.e., a fully heterogeneous intelligent entity) collects real-time local observation data of the physical environment through various onboard sensors, including but not limited to vehicle position, speed, acceleration, distribution of surrounding obstacles, and communication signal strength. Due to the strong partial observability of the physical environment, the collected data is a discrete time-series sequence. Each vehicle encapsulates its current local observation data and asynchronously reports it to the edge cloud deployed in the information space via a wireless network, specifically received by the data and computing collaboration module in the middleware layer. This reporting process is asynchronous, allowing each vehicle to flexibly send data based on its own computing and communication resources, avoiding network congestion caused by synchronous reporting.

[0139] Step 2: Aggregate the global state.

[0140] The data and computing collaboration module of the edge cloud middleware layer receives local observation status data reported by multiple vehicle-end data synchronization Actors. Due to differences in perception capabilities, data formats, and observation dimensions among the various vehicles, the reported data is asymmetric. The data and computing collaboration module first preprocesses the heterogeneous data, including data cleaning, format unification, and time alignment, and then aggregates the processed multi-source data into global state information. This global state information reflects the distribution and environmental situation of all intelligent agents in the physical space at the current moment, providing input for subsequent cloud-based decision-making.

[0141] Step 3: Cloud-based decision-making.

[0142] The cloud computing orchestration Actor (embedded GRU) in the edge cloud middleware layer receives aggregated global state information from the data and computing collaboration modules and makes cloud-based decisions based on the currently deployed decision logic. These cloud-based decisions include, but are not limited to, computing resource scheduling strategies, data synchronization task allocation, and collaborative control command generation. The cloud computing orchestration Actor then transmits the decision results and aggregated global experience data to the information virtual layer within the same edge cloud for subsequent offline training. This step falls under the online / real-time processing scope of the middleware layer, ensuring that real-time data from the physical space can be promptly transmitted to the information virtual layer.

[0143] Step 4: Issuing instructions.

[0144] After the edge cloud information virtual layer completes model training and generates updated policy model parameters, it sends the model parameters to the version controller in the middleware layer. The version controller continuously monitors the environmental characteristic distribution status of the cyber-physical system. When it determines that the preset environmental drift triggering conditions are met (including cumulative performance deficit exceeding a threshold or structural changes in macroscopic operating conditions), it triggers the action of sending an updated model to the physical terminal. In addition to model weight parameters, the data packet also includes a timestamp tag and the historical macroscopic semantic feature vector on which the model was based during training, for the receiving end to perform subsequent spatiotemporal semantic security verification.

[0145] Step 5: Local action execution and environmental feedback.

[0146] Each vehicle-side data synchronization actor makes real-time decisions based on a local policy network with an embedded GRU. At each decision time step, the vehicle inputs its current local observation data into the local policy network, uses the GRU's gating mechanism to filter temporal noise, maps discrete observations into a continuous belief state vector, and outputs real-time local actions based on this vector, such as speed adjustment, steering commands, or data synchronization task triggering. After the action is executed in the real physical environment, the environment returns immediate reward feedback, including indicators such as task completion efficiency, driving safety, and resource consumption. This feedback, together with the state and action, constitutes a complete experience trajectory.

[0147] Step 6: Collect online data.

[0148] The MAPPO training module of the edge cloud information virtual layer collects aggregated global experience data from the middleware layer, including state sequences, action sequences, and corresponding instant rewards reported by each vehicle. This data reflects the local environmental state and decision-making behavior of each node in the physical space at different times, and serves as the basic input for offline training. The collection process is triggered periodically or on demand, asynchronously with the real-time business flow of the physical space, ensuring that the training process does not interfere with the real-time response of the physical entities.

[0149] Step 7: Calculate the combined action effect and reward.

[0150] With the aid of simulation environment verification, the MAPPO training module of the information virtual body layer processes the collected online data to calculate the joint action effect and reward. Specifically, the module inputs the global state and joint action into the central critic Critic, uses the embedded self-attention mechanism to calculate the attention weights between the state features of different intelligent agents, mines the temporal dependency and cross-domain causal relationship of the global state, generates a context feature vector with global dynamic insight capability, and outputs an estimate of the expected reward for the joint action based on this.

[0151] Step 8: Generate global value and advantage functions.

[0152] Based on the calculation results in step 7, the MAPPO training module generates a global value function and an advantage function. The value function is used to evaluate the expected cumulative return of the system under a specific global state, and the advantage function is used to measure the superiority or inferiority of each action relative to the average level. These functions provide training objectives for the centralized value evaluation network, optimize network parameters by minimizing temporal difference error, and provide gradient guidance for updating the policy network.

[0153] Step 9: Generate new Actor model parameters.

[0154] Based on the value assessment results, the MAPPO training module iteratively generates updated local policy network parameters through policy gradient or other reinforcement learning optimization algorithms. These parameters aim to enable the heterogeneous intelligent agents to make collaborative decisions that yield higher expected returns in subsequent execution cycles. The generated new model parameters are temporarily stored in the information virtual layer, along with timestamp labels and macroscopic semantic feature vectors, awaiting distribution to the model repository in the middleware layer.

[0155] Step 10: Update the model.

[0156] The new Actor model parameters are distributed from the information virtual layer to the model repository in the middleware layer. The version controller determines the distribution timing based on environmental drift triggering conditions, and the distribution data packet is first transmitted to the built-in logical review unit on each vehicle. The review unit performs spatiotemporal semantic security review, extracting the timestamp and macroscopic semantic feature vector carried by the model, and comparing them with the current semantic feature vector obtained by mapping the real-time microscopic features captured by the current sensors. If the timestamp does not exceed the physical security validity period and the cosine similarity is greater than the security coherence threshold, it is determined to be semantically coherent, and the new model parameters are hot-loaded into the local policy network to complete the update; otherwise, it is determined to be semantically incoherent, the parameter is rejected, and the security baseline policy is maintained. Through the above review mechanism, it is ensured that the update is only performed when the model is semantically coherent with the current physical environment, effectively blocking the control risks caused by asynchronous latency.

[0157] Through the collaborative work of the above 10 steps, this embodiment realizes a complete closed loop from real-time perception and execution in physical space to global aggregation and offline training in information space, and then to on-demand distribution and security audit. Under the premise of ensuring the security of real-time response in physical space, it effectively solves the risk of model expiration caused by asynchronous latency between cloud and terminal, and realizes the secure collaborative evolution of multiple heterogeneous intelligent subjects in cyber-physical systems.

[0158] This invention provides an evolved cooperative controller, such as... Figure 6As shown, this controller is deployed on the cyberspace core side (i.e., the central cloud) of the cyber-physical system. As the executor of the slow management flow, it is responsible for the aggregation of global experience data, centralized value assessment, and triggering decisions for model distribution. This collaborative controller includes the following functional modules: Global State Aggregation Module 601: Used to receive experience trajectories asynchronously reported by various heterogeneous intelligent agents. In this embodiment, multiple vehicle-side data synchronization Actors (heterogeneous intelligent agents) deployed in the physical space continuously collect local observation data, execute actions, and receive environmental feedback during the execution of rapid business flows, generating experience trajectories that include status, actions, and immediate rewards. These experience trajectories are asynchronously reported via a wireless network to the data and computing collaboration module in the edge cloud for preliminary aggregation and preprocessing, and finally uploaded to the global state aggregation module in the central cloud.

[0159] Due to differences in perception capabilities, computing resources, and functional positioning among the heterogeneous intelligent agents, the reported experience trajectories exhibit asymmetry, meaning that the state dimensions, representation methods, and data formats differ among different agents. The global state aggregation module first performs a unified format conversion and time alignment on the received multi-source heterogeneous data. Then, it stores the processed experience trajectory data in a global experience replay buffer for subsequent use by the global value assessment module. This module is also responsible for maintaining data indexes and metadata, including the agent identifier, timestamp, and environmental context label corresponding to each experience trajectory, providing a data foundation for subsequent feature mapping and value assessment.

[0160] Global Value Assessment Module 602: The global value assessment module is the core computing unit of the evolved collaborative controller. This module embeds a global feature cross-focus mechanism for in-depth mining and value assessment of the aggregated global experience trajectories. Specifically, this module performs the following processing steps: First, the global value assessment module retrieves batch experience trajectory data from the global state aggregation module and extracts the state features of each heterogeneous intelligent agent contained therein. Since the states of each agent are asymmetric, the module maps state features of different dimensions to the same feature space through a unified feature embedding layer, and further generates corresponding query vectors, key vectors, and value vectors for each agent's state features through linear transformations. The query vector represents the current agent's attention needs to other agents, the key vector represents the feature identifier of the agent's own state being of interest to other agents, and the value vector carries the specific information content of the agent's state.

[0161] Secondly, the global value assessment module utilizes a global feature cross-attention mechanism to calculate attention weights among the state features of different intelligent agents. Specifically, the module calculates the dot product similarity between the query vector of each agent and the key vectors of all other agents, and normalizes it using the Softmax function to obtain an attention weight matrix. Each element in this weight matrix represents the degree of influence or attention intensity of one agent's state feature on another agent's state feature during the global state evolution. Through the calculation of the above attention weights, the module can automatically uncover the implicit temporal dependencies in the global state (such as the influence of an agent's action in a previous time step on the state of other agents in subsequent time steps) and cross-domain causal relationships (such as the coupling relationship between the motion state of a vehicle in physical space and the resource load of computing nodes in information space). Based on the calculated attention weights, the module performs a weighted summation of the value vectors of all agents, fusing them to generate a context feature vector with global dynamic insight capabilities. This vector is a high-dimensional compressed representation of the overall macroscopic state of the system.

[0162] Finally, the global value assessment module fuses the generated context feature vector with the joint action information recorded in the experience trajectory and the global collaborative reward signal from the environmental feedback, and inputs this into the value function network to calculate the estimated cumulative reward that each agent can obtain by jointly performing this set of actions in the current state. The module performs backpropagation training by minimizing the temporal difference error between this estimate and the actual cumulative reward, continuously optimizing its network parameters. After training convergence, the global value assessment module can provide accurate value assessments for any given combination of global state and joint actions, and generate updated policy model parameters based on the assessment results using a policy gradient algorithm to guide the policy evolution direction of each heterogeneous intelligent agent. The above training and parameter generation process is carried out on an hourly or dayly timescale, which is a slow management flow and does not interfere with the real-time response in the physical space.

[0163] Version Triggering and Distribution Module 603: This module monitors the distribution of environmental characteristics within the cyber-physical system, determines when to distribute updated policy model parameters to all heterogeneous intelligent agents, and executes the actual distribution operation. This module incorporates an environmental drift triggering mechanism, replacing the traditional periodic synchronization method and linking model updates to the real needs of the physical environment.

[0164] In one implementation of this embodiment, the version triggering and distribution module determines whether an updated model needs to be distributed by monitoring the system's long-term cumulative performance. Specifically, the module maintains a long observation window of a preset length, calculates the average joint reward reported by all heterogeneous intelligent agents within this window, and compares it with a preset expected optimal value. The expected optimal value can be calculated based on historical optimal performance records, simulation environment calibration, or theoretical Pareto boundary. The module calculates the difference between the average joint reward and the expected optimal value, and integrates this difference over time to obtain the cumulative performance deviation integral. When the cumulative performance deviation integral exceeds a preset performance deficit threshold, the module determines that the system has fallen into a sustained low level of collaborative efficiency, and the existing strategy model cannot adapt to the current environmental requirements, thereby triggering the model distribution action.

[0165] In another implementation of this embodiment, the version triggering and delivery module determines whether an updated model needs to be delivered by monitoring structural changes in the physical environment. The module extracts the current state space distribution features from the global experience trajectory and retrieves the historical state space distribution corresponding to the last time the policy model parameters were delivered from the storage unit. The module uses relative entropy or cosine similarity as a metric to calculate the degree of difference between the current state distribution and the historical state distribution. When the calculated relative entropy is greater than a preset structural mutation threshold, or the cosine similarity is lower than a preset similarity threshold, the module determines that a fundamental mode shift has occurred in the physical environment, and the original policy model is no longer applicable to the new environmental mode, thereby triggering the model delivery action.

[0166] After triggering the distribution action, the version triggering and distribution module retrieves the latest updated policy model parameters from the global value assessment module and encapsulates them into a distribution data packet. In addition to the model weight parameters, the distribution data packet includes two types of metadata: a timestamp tag, recording the specific time the model parameters were trained and generated in the cloud; and a macroscopic semantic feature vector, a high-dimensional feature representation extracted from the global experience trajectory by the global value assessment module during the model training phase. This vector encodes the overall environmental distribution state of the cyber-physical system at the time of model generation. After encapsulation, the module initiates the distribution operation to various heterogeneous intelligent agents via a wireless network, allowing the receiving end to perform subsequent spatiotemporal semantic security verification.

[0167] Through the collaborative work of the above three modules, the evolutionary collaborative controller realizes a complete closed loop from global experience aggregation and centralized value assessment to on-demand model distribution. Under the premise of ensuring the security of real-time response in physical space, it effectively promotes the strategic collaborative evolution of all heterogeneous intelligent agents.

[0168] This invention provides a fully heterogeneous execution device deployed across the edge layers of physical or information space. As the physical carrier of a fully heterogeneous intelligent entity, it is responsible for real-time decision execution in rapid business flows and the security auditing of cloud-deployed models. See also... Figure 7 The device includes the following functional modules: The temporal belief policy module 710 is used to input the local temporal observation data into the gated loop unit embedded in the local policy network, filter temporal noise based on the gated update mechanism, map the discrete local observation data into a continuous belief state vector, and output control actions based on the belief state vector. The real-time control module 720 is used to execute the control action in the physical space and the information space and asynchronously report the experience trajectory; The logic review unit 730 is connected in series between the external receiving port and the time-series belief strategy module. It is used to extract the timestamp tag and macro-semantic feature vector carried in the updated strategy model parameters sent from the cloud; obtain the real-time micro-features captured by the current sensor of the heterogeneous intelligent subject and map them into the current semantic feature vector; and calculate the similarity between the macro-semantic feature vector and the current semantic feature vector. The model hot update unit 740 is used to confirm semantic coherence and hot-load the updated policy model parameters into the local policy network for online switching when the logic review unit determines that the timestamp tag has not exceeded the set physical security validity period and the similarity is greater than the preset security coherence threshold.

[0169] The following describes the dual-closed-loop collaborative evolution computing platform for heterogeneous intelligent agents provided by the embodiments of the present invention. The dual-closed-loop collaborative evolution computing platform for heterogeneous intelligent agents described below can be referred to in correspondence with the dual-closed-loop collaborative evolution control method for heterogeneous agents described above.

[0170] This invention provides a dual-closed-loop collaborative evolution computing platform for heterogeneous intelligent agents. Deployed across physical and information spaces, the platform achieves collaborative control and strategy evolution of multiple heterogeneous intelligent agents through a fast-slow dual-closed-loop architecture. The platform includes at least one evolving collaborative controller and multiple heterogeneous execution devices deployed across different spatial levels. Each heterogeneous execution device serves as the physical carrier of a heterogeneous intelligent agent, and asynchronous interaction between business and management flows is achieved through network connections.

[0171] Heterogeneous execution devices, as the physical carriers of intelligent agents, manifest themselves in the following two forms: The data synchronization entity deployed in physical space, typically represented by intelligent connected vehicles, is a local policy network equipped with an embedded gated loop unit. It constructs a belief state vector based on local observation data and outputs real-time control actions, asynchronously reporting the experience trajectory after execution. This entity has a built-in logic review unit that performs spatiotemporal semantic security review when receiving cloud models.

[0172] The computing orchestration entity deployed at the edge layer of the information space is the cloud computing orchestration Actor. It is also equipped with a local policy network with an embedded gated loop unit. Based on local observation data of the information space (such as node load and communication quality), it outputs cloud decision actions, reports the experience trajectory after execution, and has the same built-in security audit mechanism.

[0173] The evolved collaborative controller connects to various heterogeneous execution devices via a network, enabling asynchronous interaction between business and management flows: Business flow: Each execution device makes real-time decisions and executes based on local policies, and asynchronously reports the experience trajectory to the controller to ensure real-time response in the physical space.

[0174] Management flow: The controller performs offline training based on aggregated data and triggers model distribution on demand; after receiving the data, the device verifies the semantic coherence through a logical review unit. If it passes the verification, it performs a hot update; otherwise, it rejects the data and maintains the safety baseline policy.

[0175] Through the above architecture, this platform achieves decoupling and collaboration between real-time response in physical space and long-term evolution in information space, promoting the continuous evolution of strategies for heterogeneous intelligent agents while ensuring security.

[0176] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840. The processor 810, communication interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a dual-closed-loop collaborative evolution control method for heterogeneous agents. This method includes: controlling each heterogeneous intelligent agent to construct a belief state vector based on a local policy network with embedded time-series memory and gating update modules, constructing a belief state vector based on real-time collected local time-series observation data, and independently outputting real-time control actions accordingly. The method also includes asynchronously reporting experience trajectories containing state, actions, and feedback rewards to the collaborative controller to execute a fast business flow; and controlling the collaborative controller to aggregate the experience trajectories reported by each heterogeneous intelligent agent, and performing global feature mining and value evaluation based on a centralized value evaluation network with an embedded global feature cross-attention mechanism. The system estimates and generates updated policy model parameters to execute a slow management flow; it controls the collaborative controller to monitor the environmental feature distribution status of the cyber-physical system, and when it determines that the preset environmental drift triggering condition is met, it triggers the action of sending the updated policy model parameters to each of the heterogeneous intelligent agents; it controls the heterogeneous intelligent agents to receive the updated policy model parameters, and through the built-in spatiotemporal semantic security audit mechanism, it verifies the semantic coherence between the historical benchmark features on which the policy model parameters were generated and the current instantaneous physical features of the heterogeneous intelligent agents. If it is determined to be coherent, it performs online hot update; if it is determined to be incoherent, it rejects the parameter and maintains the preset security baseline policy.

[0177] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the dual-closed-loop cooperative evolution control method for heterogeneous agents provided by the above methods. The method includes: controlling each of the heterogeneous intelligent agents to construct a belief state vector based on a local policy network with embedded time-series memory and gating update modules, constructing a belief state vector based on real-time collected local time-series observation data, and independently outputting real-time control actions accordingly, and asynchronously reporting the experience trajectory including state, action, and feedback reward to the cooperative controller to execute a fast business flow; controlling the cooperative controller to aggregate the experience trajectories reported by each of the heterogeneous intelligent agents, based on the embedded... A centralized value assessment network with a global feature cross-attention mechanism performs global feature mining and value assessment to generate updated strategy model parameters for executing a slow management flow. The collaborative controller monitors the environmental feature distribution status of the cyber-physical system. When a preset environmental drift trigger condition is met, it triggers the distribution of the updated strategy model parameters to each of the heterogeneous intelligent agents. The heterogeneous intelligent agents receive the updated strategy model parameters and, through a built-in spatiotemporal semantic security audit mechanism, verify the semantic coherence between the historical baseline features on which the strategy model parameters were generated and the current instantaneous physical features of the heterogeneous intelligent agents. If coherent, an online hot update is performed; if incompatible, the parameter is rejected and a preset security baseline strategy is maintained.

[0179] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the dual-closed-loop cooperative evolution control method for heterogeneous agents provided by the above methods. This method includes: controlling each of the heterogeneous intelligent agents to construct a belief state vector based on real-time collected local time-series observation data using a local policy network with embedded time-series memory and gating update modules; independently outputting real-time control actions accordingly; and asynchronously reporting experience trajectories including state, actions, and feedback rewards to the cooperative controller to execute a fast business flow; controlling the cooperative controller to aggregate the experience trajectories reported by each of the heterogeneous intelligent agents, based on a centralized system with an embedded global feature cross-attention mechanism. The value assessment network performs global feature mining and value assessment to generate updated strategy model parameters for executing a slow management flow. The collaborative controller monitors the environmental feature distribution status of the cyber-physical system. When a preset environmental drift trigger condition is met, it triggers the distribution of the updated strategy model parameters to each of the heterogeneous intelligent agents. The heterogeneous intelligent agents receive the updated strategy model parameters and, through a built-in spatiotemporal semantic security audit mechanism, verify the semantic coherence between the historical baseline features upon which the strategy model parameters were generated and the current instantaneous physical features of the heterogeneous intelligent agents. If coherent, an online hot update is performed; if incoherent, the parameter is rejected and a preset security baseline strategy is maintained.

[0180] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0181] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dual-closed-loop cooperative evolution control method for heterogeneous entities, characterized in that, The method is applied to a cyber-physical system, which includes multiple heterogeneous intelligent agents deployed across physical and information spaces, and a cooperative controller deployed in the information space. The heterogeneous intelligent agents possess asymmetric observation dimensions and unequal action spaces. The method includes: The control system of each heterogeneous intelligent agent is based on a local policy network with embedded time-series memory and gating update modules. It constructs a belief state vector based on real-time collected local time-series observation data, and independently outputs real-time control actions accordingly. It also asynchronously reports the experience trajectory, which includes state, action and feedback reward, to the collaborative controller to execute a fast business flow. The collaborative controller aggregates the experience trajectories reported by each of the heterogeneous intelligent entities, performs global feature mining and value assessment based on a centralized value assessment network with an embedded global feature cross-attention mechanism, and generates updated policy model parameters to execute the slow management flow. The collaborative controller monitors the distribution status of environmental characteristics of the cyber-physical system. When it is determined that the preset environmental drift triggering condition is met, it triggers the action of sending the updated strategy model parameters to each of the heterogeneous intelligent agents. The system controls the heterogeneous intelligent agent to receive the updated policy model parameters. Through a built-in spatiotemporal semantic security audit mechanism, it verifies the semantic coherence between the historical benchmark features on which the policy model parameters were generated and the current instantaneous physical features of the heterogeneous intelligent agent. If coherence is determined, an online hot update is performed. If incoherence is determined, the parameter is rejected and the preset security baseline policy is maintained.

2. The method according to claim 1, characterized in that, The process of controlling each of the heterogeneous intelligent agents, based on a local policy network with embedded time-series memory and gating update modules, constructing a belief state vector based on real-time collected local time-series observation data, and independently outputting real-time control actions accordingly, includes: The current local observation data sequence collected by the heterogeneous intelligent agent is input into the local policy network; The temporal memory and gating update module filters out some observable temporal noise in the physical environment, maps discrete and delayed local observation data into a continuous belief state vector, and outputs the real-time control action based on the belief state vector in a resource-constrained environment.

3. The method according to claim 1, characterized in that, The process of global feature mining and value assessment based on a centralized value assessment network with an embedded global feature cross-attention mechanism includes: Map the asymmetric state features reported by different heterogeneous intelligent agents into query vectors, key vectors, and value vectors; The global feature cross-attention mechanism is used to calculate the attention weights between state features of different intelligent agents, mine the temporal dependence and cross-domain causal relationship of the global state, and generate a context feature vector with global dynamic insight capability. Based on the context feature vector, a long-term value assessment is performed on the joint actions in the experience trajectory to guide the generation of the updated policy model parameters.

4. The method according to claim 1, characterized in that, The determination that the preset environmental drift triggering condition is met specifically includes any of the following determination methods: Method 1: Obtain the average joint reward actually acquired by the cyber-physical system within a preset long observation window, and calculate the cumulative performance deviation integral between the average joint reward and the expected optimal value. When the cumulative performance deviation integral exceeds a preset performance deficit threshold, it is determined that the environmental drift triggering condition is met. Method 2: Calculate the relative entropy or cosine similarity between the state space distribution in the current experience trajectory and the historical state space distribution when the policy model parameters were issued last time; when the relative entropy or cosine similarity is greater than the preset structural mutation threshold, it is determined that the environmental drift triggering condition is met.

5. The method according to claim 1, characterized in that, The process of verifying the semantic coherence between the historical benchmark features on which the strategy model parameters are based and the current instantaneous physical features of the heterogeneous intelligent agent, through the built-in spatiotemporal semantic security audit mechanism, includes: Extract the timestamp tags and macro-semantic feature vectors carried in the updated strategy model parameters sent from the cloud; Obtain the real-time microscopic features captured by the current sensors of the heterogeneous intelligent subject and map them into the current semantic feature vector; Calculate the similarity between the macro-semantic feature vector and the current semantic feature vector; If the timestamp tag does not exceed the set physical security validity period and the similarity is greater than the preset security coherence threshold, it is determined to be semantically coherent, and the updated policy model parameters are hot-loaded into the local policy network for online switching; otherwise, it is determined to be semantically incoherent, the update instruction is intercepted and the preset security baseline policy is rolled back to block the control risks caused by asynchronous latency.

6. A heterogeneous execution device with a security audit mechanism, characterized in that, Deployment across the edge layers of physical or cyberspace, including: The temporal belief policy module is used to input local temporal observation data into the gated loop unit embedded in the local policy network, filter temporal noise based on the gated update mechanism, map discrete local observation data into continuous belief state vectors, and output control actions based on the belief state vectors. A real-time control module is used to execute the control actions in both physical and information spaces and asynchronously report the experience trajectory; The logic review unit, connected in series between the external receiving port and the time-series belief strategy module, is used to extract the timestamp tag and macro-semantic feature vector carried in the updated strategy model parameters sent from the cloud; obtain the real-time micro-features captured by the current sensors of the heterogeneous intelligent agent and map them into the current semantic feature vector; and calculate the similarity between the macro-semantic feature vector and the current semantic feature vector. The model hot update unit is used to confirm semantic coherence and hot-load the updated policy model parameters into the local policy network for online switching when the logic review unit determines that the timestamp tag has not exceeded the set physical security validity period and the similarity is greater than the preset security coherence threshold.

7. A dual-closed-loop collaborative evolution computing platform for heterogeneous intelligent agents, characterized in that, The platform includes: At least one evolved cooperative controller, and multiple heterogeneous execution devices as described in claim 6 deployed across different spatial levels, each of the heterogeneous execution devices serving as the physical carrier of the heterogeneous intelligent entity; the evolved cooperative controller is deployed on the cyberspace core side of the cyber-physical system, including: The global state aggregation module is used to receive the experience trajectories asynchronously reported by various heterogeneous intelligent agents; The global value assessment module is used to map the asymmetric state features reported by different heterogeneous intelligent agents into query vectors, key vectors, and value vectors; calculate the attention weights between the state features of different intelligent agents using a global feature cross-attention mechanism, mine the temporal dependencies and cross-domain causal relationships of the global state, and generate a context feature vector with global dynamic insight capabilities; based on the context feature vector, perform long-term value assessment on the joint actions in the experience trajectory to guide the generation of the updated policy model parameters; The version triggering and distribution module is used to obtain the average joint reward actually obtained by the cyber-physical system within a preset long observation window, and calculate the cumulative performance deviation integral between the average joint reward and the expected optimal value. When the cumulative performance deviation integral exceeds the preset performance deficit threshold, it is determined that the environmental drift triggering condition is met, and the updated strategy model parameters are distributed to each of the heterogeneous intelligent agents. As a specific physical scenario application, the fully heterogeneous intelligent entity includes a data synchronization entity deployed in the physical space for processing local state observation, and a computing orchestration entity deployed at the edge layer of the information space for processing computing resource scheduling; The evolved collaborative controller is connected to each of the heterogeneous execution devices via a network to enable asynchronous interaction between business flows and management flows.

8. An electronic device, characterized in that, include: A processor and a memory storing a computer program, wherein the processor, when executing the computer program, implements the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that: It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images