A distributed training synchronization control method, device and equipment for reinforcement learning agent-simulation environment step frequency mismatch

Abstract

The application provides a distributed training synchronization control method, device and equipment for the step frequency mismatch of a reinforcement learning intelligent agent-simulation environment, a global collaborative control server receives multiple heterogeneous simulation data, maps the multiple heterogeneous simulation data into a standard state vector, and constructs a decision information age index in combination with an observation time, an action transmission delay and a policy output change; a local advancing clock of each simulation engine node is collected, and a step frequency mismatch type is identified in combination with a clock synchronization deviation; a dynamic causal dependence graph is constructed based on the standard state vector, and an exploration risk index is calculated; when a risk trigger condition is met, an action shielding, freezing, state rollback and safe alternative action injection are performed on a related local entity, and a processing result is written back to a standard state space; subsequently, physical simulation regulation and / or training value regulation are performed according to the mismatch type, and a logical time advancing signal is broadcasted when a global advancing condition is met.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of distributed reinforcement learning, distributed parallel numerical simulation, intelligent control, and collaborative training of complex systems, and in particular to a distributed training synchronization control method, device, and equipment for addressing step-frequency mismatch between reinforcement learning agents and simulation environments. Background Technology

[0002] Distributed reinforcement learning training typically requires connecting multiple simulation environment nodes, agent decision nodes, and training computation nodes to run in the same closed-loop system. Due to differences in data formats, coordinate representations, state fields, step sizes, and computational loads among different simulation engines, the system needs to continuously perform operations such as state alignment, action issuance, simulation advancement, and training updates during the training process.

[0003] Existing distributed training synchronization methods often focus on network clock synchronization, communication delay control, or fixed-step progression. They typically determine synchronization based solely on node clock differences or message arrival times, failing to incorporate state changes after standardization of heterogeneous simulation data, agent policy output changes, action activation delays, and training value sensitivity into the same synchronization control criteria. Consequently, when the agent's decision-making cycle and the simulation environment's progression cycle are inconsistent, the system struggles to accurately determine the source of the mismatch and distinguish whether adjustments to simulation progression, training distribution, or merely state processing for relevant local entities should be implemented.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This invention discloses a distributed training synchronization control method, device, and equipment for addressing the mismatch between the action frequency of a reinforcement learning agent and the simulation environment. It aims to solve the problem of lacking unified synchronization control over state timeliness, node advancement deviation, entity constraint relationships, and training update rhythm when the action frequency of the agent's decision output is inconsistent with that of the simulation environment in heterogeneous distributed reinforcement learning training.

[0006] The first embodiment of the present invention provides a distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment, comprising: The system receives raw situation data streams from multiple heterogeneous simulation engines, converts the raw situation data streams into standard state space vectors, and constructs an age index for agent decision information based on the standard state space vectors, agent observation time, end-to-end action delay, and policy network output distribution changes. Collect the local propulsion clock of each simulation engine node, and identify the current step frequency mismatch type by combining the decision information age index; Based on the standard state space vector, a causal dependency graph between entities is constructed, and the risk is calculated and explored according to the entity's position, current action, and constraint boundary. When the exploration risk meets the preset triggering conditions, the local causal connected subgraph related to the risk action is extracted from the causal dependency graph. Only the entities in the subgraph are subject to action masking, partial freezing, state rollback and safe alternative action injection. The entity state after rollback and alternative action injection is written back to the standard state space vector. Based on the standard state space vector written back and the standard state space vector when the preset triggering condition is not triggered, physical fidelity control and / or training value control are performed according to the step frequency mismatch type; when the local processing is completed and the global advancement condition is met, a global logical time advancement signal is broadcast.

[0007] The second embodiment of the present invention provides a distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, comprising: The timeliness information construction unit is used to receive the raw situation data streams sent by multiple heterogeneous simulation engines, convert the raw situation data streams into standard state space vectors, and construct an age index for agent decision information based on the standard state space vectors, agent observation time, end-to-end delay of actions, and changes in the output distribution of the policy network. The step frequency mismatch identification unit is used to collect the local propulsion clock of each simulation engine node and identify the current step frequency mismatch type by combining the decision information age index; The risk assessment unit is used to construct a causal dependency graph between entities based on the standard state space vector, and to calculate and explore risks based on entity positions, current actions, and constraint boundaries. The safety action injection unit is used to extract the local causal connected subgraph related to the risk action from the causal dependency graph when the exploration risk meets the preset triggering conditions, and only perform action masking, local freezing, state rollback and safety alternative action injection on entities in the subgraph, and write the entity state after rollback and alternative action injection back to the standard state space vector. The dual-ring synchronous control unit is used to perform physical fidelity control and / or training value control according to the step frequency mismatch type based on the standard state space vector written back and the standard state space vector when the preset triggering condition is not triggered; when the local processing is completed and the global advancement condition is met, it broadcasts a global logical time advancement signal.

[0008] The third embodiment of the present invention provides a distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement the distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in any of the above embodiments.

[0009] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a distributed training synchronization control method for step frequency mismatch of reinforcement learning agent-simulation environment as described in any of the above embodiments.

[0010] Beneficial effects: By uniformly mapping the raw situational data output from multiple heterogeneous simulation engines to a standard state space, and constructing decision information age indicators, node advancement clock deviation, and step frequency mismatch type determination mechanisms on this standard state space, the timeliness of agent decision outputs, the advancement rhythm of the simulation environment, and the synchronization deviation between different simulation nodes can be measured and identified under the same control framework. Thus, the system can avoid the misjudgment problems caused by relying solely on a single clock or single state observation for synchronization judgment, improving state consistency, synchronization judgment accuracy, and overall operational stability during distributed reinforcement learning training.

[0011] By identifying local entity sets related to risky actions based on dynamic causal dependency graphs, and performing action masking, local freezing, temporal rollback, and safe alternative action injection on these local entity sets, combined with inner-loop physical simulation step size control and outer-loop training value update control, the system can perform targeted corrections on affected local entities when local step frequency mismatches or risk exploration actions occur, without requiring a unified pause or overall rollback of all simulation nodes. This reduces the disturbance of local abnormal advancements to the global training process, improving the continuity, safety, and reliability of simulation execution and multi-agent training convergence. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment provided in the first embodiment of the present invention. Figure 2 This is a schematic diagram of a distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, provided in the second embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0015] This invention discloses a distributed training synchronization control method, device, and equipment for addressing the mismatch between the action frequency of a reinforcement learning agent and the simulation environment. It aims to solve the problem of lacking unified synchronization control over state timeliness, node advancement deviation, entity constraint relationships, and training update rhythm when the action frequency of the agent's decision output is inconsistent with that of the simulation environment in heterogeneous distributed reinforcement learning training.

[0016] The first embodiment of the present invention provides a distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment, which can be executed by a global collaborative control server in the following steps: S101, receives raw situation data streams sent by multiple heterogeneous simulation engines, converts the raw situation data streams into standard state space vectors, and constructs an age index for agent decision information based on the standard state space vectors, agent observation time, end-to-end delay of actions, and changes in the output distribution of the policy network. The global collaborative management server receives raw situation data streams uploaded by multiple simulation engine nodes. These raw situation data streams include the spatial position, velocity, attitude, dynamic loads, task status, environmental observations, and corresponding timestamps for each entity. Because different simulation engines have different data fields, coordinate references, sampling periods, and state representation methods, the global collaborative management server first performs timestamp verification, field parsing, coordinate reference unification, and dimension normalization on each raw situation data stream. Then, based on a preset state mapping matrix, it maps the raw situation data to a unified standard state space, obtaining a standard state vector.

[0017] in, This represents the standard state vector formed at logical time t. This represents the original situational state vector received at the same logical moment. This represents the state mapping matrix used for field alignment, coordinate transformation, dimension normalization, and state filtering. Through this processing, the state information output by different simulation engines is converted into standard state data with unified dimensions, unified semantics, and unified temporal identification, enabling subsequent mismatch identification, causal relationship updates, risk assessment, partial rollback, and synchronization control to be performed based on the same state foundation.

[0018] After forming the standard state vector, the global collaborative management server constructs a decision information age index F(j,k) for the decision output of the j-th agent in the k-th synchronization control cycle. This index characterizes the degree of timeliness deviation of the agent's current decision action relative to changes in environmental observation, action transmission, and policy output. The index can be calculated using the following formula:

[0019]

[0020] Wherein, F(j,k) represents the age index of the decision information of the j-th agent in the k-th synchronization control cycle; This indicates the current logical time of the global collaborative management server. This represents the logical time corresponding to the last valid environmental observation obtained by the agent. This represents the end-to-end latency between the generation and transmission of the agent's decision-making action to the corresponding simulation engine's reception and preparation for execution. and These represent the standard state vectors corresponding to the most recent valid observation state and the current observation state, respectively; and These represent the historical policy output distribution and the current policy output distribution obtained based on the two standard state vectors mentioned above, respectively. The Kullback-Leibler divergence between the output distributions of the two policies is represented by α, β, and λ, which represent the weighting coefficients corresponding to observation lag, action transmission delay, and policy change difference, respectively, and their sum is 1. For continuous action spaces, the policy output distribution can be represented by Gaussian distribution parameters; for discrete action spaces, the policy output distribution can be represented by a classification probability vector. Therefore, the standard state vector is not only used as a unified state input for subsequent processing, but its corresponding observation time, current state, and policy output difference are also incorporated into the decision timeliness measurement. S102, Collect the local propulsion clock of each simulation engine node, and identify the current step frequency mismatch type by combining the decision information age index; After completing the standard state vector construction and the calculation of the decision information age index, the global collaborative management server periodically receives the local propulsion clocks reported by each physical simulation engine node according to the system baseline step size ΔT. The local propulsion clock reported by the i-th physical simulation engine node within the k-th synchronization control cycle is denoted as... To obtain the relative progress deviation between different simulation engine nodes, the global collaborative management server uses the clock of the slowest-progressing node among all current physics simulation engine nodes as a comparison benchmark to calculate the clock consistency synchronization deviation of the i-th physics simulation engine node:

[0021]

[0022]

[0023] in, This represents the clock deviation of the i-th physics simulation engine node relative to the slowest advancing node during the k-th synchronization control cycle. This represents the local propagation clock of the i-th physics simulation engine node during this synchronization control period. This represents the local propagation clock of the p-th physics simulation engine node within this synchronization control cycle, where p is the index used to traverse all physics simulation engine nodes. This represents the minimum local advance clock value among all physics simulation engine nodes. This calculation can be used to determine if any simulation engine nodes in the current system are advancing too fast relative to the slowest node.

[0024] After obtaining the clock consistency synchronization deviation of each physical simulation engine node, the global collaborative management server further calls the aforementioned decision information age index F(j,k), and combines it with the preset clock synchronization deviation allowable threshold δ and information age tolerance coefficient. And the system reference step size ΔT, to determine the current step frequency mismatch type. Here, δ represents the maximum allowable node advance clock deviation. This represents the threshold for determining the age index of decision information. When the maximum clock consistency synchronization deviation is greater than δ, and the age index of the decision information of each agent is not greater than... At that time, it is determined that the current main manifestation is simulation load mismatch; when there is at least one agent whose decision information age index is greater than When the above conditions are met, it is determined that there is a decision overload mismatch; when both of the above conditions are met simultaneously, it is determined that there is a mixed mismatch. The above determination results serve as the basis for subsequent control method selection, enabling the system to distinguish whether the mismatch mainly originates from the simulation propulsion side, the decision output side, or both.

[0025] S103, Construct a causal dependency graph between entities based on the standard state space vector, and calculate and explore risks based on entity position, current action and constraint boundary; After obtaining the step frequency mismatch type, the global collaborative management server constructs a dynamic causal dependency graph based on the standard state vector to describe the constraint relationships between entities:

[0026]

[0027]

[0028] in, This represents a causal dependency graph between entities, where V represents the set of entity nodes in the current simulation scene, and E represents the set of directed edges between entity nodes. Entity nodes can include physical simulation entities, intelligent agent entities, and environmental constraint objects related to state progression; directed edges represent the direct influence relationships between two entities formed due to observation relationships, communication relationships, dynamic interaction relationships, or spatial constraint relationships. The global collaborative management server reads the position, velocity, attitude, sensor range, action range, and boundary constraint information of each entity from the standard state vector, and updates the directed edges based on the distance between entities, observation reachability, overlap of action ranges, and dynamic contact conditions. To reduce the graph structure update overhead in large-scale entity scenes, a dynamic spatial grid or hash index can be constructed in memory, allowing the server to perform boundary envelope condition judgments only on entities in the same grid or adjacent grids, and combining this with hysteresis buffer distance control to control the establishment and disconnection of edges.

[0029] After updating the dynamic causal dependency graph, the global collaborative management server calculates exploration risk indicators for the decision action that the agent intends to execute. For the action that the j-th agent intends to execute at its current position, the server predicts the entity position in the next propulsion cycle based on the system baseline step size ΔT, and calculates the shortest Euclidean distance from the predicted position to the constraint boundary set. In the formula, This indicates the exploration of the risk field function value; This indicates the current location of the q-th physical entity, where q is the index of the physical entity. This represents the agent's decision action that is currently pending execution; The set of constraint boundaries is represented by ; dist represents the shortest Euclidean distance from the predicted location to the set of constraint boundaries. ω represents the collision avoidance safety radius; ω represents the penalty adjustment coefficient. The exploration risk index is zero when the predicted position maintains a sufficient distance from the constraint boundary; as the predicted position approaches the constraint boundary, the exploration risk index increases as the distance decreases. If the exploration risk index is greater than zero, or if a preset risk trigger condition is met, the agent entity that generated the risk index is designated as the source node for subsequent local processing.

[0030] S104, When the exploration risk meets the preset triggering condition, extract the local causal connected subgraph related to the risk action from the causal dependency graph, perform action masking, local freezing, state rollback and safe alternative action injection only on the entities in the subgraph, and write the entity state after rollback and alternative action injection back to the standard state space vector. When a risk indicator meets the triggering condition, the global collaborative management server does not perform a unified rollback on all simulation nodes. Instead, it takes the agent entity that generated the risk indicator as the source node and performs a local search along the directed edges in the dynamic causal dependency graph to determine the set of local entities that have direct or indirect constraints with the source node. This set of local entities is denoted as... Furthermore, this local entity set is a subset of the entity node set V. This local entity set covers entities that have a state-affected relationship with the risk action, but excludes other entities unrelated to the risk action. To keep the processing scope controllable, upper limits can be set on the retrieval depth, the number of entities, or the type of connection edges, ensuring that the local processing remains within a range that is actually related to the risk action.

[0031]

[0032] After determining the local entity set, the global collaborative management server first blocks risky exploration actions, preventing them from being sent to the corresponding physics simulation engine for execution. Simultaneously, the server sends a local freeze command to the simulation engines of the entities within the local entity set, pausing the relevant entities' progress within the current synchronization control cycle. Subsequently, based on cached historical states, the server rolls back the state of each entity in the local entity set along the timeline to the local rollback point before the risky action occurred. This indicates the point in time when the target logic for a partial rollback is executed. ΔT represents the current global logical time, and ΔT represents the system baseline step size. Through this rollback process, the entity states in the local entity set are restored to their state before the risky action was involved in the simulation.

[0033] After completing the partial time-series rollback, the global collaborative management server generates a safety alternative action corresponding to the original exploration action based on preset safety action generation rules, expert rules, or a lightweight safety controller. This safety alternative action is denoted as... The data is injected into the corresponding physics simulation engine to replace the blocked exploration actions, allowing the relevant entities to continue advancing according to the actions that meet the constraints after the freeze is lifted. For other relevant entities in the local entity set besides the risk source entity, the server can restore their normal advancement based on their rolled-back state and the advancement clock of the corresponding simulation engine; for entities not belonging to the local entity set, they are allowed to continue advancing at their original pace without intervention. After the local rollback and safe alternative action injection are completed, the global collaborative management server rewrites the rolled-back state, safe alternative action execution state, and advancement state after the freeze is lifted of each entity in the local entity set into the standard state space, forming an updated standard state vector. This updated standard state vector serves as the input for subsequent synchronization control and global advancement judgment; if no exploration risk is triggered within the current synchronization control cycle, subsequent processing continues directly based on the current standard state vector.

[0034] S105, based on the standard state space vector written back and the standard state space vector when the preset triggering condition is not triggered, perform physical fidelity control and / or training value control according to the step frequency mismatch type; when the local processing is completed and the global advancement condition is met, broadcast the global logical time advancement signal.

[0035] After obtaining the step frequency mismatch type, the global collaborative management server executes dual-loop synchronization management oriented towards training value, based on the standard state vector written back to the standard state space after local processing, or based on the current standard state vector when no local risk processing is triggered. The dual-loop synchronization management includes inner-loop control oriented towards the pace of physical simulation advancement and outer-loop control oriented towards the pace of agent training updates. The inner-loop control is mainly used to handle simulation load mismatch caused by inconsistent advancement step sizes between simulation engine nodes, while the outer-loop control is mainly used to handle decision overload mismatch caused by timeliness deviations in agent decision outputs relative to the current standard state. When a mixed mismatch is determined, the global collaborative management server can simultaneously activate both inner-loop and outer-loop control.

[0036] When the current system is in a simulated load mismatch state, the global collaborative management server calculates the elastic simulation propulsion step size adopted by the fast node in the next synchronization control cycle based on the clock consistency synchronization deviation, allowable deviation threshold, system reference step size, step size stretching sensitivity factor, spectral radius of the dynamic system Jacobian matrix, and integral stability constant of the h-th fast node in the k-th synchronization control cycle.

[0037] in, This represents the elastic simulation advancement step size used by the h-th fast node simulation engine at the (k+1)th logical time, where h is the index of the fast node simulation engine. This indicates the clock consistency synchronization error of the fast node simulation engine; Indicates the step-size stretching sensitivity factor; The Jacobian matrix represents the current physical differential equation of state; ρ represents the spectral radius operator. This represents the integral stability constant. Through this process, when the propulsion deviation of a fast node relative to the slowest node exceeds the allowable range, its subsequent propulsion step size is limited to the range determined by the dynamic stability condition.

[0038] After determining the elastic simulation propulsion step size, the corresponding physical simulation engine uses a fourth-order Runge-Kutta integral method to execute state propulsion. For the dynamic equation y'=f(t,y), at a step size of... In this case, the intermediate slope and the next integration state can be calculated sequentially as follows:

[0039] in, and These represent the current integration time and the current integration state, respectively; f represents the physical differential state equation. , , , This represents the middle slope of the fourth-order Runge-Kutta integral; y represents the next integral state. Therefore, the inner-loop control not only adjusts the step size of the fast nodes, but also maintains the continuity and numerical stability of state progression through high-precision integration.

[0040] When the current system is in a decision overload mismatch state, the global collaborative management server reads the current standard state vector s and calculates the value gradient norm of the value evaluation network for this state input. Simultaneously, it reads the backlog depth of the communication buffer queue corresponding to the m-th agent. Then, based on the preset buffer overflow threshold, value gradient weights, and the maximum allowed frame skipping limit, the server calculates the adaptive advancement frame skipping number for the agent.

[0041]

[0042]

[0043] in, This represents the adaptive advancement frame skipping number for the m-th agent, where m is the agent index; This represents the backlog depth of the network input communication buffer queue for the m-th agent; Indicates the preset buffer overflow threshold; Indicates the value gradient weight factor; Let V represent the gradient of the value network V with respect to the standard state space vector s. The norm of the gradient is represented; Indicates the maximum allowed frame skipping limit; This represents the multi-step target value used for training updates; Indicates the reward at the current moment; Represents the discount factor; Q represents the action value function; This represents the standard state space vector corresponding to the frame skipping process. This represents the candidate action. For intermediate actions missing during frame skipping, the simulation engine uses third-order Hermitian spline interpolation to fill in the missing actions. , , , This represents the third-order Hermitian basis function. In this way, outer-loop regulation can maintain the continuity of the action trajectory while reducing overly frequent decision updates.

[0044] After the inner-loop and outer-loop controls are completed, the global collaborative management server further determines whether the global logical time advancement conditions are met. Specifically, the server determines whether the maximum clock consistency synchronization deviation of all physical simulation engine nodes is not greater than the allowable deviation threshold, and whether the average decision information age of all agents is not greater than the average information age threshold.

[0045]

[0046] Where M represents the total number of agents participating in synchronization control, F(m,k) represents the decision information age index of the m-th agent in the k-th synchronization control cycle, and θ represents the average information age threshold. If local rollback and safe alternative action injection have been triggered in the current synchronization control cycle, the server needs to confirm that the rollback status of the relevant local entity set, the execution status of the safe alternative action, and the updated standard state vector have been written back before judging the above global advancement conditions. Only when the local processing is completed and both of the above conditions are met, the global collaborative control server broadcasts the global logical time advancement signal tick(k+1) to each simulation engine node, driving the system to enter the next synchronization control cycle. Thus, the aforementioned processing forms a closed-loop execution process from standard state access, mismatch identification, risk local processing, dual-loop synchronization control to global advancement.

[0047] To illustrate with a few examples: the internal execution flow and data log-level evolution of the method of the present invention in a highly dynamic and sensitive dogfight scenario of close-range dogfight between two high-speed fighter jets.

[0048] Determine the initial parameter configuration Global reference physical propulsion step size ; Maximum synchronization tolerance threshold for clock skew ; Upper age threshold for decision information ; Regulation coefficient configuration: ; Buffer overflow safety threshold Maximum allowed frame skipping limit Gradient modulation factor .

[0049] System execution flow and log data chain: as the system logic progresses to global time... (Corresponding promotion cycle) )hour The global collaborative management server reads system indicator data through a distributed telemetry interface, and the real-time data status is recorded in the table below:

[0050] Policy divergence and timeliness calculation: The red fighter jet is currently in a high-value, sensitive state, tightly locked by the blue fighter jet's radar and undergoing a severe overload escape maneuver. Its deep learning computing nodes have performed deep forward inference calculations. In the current state, the policy network (Gaussian distribution) has reached the [missing value] 1 / [missing value]. Output With the Output at time A significant drift occurred.

[0051] The calculated KL divergence of the current policy change is: .

[0052] The age of agent 1's decision information is calculated according to the formula. : Agent 2 (in a stable flight state, KL divergence is close to 0): its decision information age .

[0053] Online identification of mismatch states Calculate the maximum physical clock synchronization deviation: .

[0054] Determine the type of mismatch: due to agent 1's The system immediately determines that the current system is in type B (decision overload mismatch state) because "the decision network of agent 1 has a serious reasoning overload, and the timeliness of decision-making is at the danger boundary of degradation".

[0055] Outer-loop adaptive control strategy execution (value gradient modulation) The global control server reads the output state value gradient of the Critic network corresponding to Red Agent 1 through backpropagation: its gradient norm 2 is calculated as follows. .

[0056] Substitute into the exponential adaptive frame skipping decision formula: ; Control Result: Although the Red Agent currently has up to 72 frames of data packets piling up in its message input buffer, it is on the critical edge of a life-or-death battle (high value gradient). The outer loop controller adaptively determines its frame skipping number. (That is, never skip frames).

[0057] Inner-loop auxiliary suspension coordination: To prevent outdated actions from disrupting strategy convergence, the global control server performs "lightweight suspension coordination" to suspend the physical propulsion of the AFSIM flight engine. (Suspend without stretching step size) The idle communication bus bandwidth is exclusively allocated to the red team's computing nodes, enabling their inference actions to take effect quickly and refreshing their input queues. This strongly protects the learning accuracy of the control strategy network during the extremely sensitive period and prevents it from diverging due to delayed actions.

[0058] When an agent's decision leads to a physical collision or out-of-bounds exploration, a local spatiotemporal state rollback execution scheme is implemented.

[0059] Physical scenario and parameter state: In large-scale maritime collaborative exercises, a directed causal graph is included: V = {A1 (Red Force's No. 1 intelligent agent UAV), A2 (Red Force's No. 2 intelligent agent UAV), S1 (Red Force's communication relay), B1 (Blue Force's surveillance submarine), B2 (Blue Force's radar destroyer)} The topology of the directed dependency relationship between real-time physical detection range and guidance link establishment includes the following arcs: Directed edge: ( pass Relay communication); Directed edge: ( Currently in contact with the Blue Team destroyer (within the close-range entanglement radar detection area) Directed edge: ( (Using the same communication node) The submarine is extremely far from other entities and has no physical, detection, or communication interactions with them (its nodes are not connected to other entities).

[0060] The execution process and state transitions, in the process of advancing to (time Conflict detection and risk prediction: Red Team No. 1 UAV Intelligent Agent At this point, a set of exploration posture actions were output. .

[0061] The global management server calculates the action to be executed through multi-step prediction. Afterwards, the drone will enter the Blue Team destroyer. Anti-space killing no-fly zone The shortest prediction distance is Smaller than the collision avoidance safety radius .

[0062] Calculate and explore the risk field function:

[0063] If the action is deemed to be a dangerous overstepping of boundaries, a partial reversal must be performed.

[0064] Extraction of directed causal connected subgraphs: The management server will control conflicting entities Set as traversing the source node, in the current causal graph Perform depth-first traversal (DFS) in the middle. The extracted coherent controlled connected subgraph entity set is as follows (in Because of the shared relay communication node, and (There is coupling, and it is automatically included in the causal ripple chain). Independent incoherent entities Not here In a connected subgraph; Local spatiotemporal rollback and motion safety shielding The control server completely discards shielding from the input ends of the AFSIM engine and VR-Forces engine. Dangerous action instructions generated .

[0065] Global collaborative management server to subgraph All host nodes within the system were sent local forced freeze and time-space reset commands, Rewriting and rolling back the physical and control states to the previous moment ; Meanwhile, the blue team submarines, belonging to the incoherent set... The physics simulation engine process it resides in does not perform any suspension or rollback; its clock continues to advance normally until... above.

[0066] Safety actions are injected and de-frozen in parallel: Retrieve the local lightweight safety controller (with trajectory collision avoidance correction rules) and inject a safety alternative action:

[0067] Will Write to AFSIM air engine. Engine unfreezing collection. Restore the state of all entities within the simulation and move it forward.

[0068] Operational results: The drone successfully bypassed the no-fly zone, continued high-speed concurrent advancement in unrelated areas, and the entire rollback overhead was less than that of the global snapshot. Furthermore, the probability of a global distributed deadlock triggered by boundary collisions is reduced to zero.

[0069] Please see Figure 2 The second embodiment of the present invention provides a distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, comprising: The timeliness information construction unit is used to receive the raw situation data streams sent by multiple heterogeneous simulation engines, convert the raw situation data streams into standard state space vectors, and construct an age index for agent decision information based on the standard state space vectors, agent observation time, end-to-end delay of actions, and changes in the output distribution of the policy network. The step frequency mismatch identification unit is used to collect the local propulsion clock of each simulation engine node and identify the current step frequency mismatch type by combining the decision information age index; The risk assessment unit is used to construct a causal dependency graph between entities based on the standard state space vector, and to calculate and explore risks based on entity positions, current actions, and constraint boundaries. The safety action injection unit is used to extract the local causal connected subgraph related to the risk action from the causal dependency graph when the exploration risk meets the preset triggering conditions, and only perform action masking, local freezing, state rollback and safety alternative action injection on entities in the subgraph, and write the entity state after rollback and alternative action injection back to the standard state space vector. The dual-ring synchronous control unit is used to perform physical fidelity control and / or training value control according to the step frequency mismatch type based on the standard state space vector written back and the standard state space vector when the preset triggering condition is not triggered; when the local processing is completed and the global advancement condition is met, it broadcasts a global logical time advancement signal.

[0070] The third embodiment of the present invention provides a distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to implement the distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in any of the above embodiments.

[0071] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement a distributed training synchronization control method for step frequency mismatch of reinforcement learning agent-simulation environment as described in any of the above embodiments.

[0072] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment, characterized in that, include: The system receives raw situation data streams from multiple heterogeneous simulation engines, converts the raw situation data streams into standard state space vectors, and constructs an age index for agent decision information based on the standard state space vectors, agent observation time, end-to-end action delay, and policy network output distribution changes. Collect the local propulsion clock of each simulation engine node, and identify the current step frequency mismatch type by combining the decision information age index; Based on the standard state space vector, a causal dependency graph between entities is constructed, and the risk is calculated and explored according to the entity's position, current action, and constraint boundary. When the exploration risk meets the preset triggering conditions, the local causal connected subgraph related to the risk action is extracted from the causal dependency graph. Only the entities in the subgraph are subject to action masking, partial freezing, state rollback and safe alternative action injection. The entity state after rollback and alternative action injection is written back to the standard state space vector. Based on the standard state space vector written back and the standard state space vector when the preset triggering condition is not triggered, physical fidelity control and / or training value control are performed according to the step frequency mismatch type. When local processing is completed and the global advancement conditions are met, a global logical time advancement signal is broadcast.

2. The distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in claim 1, characterized in that, The standard state space vector is obtained by uniformly transforming the original situation data through a multi-dimensional state mapping matrix: in, This represents the standard state-space vector at the t-th logical time. This represents the initial state of affairs at that logical moment. The multidimensional state mapping matrix is ​​represented; the standard state space vector includes at least entity position, velocity, attitude, load state, and survival state; the decision information age index is: Where j and k represent the j-th agent and the k-th logical moment, respectively; Indicates the global coordination logic time. This indicates the logical time when the agent received the previous frame of valid environmental observations. This represents the end-to-end delay from the output of the agent's previous decision to its receipt and activation by the simulation engine; and These represent the standard state space vectors corresponding to the valid environmental observations of the previous frame and the current observation, respectively. and These represent the historical policy network action probability distribution and the current policy network action probability distribution, respectively, based on the output of the two standard state space vectors mentioned above. Let represent the Kullback-Leibler divergence between the probability distributions of actions in the two policy networks; α, β, and λ represent the observation time difference weight, action delay weight, and policy change weight, respectively, and the sum of the three is 1.

3. The distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in claim 1, characterized in that, The clock consistency synchronization error of each simulation engine node is: in, This represents the clock consistency synchronization error of the i-th simulation engine node at the k-th logical moment. This represents the local propagation clock of the i-th simulation engine node. This represents the minimum local advance clock obtained by traversing all simulation engine node indices p; i and p are both simulation engine node indices, and k is the logical time index. The current step frequency mismatch type is determined according to the following conditions: Type A: When the condition is met Furthermore, it applies to any reinforcement learning agent across the entire system. At that time, it was determined that the mismatch was caused by fluctuations in the computational load at the underlying level of the simulation node's host machine; Type B: When the condition is met, there exists at least one agent. satisfy At that time, the mismatch was determined to be caused by a sudden increase in inference latency in the deep reinforcement learning policy network; Type C: When all of the above conditions are met, the system is determined to be in a state of severe mixed mismatch. Where δ represents the allowable threshold for clock synchronization deviation. ΔT represents the age tolerance coefficient of decision information, and ΔT represents the baseline step size of the distributed simulation network.

4. The distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in claim 1, characterized in that, The causal dependency graph is a dynamic directed acyclic graph, and its expression is: in, The graph represents the causal dependency between entities, where V represents the set of entity nodes and E represents the set of directed edges. The entity nodes include physical entity nodes and agent entity nodes. The directed edges represent direct dynamic coupling links or information coupling links formed by physical observation, communication, collision constraints, or information interaction. The causal dependency graph is updated based on the entity position, velocity, attitude, and state attributes in the standard state space vector and is maintained by a dynamic space hash grid and hysteresis buffer boundaries.

5. A distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in claim 1, characterized in that, The exploration risk is determined by the following formula: in, This indicates the exploration of the risk field function value; This indicates the current location of the q-th physical entity, where q is the index of the physical entity. This represents the agent's decision action that is currently pending execution; The set of constraint boundaries is represented by ; dist represents the shortest Euclidean distance from the predicted location to the set of constraint boundaries. ω represents the collision avoidance safety radius; ω represents the penalty adjustment coefficient, which satisfies the preset triggering condition when the exploration risk field function value is greater than 0.

6. A distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in claim 5, characterized in that, When the risk exploration meets the preset triggering conditions, a local causal connected subgraph related to the risk action is extracted from the causal dependency graph. Action masking, partial freezing, state rollback, and safe alternative action injection are performed only on entities within this subgraph. The entity state after rollback and alternative action injection is then written back to the standard state space vector. Specifically: A depth-first traversal is performed on the causal dependency graph to extract the set of entity nodes within the local causal connected subgraph. Local rollback and freeze commands are then sent to this set of entity nodes to roll back their state. The expression for this is: in, This indicates the point in time when the target logic for a partial rollback is executed. ΔT represents the global collaborative logic time, and ΔT represents the baseline step size. Normal progress is maintained for incoherent entity nodes. After injecting a safety alternative action generated by the safety controller into the local causal connected subgraph, the freeze is lifted, and the updated state of the entities in the subgraph is written back to the standard state space vector.

7. A distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment as described in claim 1, characterized in that, When a simulated load mismatch state is identified, the elastic simulation advance step size of the fast node simulation engine is determined based on physical stability constraints, and its expression is: in, This represents the elastic simulation advancement step size used by the h-th fast node simulation engine at the (k+1)th logical time, where h is the index of the fast node simulation engine. This indicates the clock consistency synchronization error of the fast node simulation engine; This represents the step-size stretching sensitivity factor. The Jacobian matrix represents the current physical differential equation of state; ρ represents the spectral radius operator. The integral stability constant is represented by the fast node simulation engine, which employs a fourth-order Runge-Kutta integral propulsion with the elastic simulation propulsion step size as the step size. in, and These represent the current integration time and the current integration state, respectively; f represents the physical differential state equation. , , , This represents the middle slope of the fourth-order Runge-Kutta integral; y represents the next integration state. This represents the elastic simulation propulsion step size used by the j-th fast node simulation engine after step size stretching adjustment.

8. A distributed training synchronization control method for step frequency mismatch between reinforcement learning agent and simulation environment according to claim 1, characterized in that, When a decision overload mismatch state is identified, the number of adaptive hop frames is determined based on the state value gradient and queue backlog depth, and its expression is: in, This represents the adaptive advancement frame skipping number for the m-th agent, where m is the agent index; This represents the backlog depth of the network input communication buffer queue for the m-th agent; Indicates the preset buffer overflow threshold; Indicates the value gradient weight factor; Let V represent the gradient of the value network V with respect to the standard state space vector s. The norm of the gradient is represented; Indicates the maximum allowed frame skipping limit; This indicates rounding down, and updating the training parameters using a multi-step target value based on the adaptively advancing frame skipping number: in, This represents the multi-step target value used for training updates; Indicates the reward at the current moment; Represents the discount factor; Q represents the action value function; This represents the standard state space vector corresponding to the frame skipping process. Candidate actions are represented, and for missing intermediate frame actions, third-order Hermitian spline interpolation is used: in, This represents the padding action for the (t+n)th intermediate frame; and These represent the action at the start of the frame jump and the action at the end of the frame jump, respectively. and These represent the motion change rates at the start time and the end time of the frame jump, respectively; , , , Let represent the third-order Hermitian spline basis functions.

9. A distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, characterized in that, include: The timeliness information construction unit is used to receive the raw situation data streams sent by multiple heterogeneous simulation engines, convert the raw situation data streams into standard state space vectors, and construct an age index for agent decision information based on the standard state space vectors, agent observation time, end-to-end delay of actions, and changes in the output distribution of the policy network. The step frequency mismatch identification unit is used to collect the local propulsion clock of each simulation engine node and identify the current step frequency mismatch type by combining the decision information age index; The risk assessment unit is used to construct a causal dependency graph between entities based on the standard state space vector, and to calculate and explore risks based on entity positions, current actions, and constraint boundaries. The safety action injection unit is used to extract the local causal connected subgraph related to the risk action from the causal dependency graph when the exploration risk meets the preset triggering conditions, and only perform action masking, local freezing, state rollback and safety alternative action injection on entities in the subgraph, and write the entity state after rollback and alternative action injection back to the standard state space vector. The dual-ring synchronous control unit is used to perform physical fidelity control and / or training value control according to the step frequency mismatch type based on the standard state space vector written back and the standard state space vector when the preset triggering condition is not triggered. When local processing is completed and the global advancement conditions are met, a global logical time advancement signal is broadcast.

10. A distributed training synchronization control device for step frequency mismatch between reinforcement learning agent and simulation environment, characterized in that, The method includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a distributed training synchronization control method for step frequency mismatch in reinforcement learning agent-simulation environment as described in any one of claims 1 to 8.

Images