A micro-agent zero-sample navigation method and system for non-stationary fluid disturbance

Abstract

The application relates to a kind of micro intelligent agent zero sample navigation methods and systems for non-stationary fluid disturbance, belong to intelligent autonomous control and deep reinforcement learning technical field.For the high heterogeneity of real fluid environment, non-stationary, and the generalization failure of traditional reinforcement learning, safety deficiency, and the problem of limited edge computing power, an extreme fluid environment safety learning meta-model architecture is proposed: a domain randomization digital twin simulation pool is constructed to generate a variable disturbance flow field; a selective state space perception module is integrated to infer the spatio-temporal characteristics with linear complexity; a stream line guided asymmetric potential energy reward remodeling is used in the training stage; and a physical guided control barrier function is integrated for action safety projection in the deployment stage.The application can realize zero sample generalization safety navigation in unknown abnormal fluid pipe network, significantly reducing the collision rate and edge reasoning delay.

Description

Technical Field

[0001] This invention belongs to the field of intelligent autonomous control and deep reinforcement learning technology, specifically relating to a zero-shot navigation method and system for micro-intelligent agents facing non-stationary fluid disturbances. Background Technology

[0002] Precise navigation of micro-intelligent agents in complex fluid networks (such as precision industrial pipelines and underground micro-fluid networks) has significant engineering application value. However, the real fluid physics environment exhibits extremely high spatiotemporal heterogeneity and non-stationarity, such as periodic pulsating flow and anomalous turbulent disturbances. This highly dynamic environment poses a severe challenge to autonomous navigation algorithms. Existing navigation control frameworks mainly face the following bottlenecks:

[0003] Existing deep reinforcement learning (DRL) algorithms typically rely on the assumption of a stationary Markov decision process (MDP). When faced with out-of-distribution (OOD) scenarios that include unseen anomalous eddies or extreme fluid disturbances, these algorithms are highly susceptible to generalization failures and system paralysis.

[0004] In highly dynamic pipeline networks, micro-agents, limited by their size, can only acquire low-dimensional local observation data, resulting in partial observability issues. Traditional sequence-based reinforcement learning (such as Transformer-based networks) has a quadratic increase in computational complexity, making it difficult to deploy on computationally limited edge detection nodes.

[0005] Existing reinforcement learning inherently possesses trial-and-error exploration characteristics. In physically constrained microscale heterogeneous pipelines, unconstrained exploration actions can easily lead to hard collisions with the pipeline walls, posing potential system safety hazards. Currently, there is a lack of mechanisms to effectively integrate rigid physical constraints with the soft exploration of algorithms.

[0006] Therefore, there is an urgent need in this field for a comprehensive computing framework that can efficiently extract the spatiotemporal features of fluids from low-dimensional observations and has zero-sample generalization capability and rigorous mathematical security guarantees when facing unknown fluid disturbances. Summary of the Invention

[0007] The purpose of this invention is to provide a zero-shot navigation method and system for micro-intelligent agents facing non-stationary fluid disturbances. It proposes an Extreme Fluid Environment Safety Learning Meta-Model (HELM) framework. This invention tightly couples digital twin simulation, sequence inference, and mathematical constraint control into a unified edge intelligence framework, effectively bridging the gap between physical constraints and algorithm security.

[0008] To achieve the above objectives, the technical solution of the present invention is: a zero-shot navigation method for micro-intelligent agents oriented towards non-stationary fluid disturbances, comprising:

[0009] A digital twin simulation environment based on domain randomization is constructed. By parameterizing the boundary of dynamic physical conditions and mapping it to the reinforcement learning state space, a simulation environment pool including non-stationary eddies and fluid disturbances is generated, which serves as the data base for zero-shot policy training.

[0010] A selective state space perception module is integrated into the navigation strategy network of the micro-agent. The low-dimensional continuous temporal observation sequence collected by the micro-agent is input into the selective state space perception module to implicitly infer the spatiotemporal fluid context features of the current environment.

[0011] Based on the inferred spatiotemporal fluid context features, a dual-track optimization and control mechanism is constructed: during the model training phase, the reward is reshaped using an asymmetric potential energy function based on the flow field streamlines to guide the agent's policy optimization.

[0012] During the model deployment and action execution phase, the physical boundary-guided control barrier function is integrated into the action output of the policy network to perform security assessment and forced projection interception on the original actions output by the agent, and output the final security control command.

[0013] Furthermore, the construction process of the domain-randomized digital twin simulation environment includes uniform or truncated normal sampling within a set physical boundary and hydrodynamic parameter range to generate a physical flow field covering extreme disturbance conditions, wherein the environmental parameter vector Follows a domain randomized distribution:

[0014]

[0015] and Let be the minimum and maximum parameter values ​​of the physical boundary, respectively, and u(⋅) be the uniform distribution sampling function.

[0016] Furthermore, the selective state space sensing module's state update and output are represented as follows:

[0017]

[0018]

[0019] in, Let be the hidden state at time t. and It is a parameter matrix that is dynamically generated through a selective mechanism. Given the input observation sequence at time t, The output features at time t and A fixed or learnable parameter matrix.

[0020] Furthermore, the expression for the asymmetric potential energy function based on the streamlines of the flow field is as follows:

[0021]

[0022] in, To reshape the reward value, As a discount factor, Let be the state potential energy function at time t. Let be the state potential energy function at time t+1. These are the weighting coefficients. The velocity vector of the agent. Let the streamline vectors of the local flow field be... This represents the cosine similarity between two vectors.

[0023] Furthermore, the control barrier function defines a safety set. ,in To characterize the continuously differentiable obstacle function representing the distance between the agent and the inner wall of the pipe, the obstacle control condition is as follows:

[0024]

[0025] in, The state of the agent. The original action output by the policy network. The control gain parameter is positive.

[0026] Furthermore, when determining the original action lead to When <0, the safe action is obtained by solving a quadratic programming problem. The objective function of the quadratic programming problem is:

[0027]

[0028] The constraints are:

[0029]

[0030] in, The original action output by the policy network. For barrier function State gradient, For the agent in state Next action The subsequent state transition function.

[0031] Furthermore, the gradient of the asymmetric potential energy function is dynamically adjusted according to the angle between the agent's velocity and the local streamline vector. When the angle is less than 90°, the gradient is positive to reward the agent; when the angle is greater than or equal to 90°, the gradient is negative to punish the agent.

[0032] This invention also provides a zero-shot navigation system for micro-intelligent agents accommodating non-stationary fluid disturbances, comprising:

[0033] The simulation environment construction unit builds a digital twin simulation environment based on domain randomization. By parameterizing the dynamic physical condition boundary and mapping it to the reinforcement learning state space, a simulation environment pool containing non-stationary eddies and fluid disturbances is generated.

[0034] The perception processing unit integrates a selective state space perception module into the navigation strategy network of the micro-agent, inputting the low-dimensional continuous temporal observation sequence collected by the micro-agent into the perception module to implicitly infer the spatiotemporal fluid context features of the current environment.

[0035] The dual-track optimization unit constructs a dual-track optimization and control mechanism based on the inferred spatiotemporal fluid context features. During the model training phase, it uses an asymmetric potential energy function based on the flow field streamline to reshape the reward, thereby guiding the agent to optimize its strategy.

[0036] The safety control unit integrates the physical boundary-guided control barrier function into the action output end of the policy network during the model deployment and action execution phases. It performs safety assessment and forced projection interception on the original actions output by the agent and outputs the final safety control command.

[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0038] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described above.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) Strong zero-shot generalization ability: Through the coupling of domain randomization and implicit spatiotemporal inference, the HELM framework achieves a stable zero-shot navigation success rate on the out-of-distribution (OOD) generalization test set covering a multidimensional continuous parameter space.

[0041] (2) Extremely low deployment computation cost: By using a selective state space model to replace the traditional Transformer, the computational bottleneck of sequence length-dependent inference is broken, and the inference latency of edge nodes is significantly reduced.

[0042] (3) It has a strict mathematical safety bottom line: The dual-track safety mechanism not only guides efficient exploration during the training phase, but also provides an absolute collision protection mechanism that does not rely on network generalization ability during physical deployment through secondary planning, thereby improving the navigation robustness in complex environments. Attached Figure Description

[0043] Figure 1 A schematic diagram of the overall meta-learning architecture (HELM) of the micro intelligent agent navigation system provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the randomization and non-stationary fluid disturbance generation process in a digital twin environment domain.

[0045] Figure 3 This is a schematic diagram of the feature inference structure of a selective state-space perception engine.

[0046] Figure 4 This is a schematic diagram of a dual-track safety action control mechanism based on asymmetric potential energy reshaping and control barrier function. Detailed Implementation

[0047] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0048] This invention provides a zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances, comprising:

[0049] A digital twin simulation environment based on domain randomization is constructed. By parameterizing the boundary of dynamic physical conditions and mapping it to the reinforcement learning state space, a simulation environment pool including non-stationary eddies and fluid disturbances is generated, which serves as the data base for zero-shot policy training.

[0050] A selective state space perception module is integrated into the navigation strategy network of the micro-agent. The low-dimensional continuous temporal observation sequence collected by the micro-agent is input into the selective state space perception module to implicitly infer the spatiotemporal fluid context features of the current environment.

[0051] Based on the inferred spatiotemporal fluid context features, a dual-track optimization and control mechanism is constructed: during the model training phase, the reward is reshaped using an asymmetric potential energy function based on the flow field streamlines to guide the agent's policy optimization.

[0052] During the model deployment and action execution phase, the physical boundary-guided control barrier function is integrated into the action output of the policy network to perform security assessment and forced projection interception on the original actions output by the agent, and output the final security control command.

[0053] The following is a detailed implementation process of the present invention.

[0054] A zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances includes the following steps:

[0055] Step S1: Construct a data-driven domain randomized digital twin simulation pool

[0056] A simulation training environment is constructed by parameterizing the dynamic physical boundaries and mapping them to the state space of reinforcement learning. Let the environment parameter vector be... By introducing a domain randomization distribution within a preset boundary domain :

[0057]

[0058] This environment generates a variable and anomalous fluid disturbance field, ensuring that the model undergoes rich fluid dynamic evolution during the training phase, providing a data foundation for training robust zero-shot strategies.

[0059] Step S2: Efficient edge spatiotemporal inference based on selective state-space model

[0060] To address partial observability issues in complex fluids, a selective state-space awareness engine is integrated into the policy network. This engine integrates low-dimensional continuous observations. Mapping to hidden state In the process of discretizing parameters, the system's state update and output are determined. Represented as:

[0061]

[0062]

[0063] in and It is a parameter matrix dynamically generated through a selective mechanism. This module can implicitly infer complex spatiotemporal fluid contexts while maintaining... It has linear computational complexity and is highly adaptable to the hardware limitations of edge computing devices.

[0064] Step S3: Exploring the initial streamline-guided asymmetric potential energy reward reshaping

[0065] To overcome the extreme reward sparsity problem in the early stages of exploration in complex, long-sequence pipe networks, we introduce local flow field streamline vectors extracted by computational fluid dynamics (CFD). With the agent's velocity vector Constructing a redesigned reward function :

[0066]

[0067] This formula ensures that the agent receives dense reward guidance when it takes advantage of the effective flow field, which greatly improves the convergence speed of optimization.

[0068] Step S4: Physical boot control barrier function during deployment execution

[0069] Deploy a CBF physical isolation layer at the very end of the action network to ensure action security. Define a security set. ,in This represents the relative potential field at a distance from the pipe wall. The original action is given by the reinforcement learning network output. The following conditions must be met to control the obstacle:

[0070]

[0071] When detected When there is a risk of boundary crossing or collision, the system solves the following quadratic programming (QP) problem in real time to force the action to be projected:

[0072]

[0073]

[0074] Final output safety action Strict mathematical limits are provided to prevent pipe wall collisions.

[0075] The following are specific implementation examples of the present invention.

[0076] Example 1: A Zero-Shot Navigation Method for Micro-Agents in the Face of Non-Stationary Fluid Disturbances

[0077] Referring to Figures 1 to 4, this embodiment details the navigation implementation process of the HELM framework in complex heterogeneous pipelines:

[0078] Step S1: Construct a data-driven domain randomized digital twin environment

[0079] In this embodiment, a simulated pipe network is first constructed using high-fidelity 3D modeling software combined with a computational fluid dynamics solver. During the simulation, key operating condition boundaries such as the inlet velocity of the medium, the pipe pulsation frequency, and the wall geometric distortion rate are set as dynamic parameter sets. During each environment reset, these parameters are randomly sampled from a multidimensional joint uniform distribution to generate a non-stationary flow field containing local backflow regions and anomalous turbulence. This mechanism allows the agent to experience a wide range of fluid dynamic variations during training, providing a robust strategic foundation for dealing with unknown OOD flow fields.

[0080] Step S2: Integrate a spatiotemporal perception engine with linear complexity.

[0081] To address the problem of Partially Observable Markov Decision Processes (POMDPs) caused by the ability of micro-agents to only acquire low-dimensional feedback, this embodiment employs a selective state-space model (Mamba) at the front end of the reinforcement learning Actor-Critic architecture. In each control cycle, the model receives an observation sequence. The engine dynamically generates discretized parameters through an internal state selection mechanism. It implicitly extracts historical fluid variation features across long time sequences to achieve this. The computational complexity is reduced by reconstructing the global spatiotemporal fluid context in the feature space, effectively shortening the edge inference latency.

[0082] Step S3: Perform streamlined asymmetric reward reshaping (training phase)

[0083] An asymmetric potential energy field function is defined using background flow streamlines extracted from a digital twin environment. In addition to setting the conventional potential energy difference from the target point, a cosine reward based on the vector angle is introduced. When the agent's velocity direction forms an acute angle with the local streamline direction (i.e., downstream motion), the system provides a positive potential energy reward; when the agent gets too close to the pipe wall boundary, a significant potential energy penalty is imposed. This method provides physically meaningful dense gradient feedback in the early stages of model exploration.

[0084] Step S4: Deploy the control barrier function for physical bottom-line protection (execution phase)

[0085] When the model is actually deployed in a physical environment, the action output by the policy network As a reference instruction, the system defines a rigid safety domain in the underlying control module. Within each control microstep, the action is determined. Whether the safety inequality constraint is satisfied. Once it is determined that the action poses a risk of exceeding the safety boundary, the Control Barrier Function (CBF) mechanism immediately intervenes, invoking the underlying Quadratic Programming (QP) solver to calculate the distance within the constraint control space. The nearest and physically safe best alternative action This provides intelligent agents with physical collision avoidance barriers based on mathematical constraints.

[0086] Based on the method described, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the entire process of steps S1 to S4 included in Embodiment 1. This computer device can be an edge computing platform or an industrial control terminal, and its internal structure typically includes a bus, processor, memory, communication interface, etc.

[0087] Furthermore, this embodiment also provides a computer-readable storage medium on which a computer program is stored. When executed by a processor, the computer program implements all the steps of the method in Embodiment 1. The computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), solid-state drive (SSD), etc.

[0088] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention.

[0089] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances, characterized in that, include: A digital twin simulation environment based on domain randomization is constructed. By parameterizing the boundary of dynamic physical conditions and mapping it to the reinforcement learning state space, a simulation environment pool including non-stationary eddies and fluid disturbances is generated, which serves as the data base for zero-shot policy training. A selective state space perception module is integrated into the navigation strategy network of the micro-agent. The low-dimensional continuous temporal observation sequence collected by the micro-agent is input into the selective state space perception module to implicitly infer the spatiotemporal fluid context features of the current environment. Based on the inferred spatiotemporal fluid context features, a dual-track optimization and control mechanism is constructed: during the model training phase, the reward is reshaped using an asymmetric potential energy function based on the flow field streamlines to guide the agent's policy optimization. During the model deployment and action execution phase, the physical boundary-guided control barrier function is integrated into the action output of the policy network to perform security assessment and forced projection interception on the original actions output by the agent, and output the final security control command.

2. The zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances according to claim 1, characterized in that, The domain-randomized digital twin simulation environment is constructed by performing uniform or truncated normal sampling within a set physical boundary and hydrodynamic parameter range to generate a physical flow field covering extreme disturbance conditions, wherein the environmental parameter vector Follows a domain randomized distribution: and Let be the minimum and maximum parameter values ​​of the physical boundary, respectively, and u(⋅) be the uniform distribution sampling function.

3. The zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances according to claim 1, characterized in that, The selective state space sensing module's state update and output are represented as follows: in, Let be the hidden state at time t. and It is a parameter matrix that is dynamically generated through a selective mechanism. Given the input observation sequence at time t, The output features at time t and A fixed or learnable parameter matrix.

4. The zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances according to claim 1, characterized in that, The expression for the asymmetric potential energy function based on the streamlines of the flow field is as follows: in, To reshape the reward value, As a discount factor, Let be the state potential energy function at time t. Let be the state potential energy function at time t+1. These are the weighting coefficients. The velocity vector of the agent. Let the streamline vectors of the local flow field be... This represents the cosine similarity between two vectors.

5. A zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances according to claim 1, characterized in that, The control barrier function defines a safety set. ,in To characterize the continuously differentiable obstacle function representing the distance between the agent and the inner wall of the pipe, the obstacle control condition is as follows: in, The state of the agent. The original action output by the policy network. The control gain parameter is positive.

6. A zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances according to claim 5, characterized in that, When determining the original action lead to When <0, the safe action is obtained by solving a quadratic programming problem. The objective function of the quadratic programming problem is: The constraints are: in, The original action output by the policy network. For barrier function State gradient, For the agent in state Next action The subsequent state transition function.

7. A zero-shot navigation method for micro-intelligent agents accommodating non-stationary fluid disturbances according to claim 4, characterized in that, The gradient of the asymmetric potential energy function is dynamically adjusted according to the angle between the agent's velocity and the local streamline vector. When the angle is less than 90°, the gradient is positive to reward the agent; when the angle is greater than or equal to 90°, the gradient is negative to punish the agent.

8. A zero-sample navigation system for micro-intelligent agents accommodating non-stationary fluid disturbances, characterized in that, include: The simulation environment construction unit builds a digital twin simulation environment based on domain randomization. By parameterizing the dynamic physical condition boundary and mapping it to the reinforcement learning state space, a simulation environment pool containing non-stationary eddies and fluid disturbances is generated. The perception processing unit integrates a selective state space perception module into the navigation strategy network of the micro-agent, inputting the low-dimensional continuous temporal observation sequence collected by the micro-agent into the perception module to implicitly infer the spatiotemporal fluid context features of the current environment. The dual-track optimization unit constructs a dual-track optimization and control mechanism based on the inferred spatiotemporal fluid context features. During the model training phase, it uses an asymmetric potential energy function based on the flow field streamline to reshape the reward, thereby guiding the agent to optimize its strategy. The safety control unit integrates the physical boundary-guided control barrier function into the action output end of the policy network during the model deployment and action execution phases. It performs safety assessment and forced projection interception on the original actions output by the agent and outputs the final safety control command.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

Images