Micro-nano robot cluster navigation obstacle avoidance method and system based on deep reinforcement learning

Abstract

The invention provides a micro-nano robot cluster navigation obstacle avoidance method and system based on deep reinforcement learning, and the method comprises the steps: enabling a model to have a powerful migration capability from virtuality to reality through a multi-level domain randomization training cluster control strategy; a visual detection module is used for obtaining position information of a micro-nano robot cluster and obstacles in the environment in real time, and a Transformer strategy network of a causal self-attention mechanism including observation vectors of relative states of a target, the cluster and the obstacles and historical information input integration time expansion is constructed; and the problem of long time sequence dependence in part of observable environments can be solved. And outputting a speed control instruction for driving the cluster to move through a near-end strategy optimization algorithm, and mapping the speed control instruction into a physical parameter of an external driving magnetic field to realize closed-loop control of the cluster. After simulation training, the control strategy can be directly migrated to a physical system, fine tuning is not needed, and precise and robust complex tasks such as autonomous navigation, dynamic obstacle avoidance and target tracking are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of micro-nano robots, and particularly relates to a micro-nano robot cluster navigation obstacle avoidance method and system based on deep reinforcement learning. BACKGROUND

[0002] Cluster robot technology, with its large number of simple robots exhibiting complex collective behavior through distributed cooperation, has become a cross-disciplinary research hotspot. Compared with traditional single robot systems, robot clusters, through local interaction and decentralized control, have shown excellent robustness, scalability and flexibility in macro fields such as environmental monitoring and search and rescue. In recent years, with the development of micro-nano processing technology, micro-nano robot clusters have brought the advantages of swarm intelligence into the micro world, and have shown great application potential in biomedical fields such as targeted drug delivery, minimally invasive surgery and precision medicine.

[0003] However, achieving efficient cooperation and precise control of micro-nano robot clusters is still a daunting challenge. This is mainly due to the physical limitations of micro-nano robots: their small size makes it difficult for them to carry sensors and computing units, and their driving ability is limited; at the same time, the interaction within the cluster and between the cluster and the environment is extremely complex and difficult to model. Currently, the mainstream control method relies on a global driving field, such as a magnetic field, an electric field or an acoustic field. Among them, magnetic field driving has become one of the most promising technologies due to its non-invasive nature to biological tissues, strong penetration and ability to be integrated with clinical imaging devices (such as MRI). The operator can remotely control thousands of micro-nano robots by adjusting parameters such as frequency, intensity and direction of the external magnetic field.

[0004] Despite this, existing control strategies still have significant bottlenecks. First, they are highly dependent on the professional experience and manual adjustment of the operator, requiring the operator to continuously observe and intervene, making it difficult to cope with unknown or dynamic environments, and the level of automation and intelligence is low. Second, the micro-nano robot cluster is loosely coupled by a large number of individuals, and its shape is easily destabilized by disturbances in a fluid environment, making it very difficult to maintain a specific formation for precise navigation and obstacle avoidance. Although there have been attempts to introduce fuzzy control, rapid extended random tree and other algorithms to improve automation, these methods mostly plan paths for specific, known static environments, lacking the ability to actively and in real time avoid obstacles and make decisions in unknown, dynamic environments.

[0005] Therefore, there is an urgent need in the art to develop a completely new, adaptive, precise and robust intelligent control method that enables micro-nano robot clusters to achieve truly autonomous navigation in complex, dynamic unknown environments, completely eliminating the dependence on human intervention, thereby greatly improving their level of intelligence, autonomy and task success rate in practical applications. SUMMARY

[0006] The present invention provides a micro-nano robot cluster navigation and obstacle avoidance method and system based on deep reinforcement learning, which is used to solve the defects of the existing micro-nano robot cluster control technology in the prior art, such as strong dependence on operator experience, poor adaptability in unknown dynamic environments, and difficulty in achieving robust autonomous navigation and active obstacle avoidance. The method realizes autonomous navigation and adaptive dynamic obstacle avoidance method for micro-nano robot clusters, thereby significantly improving the intelligence, autonomy and task execution capabilities of the cluster in complex real environments.

[0007] In a first aspect, the present invention provides a micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning, comprising: Obtain video streams of the micro-nano robot cluster and obstacles in the environment, process the video streams using a fine-tuned YOLOv5 visual detection model, and identify and output a standardized state vector; The normalized state vector is input into a pre-trained Transformer-based policy network, which uses temporal attention and proximal policy optimization algorithms to output a continuous action vector. Decode the continuous motion vectors into execution instructions for the physical drive system and output the magnetic field control parameters of the three-axis Helmholtz coil system; Based on the magnetic field control parameters, the micro-nano robot cluster is physically driven and makes intelligent decisions, constantly sensing new states, repeating the cycle to achieve closed-loop control of the system.

[0008] According to a micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning provided by the present invention, the standardized state vector includes the Wiener robot cluster speed, the Wiener robot cluster position, the Wiener robot size, the target point and the obstacle information vector.

[0009] According to the present invention, a micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning is provided, which is based on a Transformer strategy network and includes: Using rotation position encoding, by querying and key Vector applies an absolute position The associated rotation matrix To achieve:

[0010] The rotation matrix It consists of multiple 2D rotation blocks:

[0011] Two different locations and After the dot product operation is performed on the encoding vector of related:

[0012] The causal self-attention mechanism with time expansion is adopted, history information is introduced in the causal self-attention calculation of the Transformer, and the hidden state of the current time is spliced with the hidden state sequence of the previous time in the calculation of the attention score of each layer:

[0013] The query (Q), key (K) and value (V) matrices of the first layer are calculated from the hidden state:

[0014] , and are the learnable weight matrices of the first layer; For a query , is obtained from , and a key , is obtained from one of the elements in , and the attention score is :

[0015] The root mean square layer normalization is adopted:

[0016] The Swish activation function is used in the feedforward neural network:

[0017] where is the sigmoid function:

[0018] The Tanh activation function is used in the output layer of the policy network and the value network to constrain the output value in the range of (-1, 1): .

[0019] According to the micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning provided by the application, a proximal policy optimization algorithm is adopted, which comprises: Initializing the Transformer decoder​​​​​​​​ ,Strategy , value function , Experience Pool , memory length and worker threads ; If the execution round is judged Greater than the total number of rounds , then the algorithm ends, otherwise it starts in parallel sampling workers and performs both environmental and perceptual randomization; A sampling worker initializes the simulation environment and obtains the initial state , in the execution steps Less than or equal to the total number of steps When , calculate the hidden state, sample actions from the strategy, execute the action to obtain the new state and reward, get the experience tuple and store the experience tuple in the experience pool If the process ends, reset the simulation environment, otherwise execute the next step ; When the number of steps Greater than the total number of steps Enter the advantage estimation stage and traverse the experience pool Each trajectory in Start iteration to , calculate the time series error, generalized advantage estimate and value regression target in turn, and perform the optimization phase; If the optimization round Less than or equal to the total number of optimization rounds , then from the experience pool Mini-batch sampling , save the current strategy, calculate the importance sampling ratio, pruning strategy loss, entropy reward and total loss in turn, use Adam optimizer to perform the next optimization round , until the optimization round Greater than the total number of optimization rounds , continue to the next round .

[0020] According to a micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning provided by the present invention, the proximal policy optimization algorithm enables the intelligent agent to obtain the maximum expected cumulative reward in an interactive environment:

[0021] The overall goal of the proximal policy optimization algorithm is to introduce the entropy reward term weighted joint optimization:

[0022] State Space The state vector consists of the relative distances of the clusters , relative angle , position , size , speed , and the action of the last time Spliced, and further splice the relative information of up to observable obstacles:

[0023] action space including continuous two-dimensional vectors in the action space ; reward function including:

[0024] AdamW optimizer includes:

[0025] where, and are the estimates of the first and second moments respectively, is the decay rate of the exponentially weighted average. is the gradient of the step; cosine annealing learning rate includes:

[0026] where, is the current step number, is the total number of steps scheduled; Gradient clipping includes:

[0027] where denotes the Euclidean norm, >0 is a given threshold.

[0028] According to the micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning provided by the application, the magnetic field control parameters include the pitch angle and the direction angle required to generate a rotating magnetic field in a specific direction by driving a three-axis Helmholtz coil system.

[0029] According to the micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning provided by the application, based on the magnetic field control parameters, the physical driving of the micro-nano robot cluster is performed and intelligent decision is made, the new state perception is continuously performed, and the cycle is repeated, realizing the closed-loop control of the system, including: obtain pre-trained weights and output real-time execution angles; initializing the pre-allocated observation buffer, the direction socket and the pitch socket, the YOLO detector and the policy network; executing the detection thread, acquiring the picture from the camera, performing YOLO target detection, and publishing the bounding box coordinates to the shared memory; executing the master thread, sending the previous action according to the current working state, judging the picture as valid in combination with the bounding box coordinates, obtaining the latest coordinate information from the shared memory, constructing a state vector in combination with the historical data, performing forward inference by the Transformer policy network, outputting the normalized action, mapping the normalized action into physical control parameters, sending the control instruction through the UDP protocol, receiving the control instruction by the lower computer, adjusting the coil current to generate a magnetic field to drive the cluster to move, re-acquiring a new working state, and continuing to execute the system control.

[0030] In a second aspect, the application further provides a micro-nanorobot cluster navigation and obstacle avoidance system based on deep reinforcement learning, comprising: a CSP-3000 power supply, a MyRIO controller, a Copley driver, a working space, a camera, a three-axis Helmholtz coil, an iron stand, and a deep learning server; The CSP-3000 power supply is connected with the Copley driver to supply power to the Copley driver; The MyRIO controller is connected with the Copley driver, the three-axis Helmholtz coil, and the deep learning server, respectively; The three-axis Helmholtz coil is composed of three pairs of large-size coils that are orthogonal to each other, forming a working space, and is driven by a programmable direct current power supply and a special amplifier. A lower computer control program written by LabVIEW generates a uniform rotating magnetic field with adjustable direction, intensity, and rotating frequency at the center of the experimental area; The iron stand is vertically installed outside the three-axis Helmholtz coil and is used to install the camera so that the camera is perpendicular to the top of the three-axis Helmholtz coil; The camera is used to capture dynamic video pictures of the micro-nanorobot cluster and obstacles in the environment in real time, and transmit the original video stream of the dynamic video pictures to the deep learning server through a high-speed data interface; The deep learning server is used to execute the micro-nanorobot cluster navigation and obstacle avoidance method based on deep reinforcement learning.

[0031] In a third aspect, the application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the micro-nanorobot cluster navigation and obstacle avoidance method based on deep reinforcement learning as described above.

[0032] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements any of the above-mentioned deep reinforcement learning-based micro / nano-robot swarm navigation and obstacle avoidance method.

[0033] The deep reinforcement learning-based micro / nano-robot swarm navigation and obstacle avoidance method and system provided by the present application has strong robustness to environmental noise, perception error, and execution delay uncertainty through the innovative time-expanded causal self-attention mechanism and systematic multi-level domain randomization training, can be directly migrated from a simulation environment to a physical system, solves the core problem of simulation-to-reality migration, makes the micro / nano-robot swarm free from dependence on human operators, and can realize autonomous navigation and active obstacle avoidance in a completely unknown and dynamically changing environment, exhibits high-level intelligence, and the proposed framework is universal, not only suitable for magnetic-driven micro / nano-robot swarms, but also can be generalized to other types of driven (such as acoustic, optical, and electrical) swarm systems and more extensive macro-robot swarm control scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0035] Figure 1 is a flowchart of the deep reinforcement learning-based micro / nano-robot swarm navigation and obstacle avoidance method provided by the present application; Figure 2 is a schematic diagram of the overall flow of the control method provided by the present application; Figure 3 is a structure diagram of the Transformer provided by the present application; Figure 4 is a structure diagram of the physical experiment system provided by the present application; Figure 5 is a schematic diagram of the overall training flow in the simulation environment provided by the present application; Figure 6 is a schematic diagram of the strategy network architecture based on the Transformer provided by the present application; Figure 7 is a cosine annealing learning rate scheduling curve diagram provided by the present application; Figure 8 is a key performance indicator change curve diagram in the reinforcement learning training process provided by the present application; Figure 9A trajectory graph for autonomous navigation and dynamic obstacle avoidance in a simulation environment is provided in the present application. Figure 10 A "perception-decision-execution" closed-loop control flowchart deployed in a real physical system is provided in the present application. Figure 11 A processing flowchart of a visual detection model is provided in the present application. Figure 12 A trajectory graph for target tracking and dynamic obstacle avoidance tasks in a real physical experiment is provided in the present application. Figure 13 A structural schematic diagram of an electronic device is provided in the present application.

[0036] Reference signs: 1: CSP-3000 power supply; 2: MyRIO controller; 3: Copley driver; 4: workspace; 5: camera; 6: three-axis Helmholtz coil; 7: iron stand; 8: deep learning server. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0038] Figure 1 A flowchart of a micro-nanorobot swarm navigation and obstacle avoidance method based on deep reinforcement learning is provided in the present application embodiment, as shown in Figure 1 , comprising: Step 100: Obtain the video stream of the micro-nanorobot swarm and the obstacles in the environment, process the video stream using the fine-tuned YOLOv5 visual detection model, and identify and output the standardized state vector; Step 200: Input the standardized state vector into the pre-trained Transformer-based policy network, use the time attention and proximal policy optimization algorithm, and output the continuous action vector; Step 300: Decode the continuous action vector into physical driving system execution instructions, and output the magnetic field control parameters of the three-axis Helmholtz coil system; Step 400: Based on the magnetic field control parameters, perform physical driving of the micro-nanorobot swarm and make intelligent decisions, constantly perceive new states, and circulate, to realize closed-loop control of the system.

[0039] Specifically, the overall process of the control method proposed in the embodiment of the present invention is as follows: Figure 2 Shown, including: (1) Environmental perception and state construction Vision devices such as cameras deployed above the system capture real-time video streams of the micro-nano robot cluster and obstacles (static or dynamic) in the environment. A fine-tuned YOLOv5 visual detection model processes the video frames, identifies and outputs the bounding boxes of the cluster and obstacles in real time, and then calculates their position, size, and other information in the world coordinate system. This information, along with the location of the navigation target point and the action performed by the system at the previous moment, is integrated into a standardized state vector , as the input of the policy network.

[0040] (2) Policy Reasoning and Action Generation State vector is fed into the core of this invention - a pre-trained, Transformer-based policy network. This network not only processes the current state , and also uses its built-in time-extended causal self-attention mechanism to review and integrate the state information of a series of past moments (constituting the context), so as to understand the dynamic changes of the environment and make more forward-looking decisions. After network inference, it outputs a two-dimensional continuous action vector , represents the expected normalized speed that the cluster should achieve at the next moment.

[0041] (3) Action mapping and physical driving The control system converts the normalized velocity vector output by the policy network into Decoded into specific instructions that can be executed by the physical drive system. In this embodiment, the instruction is the magnetic field control parameter of the three-axis Helmholtz coil system, specifically the pitch angle and direction angle These parameters determine the direction and strength of the rotating magnetic field, which precisely drives the cluster to produce the corresponding movement.

[0042] (4) Closed-loop control The above steps form a high-speed closed-loop control system. The system continuously perceives new states, makes intelligent decisions, and executes physical actuation in a repetitive cycle, thus enabling real-time, autonomous navigation and obstacle avoidance for the swarm in unknown environments.

[0043] The core of this invention is to empower micro-nanorobotic swarms with advanced artificial intelligence algorithms, enabling them to achieve highly autonomous navigation and obstacle avoidance in complex, dynamic, and unknown environments without human intervention. This approach seamlessly integrates visual perception, intelligent decision-making, and physical actuation to form a complete and efficient closed-loop control system.

[0044] In one embodiment, the present embodiment proposes a Transformer-based policy network architecture. The policy network designed by the present embodiment adopts an actor-critic model with a shared backbone, the core of which is a multi-layer Transformer structure, as shown in Figure 3 The design can effectively process time series data and manage partial observability problems, including the following key modules: Rotary Position Encoding: In order to enable the model to understand the time sequence relationship of each state in the input sequence, the present embodiment adopts the rotary position encoding technology. Unlike traditional position encoding methods, rotary position encoding is achieved by applying a rotation matrix related to the absolute position of the query and key vectors.

[0045] (1) The rotation matrix is composed of a series of two-dimensional rotation blocks: (2) The key feature is that the dot product of the encoding vectors of two different positions and is only related to the relative position , as shown in equation (3). This enables the model to naturally handle relative time relationships and generalize to sequence lengths not seen during training.

[0046] (3) Causal Self-Attention Mechanism with Time Expansion: Traditional reinforcement learning agents usually only focus on the current state. However, in a partially observable dynamic environment, it is not enough to infer the motion trend of obstacles or the dynamic characteristics of oneself based on only one frame of information. Therefore, the present embodiment introduces historical information into the causal self-attention calculation of the Transformer. When calculating the attention score at each layer, we concatenate the hidden state of the current time with the hidden state sequence of the previous times.

[0047] (4) In this way, the model can focus on the key and value sequence composed of historical and current states when generating the query vector for the current state , thereby capturing the dynamic changes of the environment and itself. To ensure training stability, the gradient of the historical state is truncated during backpropagation. The query vector of the first （ ​​​ ）、 key （ ） and value （ ） The matrix is calculated from the hidden state.

[0048] (5) , and is the first layer learnable weight matrix. At this time, the causal attention score is calculated by scaling dot product attention and combined with rotated position encoding for relative position encoding. For a query (from ) and a key (from one of the elements in .

[0049] (6) Root mean square layer normalization: In order to improve the calculation efficiency and stability, the root mean square layer normalization is used instead of the traditional layer normalization. It omits the mean subtraction step and only scales through the root mean square of the input.

[0050] (7) Swish activation function: Swish activation function is used in feedforward neural network, and its smooth and non-monotonic characteristics help to improve gradient flow and learning representation.

[0051] (8) where, is the sigmoid function.

[0052] (9) Tanh activation function: Tanh activation function is used in the output layer of the policy network and the value network, which restricts the output value in the range of (-1, 1), which is conducive to stabilizing the learning process and generating normalized actions.

[0053] (10) In one embodiment, the present application uses the proximal policy optimization algorithm to train the above-mentioned Transformer network end-to-end. The proximal policy optimization algorithm limits the amplitude of each policy update through a clipped objective function, ensuring the stability of the learning process. The complete training algorithm process is shown in Algorithm 1.

[0054] Algorithm 1. Proximal policy optimization algorithm based on Transformer Input: Transformer decoder , policy , value function , experience pool , memory length , worker threads

[0055] 1. for epoch Perform 2. Perform in parallel sampling workers 3. Initialize simulator , get initial state

[0056] 4. for step Perform 5.

[0057] 6.

[0058] 7. Perform action , and store into

[0059] 8. If is true, break loop and reset simulator 9. for each experience trajectory Perform 10. for step decrement iteration Perform 11. / / temporal difference error 12. / / generalized advantage estimation 13. / / value regression target 14. for optimization epoch Perform 15. Sample mini-batch from

[0060] 16. for optimization epoch Perform 17. Let be the policy before update 18. For each , compute ratio ​

[0061] 19.

[0062] 20.

[0063] 21. / / Entropy reward 22.

[0064] 23. With AdamW optimizer, minimize , and update ,

[0065] The goal of the proximal policy optimization algorithm is to optimize the policy so that the agent obtains the maximum expected cumulative return in the interactive environment.

[0066] (11) Unlike traditional policy gradient methods, the proximal policy optimization algorithm introduces a clipping mechanism in the policy update process to suppress excessive policy changes, enhancing the numerical stability and convergence efficiency of training. To further enhance the exploration ability and convergence quality of the policy, the entropy reward term is introduced into the total objective of the proximal policy optimization algorithm for joint optimization.

[0067] (12) State space : The state vector is composed of the target information (relative distance , relative angle ) of the swarm, its own information (position , size , velocity ) and the action at the last time , and further concatenates the relative information of up to observable obstacles. This self-centered relative coordinate representation enhances the generalization of the policy.

[0068] (13) Action space : The action space is a continuous two-dimensional vector representing the normalized expected velocity.

[0069] Reward function : In order to guide the agent to learn the desired behavior, a sophisticated composite reward function is designed, as shown in equation (14).

[0070] (14) The function is composed of four parts of weighting: (1) Goal-oriented reward. When not reaching the goal, punish the distance to the goal; give a large positive reward when reaching. (2) Obstacle avoidance reward. Punish the close distance to the obstacle, and give reward to the behavior of successful avoidance. (3) Exploration reward. Before reaching the goal, encourage the agent to move at a certain speed to prevent it from "standing still". (4) Stable punishment. Punish the violent speed change, encourage smooth and stable movement.

[0071] Training optimization details: AdamW optimizer: Use AdamW optimizer, which is an improvement on the standard Adam optimizer. By decoupling weight decay from gradient-based update steps, it can achieve better generalization performance. By reducing the weight decay from the conventional 0.999 to 0.98, it makes it more sensitive to the possible instantaneous large gradient in training, thus improving the adaptability and stability of training.

[0072] (15) where, and are the estimates of the first and second moments, respectively, is the decay rate of the exponentially weighted average. is the gradient of the th step.

[0073] Cosine annealing learning rate: Use cosine annealing scheduler to dynamically adjust the learning rate, which decreases from the initial value smoothly to 0, which helps the model better converge to the optimal solution in the later training.

[0074] (16) where, is the current step number, is the total number of steps.

[0075] Gradient clipping: Use norm clipping to prevent gradient explosion and ensure numerical stability in the training process.

[0076] (17) where represents the Euclidean norm, is a given threshold value.

[0077] ​Multi-level domain randomization: This is a key technique to achieve robust simulation-to-reality transfer. During the simulation training phase, the invention systematically randomizes the parameters of multiple levels that affect the swarm behavior, including: (1) Environment randomization: the number, shape, size, static-dynamic ratio, etc. of obstacles. (2) Perception randomization: Gaussian noise is added to the observation state to simulate real sensor errors. (3) Execution randomization: noise and delay are added to the action mapping and execution process to simulate the imperfections of the real driving system. (4) Dynamics randomization: the size, maximum speed of the swarm itself, and the kinematic parameters of the obstacles are randomized. By training in this "worst-case" generalization environment, the policy network is forced to learn a robust control strategy that is insensitive to various disturbances, so it can be directly applied to the physical world without the need for secondary tuning.

[0078] In one embodiment, the invention provides a complete set of physical systems, as shown in Figure 4 for verifying and deploying the trained agent.

[0079] Experimental setup: The core of the system is a set of three-axis Helmholtz coils driven by a high-power power supply and amplifier, which can generate a uniform and controllable rotating magnetic field. A digital microscope camera is used as a visual perception device, which transmits real-time video streams to the host computer.

[0080] Vision detection module: A YOLOv5-based vision detection model runs on the host computer. This model is trained on a dataset containing original images and augmented images (through noise, rotation, grayscale transformation, etc.) and can accurately and quickly identify micro-nano robot swarms and obstacles in the field of view.

[0081] Simulation-to-reality deployment process: The actual deployment of the closed-loop control process is shown in Algorithm 2. Algorithm 2. Simulation-to-reality deployment Input: Pre-trained weights ，

[0082] Output: Real-time execution angle

[0083] Initialization: ; Pre-allocate observation buffer ; ; Direction socket and pitch socket ; YOLO detector ; Strategy ;

[0084] 1. While working state Execute 2. Send , / / Execute the action 3. Get the image from the camera

[0085] 4.if Effective execution 5. / / Detection 6. / / Observation construction 7. / / Observation construction 8. / / Inference 9. / / Sample and normalize the action 10.

[0086] 11.

[0087] 12.

[0088] The present invention uses an independent thread to run the YOLOv5 model, continuously processes the video stream from the camera, detects clusters and obstacles for each frame, and publishes the detected position information in real time. The main control loop periodically obtains the latest detection coordinates and constructs the state vector . Then, the state vector And historical observations (context) input has been loaded with the pre-trained weights of the policy network, complete a forward reasoning and output action . The normalized action According to the definition, the angles are mapped to the actual physical control parameters of the Helmholtz coils. The calculated angle parameters are then sent to the lower-level control program via the UDP communication protocol. LabVIEW receives the control instructions in real time and dynamically adjusts the coil currents to generate the target magnetic field, thereby driving the cluster's motion. The entire system forms an efficient "perception-decision-execution" closed loop, effectively ensuring the real-time and stable control process.

[0089] It can be understood that the implementation scheme of the present invention includes two core stages. The first stage is to train the control policy network through deep reinforcement learning in a large-scale parallel simulation environment; the second stage is to deploy the trained policy network into a real physical system for online closed-loop control and task verification.

[0090] The first stage is offline training of the policy network based on the simulation environment This embodiment will be described in detail Figure 5The complete process and technical details of end-to-end training of the proposed Transformer-based policy network in the simulation environment are shown. The ultimate goal of this embodiment is to generate an intelligent agent with high robustness and generalization ability, which can be directly migrated from the simulation environment to the physical world without fine-tuning to perform complex tasks. The entire training process strictly follows the Figure 5 framework and the logic defined by Algorithm 1.

[0091] First of all, in the construction of the overall training framework, the actor-critic architecture is adopted, which is known for its excellent data efficiency and stability in the field of reinforcement learning, and the proximal policy optimization is selected as the core policy update algorithm. In order to maximize data collection efficiency and break the temporal correlation between data, the training system is designed in a highly parallel mode. Specifically, the system will start W independent simulation environment instances simultaneously, each instance hosting a micro-nano robot cluster and a randomly generated task scenario. These parallel "sampling workers" interact with their respective environments according to the instructions of the current global policy network, continuously generating experience trajectory data containing state, action, reward, and next state. All the experience data collected by the parallel workers are unified into a global experience replay pool, providing rich and diverse training samples for subsequent policy optimization.

[0092] The decision-making core of the invention, namely the policy network, has a structure as Figure 6 shown. The network adopts the design of sharing a powerful Transformer encoder as the feature extraction backbone with the actor and critic to improve learning efficiency. A complete decision-making process is as follows: The input received by the network is the state vector that accurately describes the current situation. The construction of this vector strictly follows the structure defined by formula (13), which is a high-dimensional concatenated vector that systematically integrates four aspects of information: (1) target information: the relative position of the cluster relative to the navigation target point (distance and angle represented in polar coordinates), providing the agent with a clear navigation direction; (2) self information: the Cartesian coordinates, outer contour size, and current velocity component of the cluster, enabling the agent to have a sense of self-awareness; (3) historical information: the control action executed by the system at the last time, which helps the network learn a smoother and more stable control policy; (4) environmental information: the relative position information of the observable obstacles within the maximum field of view. All relative information is calculated with the cluster itself as the center. This "self-centered" representation is the key to achieving policy generalization, ensuring that the agent's decision-making does not depend on its absolute position in the world coordinate system.

[0093] The state vector is fed into a multi-layer stacked Transformer encoder backbone. In each layer of the encoder, information is processed through a series of carefully designed modules. To enable the model to understand the temporal order of individual states in the input sequence, the invention employs a rotational positional encoding technique. Rotational positional encoding is achieved by applying a rotation matrix corresponding to the absolute position of the query vector (q) ) and key vector (k ) in the sequence (as shown in equations (1)-(2)). The ingenuity lies in that when the two vectors after rotation are multiplied, the result is naturally only related to their relative position (as shown in equation (3)). This makes the attention score itself encode the relative time difference, allowing the model to more naturally understand the concept of "before" or "after", and this ability can be easily generalized to longer or shorter sequences than seen during training.

[0094] To solve the problem of partial observability that exists in the navigation task of micro-nano robots (for example, it is impossible to determine the motion trend of obstacles based on a single frame of image), the invention improves the Transformer architecture and proposes a time-expanded causal self-attention mechanism. As shown in equation (4), when calculating the self-attention of the current time , the model does not only focus on the input information at the current time. It will first concatenate the hidden state at the current time with a sequence of hidden states at past times stored in memory. In this way, when generating the query vector for the current state, the model can interact with a sequence of keys (K ) and values (V ) that contain rich historical information. This mechanism gives the model the ability to "remember" and enables it to capture dependencies that span long time steps, such as inferring the motion trajectory and speed of obstacles through multiple consecutive frames of observation, thereby making more predictive obstacle avoidance decisions. To ensure the stability of training, the historical states do not participate in gradient calculation during backpropagation, i.e., their gradients are truncated.

[0095] In the feedforward part of the network, the invention uses the Swish activation function as shown in equation (8), which has smooth and non-monotonic properties that help improve gradient flow and learning representation. At the same time, to improve computational efficiency and stability, the invention uses the root mean square layer normalization as shown in equation (7) instead of the traditional layer normalization.

[0096] The shared backbone encoded deep feature representation is sent to two separate output heads, each consisting of a multi-layer perceptron. The actor head is responsible for policy generation, and its output layer uses a Tanh activation function (Equation (10)) to constrain the action in the normalized range of (-1, 1), representing the desired two-dimensional velocity vector. The critic head is responsible for evaluating the value of the current state, outputting a scalar to provide a guidance signal for the actor's policy update.

[0097] Training reinforcement learning models directly on real physical systems is costly and risky. Therefore, the present invention adopts a strategy of training in a simulation environment and then deploying the model to the real world. In order to bridge the inevitable "gap" between simulation and reality, the present embodiment adopts a systematic multi-level domain randomization strategy during training. The core idea of this strategy is that, rather than having the model perfectly adapt to a single simulation environment, it is better to have it adapt to thousands of slightly different environments with random perturbations, thereby "forcing" the model to learn more essential and robust control laws that are not sensitive to these perturbations. Specifically, randomization covers at least four levels: (1) Environment randomization: At the beginning of each training round, the number, size, shape, initial position, and dynamic-static ratio of obstacles in the simulation environment are randomly sampled within a pre-set range. The speed and direction of dynamic obstacles are also randomized. (2) Perception randomization: In the state vector received by the model, the positions and sizes of the swarm and obstacles are artificially injected with Gaussian noise to simulate the measurement errors of sensors such as cameras in the real world. (3) Drive execution randomization: The delay and inaccuracy of the real drive system are simulated in the simulation. For example, a random delay is imposed on the action output by the model, and a random scaling factor and additional noise are added to simulate the imperfect response of the real drive system. (4) System dynamics randomization: The physical properties of the micro-nanorobot swarm (such as effective diameter, maximum achievable speed) and the dynamics parameters of the simulation environment are also randomly perturbed within a small range, allowing the model to adapt to different magnetic responses of the micro-nanorobots.

[0098] In the specific training optimization link, the application designs a fine composite reward function (as shown in formula (14)), which guides the agent to learn the expected behavior through four weighted items: goal-oriented reward, obstacle avoidance reward, exploration reward and stability penalty. In the optimization algorithm level, the principle of the proximal policy optimization algorithm is strictly followed. For each optimization period, the system randomly samples a small batch of data from the experience pool, calculates the generalized advantage estimation to obtain a stable and low-bias advantage function. In the optimization step, the application uses the AdamW optimizer (as shown in formula (15)), which can obtain better generalization performance by decoupling weight decay and gradient-based update steps. The learning rate adjustment follows the cosine annealing scheduling strategy (as shown in formula (16) and Figure 7 ), which smoothly decreases from a higher initial value to a final value, which helps the model to quickly explore in the early stage of training and more stably converge to the optimal solution in the later stage. At the same time, to prevent numerical instability caused by sudden increase of gradient during training, the application also uses the norm gradient clipping technique (as shown in formula (17)) to limit the L2 norm of the gradient within a preset threshold.

[0099] Finally, the total optimization objective function of the application is composed of three parts: the clipping proxy objective function specific to the proximal policy optimization, the value function loss for stable training, and an entropy reward term (formula (12)). The purpose of introducing the entropy reward is to encourage the policy to maintain a certain randomness, so as to promote the agent to explore more fully in the environment and avoid its premature convergence to a local optimal policy. Through 500 million time step iterative training, as shown in Figure 8 , the key performance indicators such as cumulative reward, policy loss and value loss in the training process all show good convergence trend, proving the effectiveness and stability of the training framework proposed by the application. After training, the obtained agent is tested in a complex simulation environment full of dynamic obstacles, and its navigation trajectory is shown in Figure 9 , the trajectory is smooth and efficient, and can accurately reach the target point and successfully avoid all dynamic obstacles, which intuitively verifies the high performance and high intelligence level of the learned strategy.

[0100] The second stage is the online deployment and closed-loop verification based on the physical system This embodiment will describe in detail how to seamlessly deploy the pre-trained strategy network with strong generalization ability in embodiment one to the real physical experiment system shown in Figure 4 , and realize the efficient and stable "perception-decision-execution" real-time closed-loop control defined in Figure 10 . The success of this embodiment is the final proof of the effectiveness of the core technology of the application, especially the multi-level domain randomization simulation to reality migration strategy.

[0101] Firstly, the physical system platform constructed by the present application has a hardware structure as shown in Figure 4 The figure is a whole of deep fusion of software and hardware. At the physical execution level, the core of the system is a set of customized high-performance three-axis Helmholtz coil system. The system is composed of three pairs of mutually orthogonal large-size coils, and is driven by high-precision, high-power programmable direct current power supply and special amplifier. Through the lower computer control program written by LabVIEW, the system can generate a uniform rotating magnetic field with direction, intensity and rotating frequency that can be accurately and dynamically controlled at the center of the experimental area. This magnetic field can exert uniform driving force on the magnetic micro-nano robot cluster placed in the working area in a remote and non-contact manner, which is the physical basis for realizing the macro motion control of the cluster. At the visual perception level, the system is equipped with a high-resolution, high-frame-rate digital microscope camera, which is vertically installed above the Helmholtz coil system. The camera is responsible for capturing dynamic video pictures of the micro-nano robot cluster and obstacles in the environment in real time, and transmitting the original video stream to the host computer through a high-speed data interface to provide real-time visual input for the whole closed-loop control system.

[0102] On the host computer, a real-time perception module developed independently by the present application is running. The core of the module is a YOLOv5 target detection model based on deep learning, and its processing flow is as shown in Figure 11 It is worth noting that the model is not directly used with general pre-trained weights, but is specially fine-tuned for the application scenario. In order to achieve this goal, the present application constructs a customized dataset containing one hundred images collected in a real experimental environment. The dataset covers different light conditions, different robot cluster morphologies and various obstacles. In addition, in order to further improve the robustness and generalization ability of the model, a large amount of data enhancement processing is performed on the original images, including random rotation, color jitter, brightness adjustment, adding Gaussian noise, etc. Through fine-tuning training on this highly diversified dataset, the final YOLOv5 model can achieve real-time and accurate detection in milliseconds in a complex and disturbed real visual background, and accurately output the overall bounding box of the micro-nano robot cluster and the bounding box coordinates of each obstacle in the field of view.

[0103] The core of the present application is to deploy the "perception-decision-execution" closed-loop control process as shown in Figure 10 The implementation of the process relies on a carefully designed multi-threaded software architecture to ensure the real-time performance and response speed of the whole system. The specific implementation details strictly follow the logic of Algorithm 2.

[0104] The whole control system is mainly composed of two parallel running key threads: asynchronous detection thread and periodic host thread.

[0105] The detection thread runs independently and continuously. Its only task is to fetch the latest video frame from the digital microscope camera and immediately send it into the YOLOv5 model loaded with fine-tuned weights for forward inference. Once the detection is done, the thread updates all the identified targets (robot swarm, obstacles) with their latest pixel coordinates and size information into a shared memory region accessible by all threads. This asynchronous design decouples the time-consuming vision processing from the control decision-making process, ensuring that the master thread always gets the latest environment perception information with extremely low latency, without being blocked by the frame rate fluctuations of the vision processing.

[0106] The master thread is the "brain" of the whole closed-loop system, which runs in a fixed high frequency loop. At the beginning of each control cycle, the master thread performs the following series of strictly defined steps: State acquisition and construction: The thread first reads the latest target coordinates provided by the detection thread from the shared memory region. Then, it combines the pre-set navigation target point position, the action performed at the last time from the historical record, and the observation information at past times maintained through the cache, to construct the state vector at the current time and the historical information context for the causal self-attention mechanism in strict accordance with the same format as during training (i.e., the definition of formula (13)). This step is the key to ensuring the successful migration from simulation to reality, ensuring the consistency of the data distribution input to the policy network.

[0107] Policy inference: The constructed state vector and historical context are input into the Transformer policy network loaded with the pre-trained weights in Example One. The network performs a complete forward propagation calculation, and the inference process only takes a few milliseconds. The output of the actor head is the two-dimensional velocity vector representing the expected achievement of the swarm at the next time, normalized by the Tanh function.

[0108] Action mapping and decoding: The normalized velocity vector output by the policy network is an abstract mathematical instruction. In order to let the physical system execute, it needs to be decoded into specific physical control parameters. In this embodiment, the mapping module converts the expected velocity vector into the pitch angle θ and the direction angle φ required to generate a specific direction rotating magnetic field by driving the three-axis Helmholtz coil system.

[0109] Instructions issuing and physical driving: The calculated pitch angle and yaw angle parameters are sent to the LabVIEW slave program responsible for controlling the Helmholtz coil power supply in real time through efficient and low-latency User Datagram Protocol (UDP). The slave machine listens to the UDP port in real time, and once it receives new angle instructions, it will immediately adjust the current size and phase applied to the three pairs of coils, thereby generating a rotating magnetic field in space that is precisely directed at the target direction. Under the action of this magnetic field, the magnetic micro-nano robot cluster will produce a macroscopic translational motion that matches the expected speed.

[0110] The above four steps are completed in one cycle and are repeated in a loop, forming a high-speed, stable and intelligent "perception-decision-execution" closed loop. The camera captures the new position of the cluster after movement, the detection thread updates the coordinates, and the main control thread reads the new coordinates and makes a new decision in the next cycle. This cycle allows the robot cluster to continuously adjust its behavior in real time based on the dynamic changes in the environment.

[0111] The final deployment effect is shown in Figure 12 In real physical experiments, the micro-nano robot cluster controlled by the present application exhibits excellent autonomous intelligence. In the dynamic target tracking task, the cluster can accurately follow a moving target and always maintain a safe distance. In the dynamic obstacle avoidance task, when an obstacle suddenly enters its path, the cluster can quickly respond and plan a smooth and efficient evasive trajectory to successfully bypass the obstacle and continue moving towards the target point. The complex task motion trajectory completed by it in the real physical system is highly consistent with the ideal trajectory obtained by training in the simulation environment Figure 9 .

[0112] Figure 13 An example of an entity structure diagram of an electronic device is shown in Figure 13As shown, the electronic device can include a processor 1310, a communications interface 1320, a memory 1330, and a communications bus 1340, wherein the processor 1310, the communications interface 1320, and the memory 1330 complete mutual communication through the communications bus 1340. The processor 1310 can invoke a logic instruction in the memory 1330 to execute the deep reinforcement learning-based micro-nano robot cluster navigation and obstacle avoidance method, which includes: acquiring a video stream of a micro-nano robot cluster and obstacles in an environment, processing the video stream by using a fine-tuned YOLOv5 visual detection model, and identifying an output standardized state vector; inputting the standardized state vector into a pre-trained Transformer-based policy network, outputting a continuous action vector by using a time attention and a near-end policy optimization algorithm; decoding the continuous action vector into a physical driving system execution instruction, and outputting a magnetic field control parameter of a three-axis Helmholtz coil system; based on the magnetic field control parameter, performing physical driving of the micro-nano robot cluster and making intelligent decisions, constantly performing new state perception, and cyclically repeating, to realize closed-loop control of the system.

[0113] In addition, the logic instruction in the memory 1330 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0114] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0115] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0116] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning, characterized in that: include: Obtain video streams of the micro-nano robot cluster and obstacles in the environment, process the video streams using a fine-tuned YOLOv5 visual detection model, and identify and output a standardized state vector; The normalized state vector is input into a pre-trained Transformer-based policy network, which uses temporal attention and proximal policy optimization algorithms to output a continuous action vector. Decode the continuous motion vectors into execution instructions for the physical drive system and output the magnetic field control parameters of the three-axis Helmholtz coil system; Based on the magnetic field control parameters, the micro-nano robot cluster is physically driven and makes intelligent decisions, constantly sensing new states, repeating the cycle to achieve closed-loop control of the system.

2. The micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning according to claim 1 is characterized in that: The standardized state vector includes the Wiener robot cluster velocity, Wiener robot cluster position, Wiener robot size, target point and obstacle information vector.

3. The micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning according to claim 1 is characterized in that: Transformer-based policy networks include: Using rotation position encoding, by querying and key Vector applies an absolute position The associated rotation matrix To achieve: The rotation matrix It consists of multiple 2D rotation blocks: Two different locations and After the dot product operation is performed on the encoding vector of related: The time-extended causal self-attention mechanism is used to introduce historical information into the causal self-attention calculation of Transformer. When calculating the attention score at each layer, the hidden state at the current moment is included. Compared with before The hidden state sequence at each moment To splice: No. Layer query ( ),key( ) and value ( ) matrix is ​​calculated from the hidden state: , and It is The layer's learnable weight matrix; For a query , Depend on Get, and a key , Depend on One of the elements in is obtained, and its attention score is : Use root mean square layer normalization: Using the Swish activation function in a feedforward neural network: in, is the sigmoid function: The Tanh activation function is used in the output layer of the policy network and the value network to constrain the output value to the range of (-1,1): 。 4. The micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning according to claim 3 is characterized in that: Adopting proximal strategy optimization algorithm, including: Initialize the Transformer decoder ,Strategy , value function , Experience Pool , memory length and worker threads ; If the execution round is judged Greater than the total number of rounds , then the algorithm ends, otherwise it starts in parallel sampling workers and performs both environmental and perceptual randomization; A sampling worker initializes the simulation environment and obtains the initial state , in the execution steps Less than or equal to the total number of steps When , calculate the hidden state, sample actions from the strategy, execute the action to obtain the new state and reward, get the experience tuple and store the experience tuple in the experience pool If the process ends, reset the simulation environment, otherwise execute the next step ; When the number of steps Greater than the total number of steps Enter the advantage estimation stage and traverse the experience pool Each trajectory in Start iteration to , calculate the time series error, generalized advantage estimate and value regression target in turn, and perform the optimization phase; If the optimization round Less than or equal to the total number of optimization rounds , then from the experience pool Medium sampling and small batch , save the current strategy, calculate the importance sampling ratio, pruning strategy loss, entropy reward and total loss in turn, use the Adam optimizer to perform the next optimization round , until the optimization round Greater than the total number of optimization rounds , continue to the next round .

5. The micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning according to claim 4 is characterized in that: The proximal policy optimization algorithm enables the agent to obtain the maximum expected cumulative reward in the interactive environment: The overall goal of the proximal policy optimization algorithm is to introduce the entropy reward term weighted joint optimization: State Space The state vector consists of the relative distances of the clusters , relative angle ,Location( ),size( ),speed( ) and the previous action spliced ​​together and further spliced ​​together up to Relative information of observable obstacles: Action Space Contains continuous two-dimensional vectors in the action space ; Reward Function include: The AdamW optimizer includes: in, and are the estimates of the first and second order moments, is the decay rate of the exponentially weighted average. For the The gradient of the step; Cosine annealing learning rates include: in, is the current step number, is the total number of scheduling steps; Gradient clipping includes: in represents the Euclidean norm, > 0 is a given threshold.

6. The micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning according to claim 1, characterized in that: The magnetic field control parameters include the pitch angle required to drive the three-axis Helmholtz coil system to generate a rotating magnetic field in a specific direction. and direction angle .

7. The micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning according to claim 1, characterized in that: Based on magnetic field control parameters, the micro-nano robot cluster is physically driven and makes intelligent decisions, continuously sensing new states, and repeating this process to achieve closed-loop control of the system, including: Get pre-trained weights and output real-time execution angles; Initialize the pre-allocated observation buffer, direction socket and pitch socket, YOLO detector and policy network; Execute the detection thread, acquire the image from the camera, perform YOLO target detection, and publish the bounding box coordinates to shared memory; The main control thread executes, sends the previous action based on the current working status, determines that the image is valid based on the bounding box coordinates, obtains the latest coordinate information from the shared memory, constructs the state vector based on historical data, and uses the Transformer policy network for forward reasoning. It outputs normalized actions, maps them to physical control parameters, and sends control instructions via the UDP protocol. The lower computer receives the control instructions, adjusts the coil current to generate a magnetic field to drive the cluster movement, re-acquires the new working status, and continues to execute system control.

8. A micro-nano robot swarm navigation and obstacle avoidance system based on deep reinforcement learning, based on the micro-nano robot swarm navigation and obstacle avoidance method based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: Includes CSP-3000 power supply, MyRIO controller, Copley drive, workspace, camera, three-axis Helmholtz coil, iron frame and deep learning server; The CSP-3000 power supply is connected to the Copley drive to supply power to the Copley drive; The MyRIO controller is connected to the Copley drive, the three-axis Helmholtz coil, and the deep learning server respectively; The triaxial Helmholtz coil consists of three pairs of large, mutually orthogonal coils that form a workspace. Driven by a programmable DC power supply and a dedicated amplifier, the coils are controlled by a lower-level computer program written in LabVIEW. This generates a uniform rotating magnetic field in the center of the experimental area, with adjustable direction, intensity, and rotation frequency. The iron stand is vertically installed outside the three-axis Helmholtz coil and is used to install the camera so that the camera is vertically above the three-axis Helmholtz coil; The camera is used to capture dynamic video images of the micro-nano robot cluster and obstacles in the environment in real time, and transmit the original video stream of the dynamic video images to the deep learning server through a high-speed data interface; The deep learning server is used to execute the micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the micro-nano robot cluster navigation and obstacle avoidance method based on deep reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images