Path task planning method of magnetic micro robot in complex environment

Abstract

The invention discloses a path task planning method for a magnetic micro robot in a complex environment, and relates to the technical field of micro-nano control, and the method comprises the steps: carrying out the formalized coding of a multi-target task demand through employing linear sequential logic LTL; a decision module based on a DQN neural network is constructed, a real-time environment state and an action execution result are combined and input to a multi-level reward function coded by LTL to generate a reward and punishment signal, and the reward and punishment signal is fed back to the network to guide a path optimization decision; in the network optimization process, a logic correction state transition model is introduced to increase the probability of compliance action in a weighted manner, so that the state transition process of the magnetic micro robot is shifted towards the direction meeting LTL constraint. According to the method, task logic features are extracted through bidirectional GRU, and the task logic features are embedded into the neural network as priori knowledge of decision making, so that the problem of low efficiency of processing time sequence tasks through traditional DQN is solved, and the interpretability and safety of decision making are improved through explicit logic constraints.

Description

Technical Field

[0001] This invention relates to the field of micro-nano control technology, and in particular to a path planning method for magnetic microrobots in complex environments. Background Technology

[0002] Magnetically controlled microrobots, with their non-invasive and remotely controllable characteristics, have shown great application potential in many fields. In the biomedical field, they can penetrate into the narrow spaces of blood vessels and organs to achieve precise targeted drug delivery, efficiently delivering drugs to diseased sites and reducing damage to healthy tissues. They can also be used in minimally invasive surgery to reduce surgical trauma and promote rapid postoperative recovery. In the field of micro-nano manufacturing, they can complete the precision assembly and fine manipulation of tiny parts, meeting the needs of high-precision manufacturing. In environmental monitoring, they can detect pollutants in complex and minute environments. However, in practical applications, magnetically controlled microrobots face many challenges. In complex environments, they must not only avoid dynamic obstacles in real time but also complete a series of tasks according to a specific time sequence. For example, in living organisms, it may be necessary to first collect samples at designated locations and then deliver therapeutic drugs to specific areas, which places high demands on their path planning and the timing control of task execution. When multiple microrobots collaborate to complete tasks, how to achieve efficient coordination, avoid collisions, and rationally allocate tasks has become an urgent problem to be solved.

[0003] Deep reinforcement learning provides a data-driven solution for robot control, but traditional deep reinforcement learning algorithms (such as DQN) are insufficient in multi-objective path planning and collaborative task coordination, lacking the ability to explicitly express temporal logical constraints, resulting in inefficient path planning and a tendency to get trapped in local optima. Traditional path planning algorithms (such as RRT and A*) struggle to encode tasks with temporal dependencies, failing to meet the stringent time-series requirements of tasks in biomedical scenarios such as sequential drug delivery and cell fusion. In dynamic scenarios with changing obstacle distribution or multi-robot interactions, traditional methods lack real-time perception-based policy adjustment mechanisms, making it difficult to balance the task progress and safety constraints of microrobots. Summary of the Invention

[0004] To address the aforementioned problems and technical requirements, the inventors have proposed a path planning method for magnetic microrobots in complex environments. The objectives of this invention include:

[0005] (1) Implementing time-sequential planning for multi-objective tasks: Using linear time-sequential logic (LTL) to formally encode multi-objective tasks (such as "always avoid obstacles" and "finally reach the target"), clarifying task priorities, execution order and safety rules, solving the shortcomings of traditional algorithms in handling time constraints, and improving the logic and efficiency of multi-objective path planning.

[0006] (2) Construct a collaborative decision-making mechanism for multi-magnetic microrobots: Utilize LTL to define mutual exclusion constraints and collaboration rules between robots (such as two robots arriving at the mixing point in sequence), and combine the adaptive learning capability of deep Q network (DQN) to achieve collision-free coordination and task allocation of multiple robots in dynamic environments.

[0007] (3) Enhance dynamic adaptability in complex environments: By combining the real-time environmental state perceived by the robot's sensors (such as robot position, obstacle distribution, etc.) with task logic constraints through the LTL-DQN framework, the navigation strategy is dynamically adjusted based on the reward mechanism to ensure that path planning can still be executed efficiently when there is obstacle interference or changes in task priority.

[0008] (4) Optimize the algorithm efficiency of discrete control scenarios: For the discrete state-action space characteristics of magnetic field driven systems, the LTL-DQN algorithm based on Q learning is adopted to avoid the inefficient training problem of continuous algorithms and realize the rapid convergence and real-time decision-making of micro robots under discrete control.

[0009] The technical solution of the present invention is as follows:

[0010] A path planning method for magnetic microrobots in complex environments is proposed. This method integrates Linear Temporal Logic (LTL) and Deep Quantization Learning (DQN) to construct a multi-objective path planning strategy suitable for magnetically controlled microrobots. Its core lies in transforming task logic into formal constraints and combining reinforcement learning to achieve dynamic decision-making, thus addressing the shortcomings of traditional methods in temporal planning and multi-robot collaboration. The technical architecture of this invention consists of the following core components:

[0011] (1) Task Logic Formal Encoding Layer: Linear temporal logic operators (such as G (always), F (finally), U (until)) are used to transform multi-objective tasks into structured formulas, explicitly defining task priorities and temporal constraints. For example, through... Encoding safety rules for "always avoiding obstacles" breaks down complex tasks into independent LTL sub-formulas, triggering task phase switching through state transitions. For example, a single robot task is divided into a "collection phase" and a "delivery phase," each corresponding to a different LTL formula, ensuring that the task is executed in logical order.

[0012] (2) Reinforcement Learning Decision Optimization Layer: Based on deep neural networks, the optimal action policy is learned, and decisions are made in the discrete action space through an ε-greedy policy. An experience replay buffer is used to store state transition data, and the training process is stabilized by a soft update mechanism of the target network. A multi-level reward function encoded in LTL (e.g., +20 points for completing the task, -2 points for violating constraints) guides path optimization. A DQN architecture is embedded to perform forward and backward temporal processing on the LTL formula sequence, concatenating the forward and backward hidden states to capture long-term dependencies of the task (e.g., "future goals" and "past experience"), enhancing the decision model's ability to understand complex temporal constraints.

[0013] (3) Dynamic Environment Adaptation and Collaboration Layer: Combining real-time environmental states (robot position, obstacle distribution, target position), a multi-level reward function is designed based on LTL constraints. For example, a second penalty value close to 0 (i.e., a slight penalty, such as -0.1 points) is applied when approaching the target to optimize path efficiency, and a first penalty value (i.e., a severe penalty, such as -2 points) is applied when violating the safety distance to ensure the robot's safe movement, guiding the robot to balance task progress and constraint requirements in a dynamic environment. Collaboration strategies are learned through DQN to achieve collision-free collaboration and task allocation among multiple robots in complex environments.

[0014] The beneficial technical effects of this invention are:

[0015] This method, through a three-layer architecture of "logic encoding-decision optimization-dynamic adaptation," deeply integrates the formal constraints of LTL with the adaptive learning capabilities of DQN, achieving efficient path planning for magnetically controlled microrobots in multi-objective, multi-robot scenarios. This provides a logically sound and robust solution for applications in the biomedical field, such as drug delivery and cell manipulation. Furthermore, the following core aspects of this method bring about the following technical effects:

[0016] 1. Timing feature fusion mechanism of LTL task and bidirectional GRU

[0017] This paper proposes to transform natural language tasks into LTL formulas and extract temporal feature vectors of forward history constraints and backward future goals simultaneously through bidirectional GRU, thereby solving the problem that traditional reinforcement learning cannot handle complex temporal logic.

[0018] 2. Logical Constraint Embedding Method for DQN Neural Networks

[0019] By embedding temporal feature vectors in the DQN decision-making process, task-oriented decision-making is achieved by jointly optimizing the environmental state and LTL features, breaking through the limitation of traditional DQN relying solely on environmental feedback.

[0020] 3. Quantization System of Reward Function with LTL Constraints

[0021] A three-layer dynamic reward mechanism is constructed to transform symbolic LTL constraints into numerical signals, thus solving the problem of quantitative unification between logical tasks and reinforcement learning objectives.

[0022] 4. Experience replay and state transition optimization for logical correction

[0023] A logically modified state transition model is proposed by combining priority sampling, weighting the probability of compliant actions, and optimizing the state transition under LTL constraints. Attached Figure Description

[0024] Figure 1 This is the overall framework diagram of the path task planning provided in this application.

[0025] Figure 2 This is a flowchart illustrating the path task planning method provided in this application. Detailed Implementation

[0026] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0027] One embodiment of this application provides a path planning method for a magnetic microrobot in a complex environment, the system architecture of which is as follows: Figure 1 As shown in the diagram, the overall framework diagram presents the core architecture of the LTL-DQN framework, which consists of an upper logic encoding layer, a middle decision execution layer, and a lower experience optimization layer. The logic encoding layer inputs the LTL formula sequence into the Bidirectional GRU through the LTL task encoder to generate a temporal feature vector, which is then fed into the DQN neural network. In the decision execution layer, the environmental state input inputs information such as the robot's position, obstacle and target distribution, etc., into the reward function module. Combined with the result of action execution, a reward / penalty signal is generated and fed back to the DQN neural network. In the experience optimization layer, the state transition data generated by the DQN neural network is stored in the experience replay buffer and then randomly sampled to optimize the network training. Each layer forms a closed loop through data flow, realizing the integration of temporal logic and reinforcement learning.

[0028] The overall process of this method is as follows: Figure 2 As shown, the specific steps include:

[0029] (1) Linear Temporal Logic (LTL) is used to formally encode the multi-objective task requirements, clarifying task priorities, execution order, and safety rules. In this step, starting from the task input, the "LTL Task Encoding" stage first receives the multi-objective task requirements in natural language form and transforms them into an LTL (Linear Temporal Logic) formula sequence. This step converts abstract task requirements into machine-understandable logical symbolic representations. Next, the "Bidirectional GRU" (Bidirectional Gated Recurrent Unit) processes the LTL formula sequence, extracting its temporal features and generating corresponding temporal feature vectors, providing logical-level feature support for subsequent decision-making.

[0030] (2) Construct a decision-making module based on a DQN neural network. The decision-making module selects actions based on the current environmental state and the ε-greedy policy. It combines the real-time environmental state and the results of the action execution with the input of a multi-level reward function encoded by LTL to generate reward and penalty signals, which are then fed back to the DQN neural network to guide path optimization decisions. This step is executed on the decision execution side. The "environmental state input" collects information about the agent's environment, such as the robot's own position, obstacle distribution, and target position. This information, along with the previously generated temporal feature vector, is input into the DQN neural network. At the same time, the results of the action execution are fed back to the "reward function module." Based on the real-time environmental state and rules such as whether LTL constraints are violated (e.g., -2 points for violation) or whether the task is completed (e.g., +20 points for completion), the reward and penalty signals are calculated and then passed to the DQN neural network to drive network learning and optimization.

[0031] (3) The state transition data generated by the DQN neural network (including the current environment state, the action to be performed, the immediate reward, and the next environment state, i.e., s, a, r, s') is stored in the experience replay buffer, where the action refers to the next movable position of the magnetic microrobot. Then, data is sampled from this buffer and input back into the DQN neural network to achieve iterative optimization of the network, forming a data-driven learning loop. During the optimization process, a logical correction to the state transition model is introduced to weight and increase the probability of compliant actions, causing the state transition process of the magnetic microrobot to shift in the direction that satisfies the LTL constraints.

[0032] This application focuses on the deep integration of LTL tasks and reinforcement learning. A unique logical encoding module, consisting of "LTL task encoding + bidirectional GRU," overcomes the limitations of traditional reinforcement learning that relies solely on environmental states. It introduces temporal logical constraints, allowing the agent's decisions to follow complex task rules. For example, task logic such as a robot completing operations in a specific order or consistently avoiding dangerous areas can be transformed into executable features through this module. The "DQN neural network," as the decision-making core, integrates environmental states and logical features to learn the optimal strategy in a dynamic environment, balancing environmental adaptation and task logic satisfaction. An "experience replay buffer" ensures training stability, addressing the complex characteristics of LTL tasks with multiple constraints and long sequences, preventing the agent from getting trapped in local optima due to erroneous decisions, and making the learning process more efficient and robust.

[0033] This can be further understood as follows: LTL constraint quantization is fundamental. By transforming symbolic logic into numerical rewards and penalties through a reward function, reinforcement learning algorithms can understand and learn logical tasks, solving the problem of converting symbolic tasks into numerical decisions and enabling complex logic to be "perceived" by the agent. Bidirectional temporal feature processing is a breakthrough. The Bidirectional GRU simultaneously captures historical and future constraints, handling long-distance temporal dependencies such as "what to do first, what to do later" and "cannot do something at a certain stage." Compared to unidirectional recurrent networks, it encodes task logic more accurately, ensuring decisions conform to temporal rules. Experience replay optimization is a guarantee. The buffer capacity (500,000) and sampling strategy (batch size 64) designed for LTL tasks balance sample diversity and training efficiency. In multi-constraint scenarios, this allows the agent to effectively explore and learn stably, improving training convergence speed and policy quality, supporting the framework's application in complex tasks (such as robot collaboration and industrial process control).

[0034] The following section further details the specific components of each step. In step (1), a mapping bridge from natural language tasks to numerical features is constructed. The core of this is to transform abstract task requirements into a computable formal expression through Linear Temporal Logic (LTL). Specifically, LTL is used to formally encode multi-objective task requirements, including:

[0035] First, LTL logical operators are used to break down the multi-objective task requirements into combinations of basic LTL sub-formulas, and to encode the mutual exclusion constraints and cooperation rules between magnetic microrobots. In this embodiment, the LTL formulas are based on the atomic proposition set AP, and each robot is assigned an independent LTL formula through timing operators (G-always, F-finally, U-until). For example, "the robot must always avoid obstacles and eventually reach the target" can be formalized as follows: A task decomposition algorithm is proposed, which breaks down complex tasks into combinations of basic LTL sub-formulas, such as "collect droplets first, then deliver". The U operator ensures the strictness of the action timing. It encodes mutual exclusion constraints, such as "Robot 2 must arrive at the mixing point after Robot 1," and defines the atomic propositions for this scenario:

[0036] r1_m: Robot 1 has reached the mixing point (m represents the position of the mixing point).

[0037] r2_m: Robot 2 reaches the mixing point

[0038] Its LTL formula is:

[0039]

[0040] Wherein, F(r2) -m This indicates that "Robot 2 must eventually reach the mixing point." This indicates "before robot 2 reaches the mixing point" Robot 1 must first reach the mixing point (r1) -m ).

[0041] Secondly, the encoded LTL formula sequence undergoes forward and backward temporal processing, and the forward and backward hidden states are concatenated to obtain a temporal feature vector, which is then passed to the DQN neural network. Specifically, to transform the symbolic temporal logic (such as the constraint relationships expressed by operators like "always," "until," and "finally") in the LTL formula sequence into numerical features that can be understood by the deep reinforcement learning model, this embodiment employs a bidirectional GRU to synchronously capture the temporal features in the LTL formula sequence through forward and backward information flows. The forward hidden state update mechanism is as follows:

[0042]

[0043] in, This is a forward hidden state used to capture the historical temporal constraints of the LTL formula sequence from the initial to the t-th step; As a forward reset gate, the weights in the [0,1] interval are output through the sigmoid function to control whether information unrelated to the current sub-formula in the historical hidden state is forgotten; As a forward update gate, the weights in the [0,1] interval are output through the sigmoid function to adjust the fusion ratio of new information and historical information; Forward candidate hidden states, a temporary hidden state calculated based on the current input and the reset historical information, incorporating the logical features of the current sub-formula; x t The input for step t is the sub-formula after splitting in the LTL formula sequence (such as the sub-constraints of "always avoid obstacles" or "collect first, then deliver"); The forward hidden state at step t-1 stores the historical temporal features up to the previous step; The weight matrix of the forward reset gate (corresponding to the input x respectively)t and historical hidden state (weight); This is the weight matrix for the forward update gate; The weight matrix for the forward candidate hidden states (same as above); This is a bias term for the forward reset gate, update gate, and candidate hidden state.

[0044] The forward hidden state update mechanism first uses the forward reset gate. Filter historical constraint information, input x from the current LTL subformula. t and the previous forward hidden state Together, they decide which historical constraint information related to the current LTL subformulation to retain; based on the filtered historical constraint information and the current LTL subformulation, they calculate the forward candidate hidden states. To integrate the timing logic of the current LTL sub-formula; then through the forward update gate Adjust the fusion ratio of new information and historical constraint information to determine the weights for retaining historical constraint information and current new features; finally, fuse the historical constraint information with the new features of the current LTL sub-formula proportionally to form the forward hidden state at step t. This stores all historical time-series constraints up to the present, providing historical logic support for subsequent decisions.

[0045] Backward hidden state The computational logic of the update mechanism is symmetrical to that of the forward hidden state update mechanism, only the temporal direction is reversed. For example, for an LTL sequence of collection followed by delivery, the forward reset gate references the historical hidden state of step 1 in step 2 (t=2).

[0046]

[0047] Then, in step 1 (t=1), the back reset gate needs to refer to the future hidden state in step 2 due to time reversal.

[0048]

[0049] Therefore, forward gating relies on historical information at "t-1", while backward gating relies on future information at "t+1". By reversing this temporal direction, the future target constraints in the LTL formula sequence can be captured in advance. (Reset gate) With the update gate The sigmoid function outputs weights between 0 and 1, which control the forgetting and retention of historical information. The final feature vector is a fusion of past constraints and future goals.

[0050] In step (2), the decision-making module based on the DQN neural network, based on deep Q-learning theory, approximates the state-action value function Q(s,a;θ) through the neural network. Its core iterative formula is:

[0051]

[0052] Where Q(s,a; θ) is the state-action value function, representing the cumulative reward predicted by a neural network with parameter θ when performing action a in the current state s; α is the learning rate, controlling the magnitude of each parameter update and determining the step size of the network adjustment towards the target value; r is the immediate reward, a single-step reward / penalty signal fed back from the environment (e.g., +20 points for completing the task, -2 points for violating the constraint); γ is the discount factor (range [0,1]), used to measure the current value of future rewards, with a value closer to 1 indicating greater emphasis on long-term rewards; s' represents the next state reached after performing action a; a' represents the actions (i.e., selectable path points) that can be chosen in the next state s'; θ is the parameter of the current Q network (including weights and biases); θ′ is the parameter of the target network, used to stabilize the Q-value calculation and avoid target drift caused by fluctuations in the current network parameters. Target value: r + γmax a′ Q(s′,a′;θ′) is calculated through the target network θ′, and the target drift problem during training is alleviated by the soft update mechanism θ′←τθ+(1-τ)θ′, where τ is the update coefficient.

[0053] In this embodiment, the DQN neural network adopts a three-layer fully connected structure: the input layer receives the current environment state. Temporal feature vectors output by LTL coding (i.e., bidirectional GRU) The resulting joint feature vector. This indicates that the vector belongs to d. s d-dimensional real space s h represents the feature dimension of the environmental state (i.e., the number of variables included in the environmental state); t (Integrating historical constraints and future goals of the LTL formula sequence) This indicates that the vector belongs to 2d. h d-dimensional real space h The dimension of the hidden state in a one-way GRU (the hidden state dimension of both forward GRU and backward GRU is d). h The total dimension after splicing is 2d. h The hidden layers use ReLU activation functions ranging from 1024 to 512 dimensions to handle nonlinear mappings, mitigating the curse of dimensionality. The output layer dimension is equal to the action space size, directly predicting the Q-values ​​of each action.

[0054] Furthermore, in this embodiment, a logistic feature regularization term is introduced into the loss function calculation part of the DQN neural network. If the forced decision-making process follows the LTL formula sequence constraint, then the loss function is defined as:

[0055]

[0056] Part One It is the core optimization term in reinforcement learning. By minimizing the mean squared error between the current Q-network prediction value Q(s,a;θ) and the target value y (including the cumulative estimate of immediate and future rewards), it ensures that the network learns an effective value estimate of the environment interaction; Part Two It is a logical feature regularization term, where h t The time-series feature vector encoded by LTL (containing logic such as task priority and execution order). The gradient of the Q-value function with respect to the network parameter θ is adjusted by the regularization term coefficient λ1, which forces the gradient change of the Q-value to be associated with the LTL temporal characteristics, so that the network parameter update direction follows the LTL constraint. Ultimately, the decision of the magnetic microrobot conforms to the value law of environmental interaction and strictly follows the temporal logic rules of the task.

[0057] The DQN neural network uses gradient backpropagation to ensure that the network parameter θ satisfies h. t The implicit logic is updated in the direction of the training process. The weight matrix W and bias b in the network structure are updated through the Adam optimizer, and the learning rate α adopts a cosine decay strategy to stabilize the training process.

[0058] In step (2), a multi-level reward function encoded with LTL is designed to convert symbolic logical constraints into numerical reward signals. This process specifically includes: the LTL monitoring automaton detects the satisfaction level of the encoded LTL formula sequence in real time through a state transition diagram, and calculates the reward function values ​​to form a reward / penalty signal. The reward functions include:

[0059] a. Final state task reward: When the final state constraints in the LTL formula sequence are met (such as satisfying F(target)), a first reward value (such as +20 points) is applied to quantify the task completion.

[0060] b. Constraint Violation Reward: When a fundamental timing constraint in the LTL formula sequence is violated (e.g., ... If the decision is not valid, apply a first penalty value (e.g., -2 points) to ensure compliance with the decision-making process; when the real-time environmental status feedback indicates that the target location is close, apply a second penalty value close to 0 (e.g., -0.1 points) to optimize path efficiency.

[0061] c. Environmental interaction rewards: Apply corresponding rewards and / or penalties based on real-time environmental status feedback (e.g., +1 point for normal robot movement, -5 points for collision) to maintain basic exploration motivation.

[0062] To address the reward sparsity issue in LTL tasks, the reward function also includes an intermediate reward based on feature similarity, expressed as:

[0063] R intermediate =λ2·cos(h t ,h target )

[0064] Among them, h target The baseline logical feature vector corresponding to the final state constraint in the LTL formula sequence is constructed as follows: the final state LTL sub-formula of the task (such as "delivered" F) is input into a bidirectional GRU, and its hidden state is extracted as h. target For example, for a task of "collecting droplets first and then delivering them", the final state formula is F(delivered). Encoding this formula and inputting it into a bidirectional GRU yields the hidden state h. target λ1 serves as the semantic benchmark for the "final state to be achieved"; λ2 is the weight, controlling the intensity of the influence of intermediate rewards on the overall reward and punishment signal. cos(·) represents the calculation of h. t with h target The cosine similarity is used to quantify the semantic distance between the state and the task target.

[0065] Specifically, the satisfaction of the LTL formula sequence generated by real-time detection encoding through the state transition diagram includes: the generation of the state transition diagram requires first parsing the LTL task formula of the magnetically controlled robot (such as "always avoid obstacles"). "Collect samples first, then deliver the drugs" Extracting atomic propositions (such as "obstacle avoidance") The process begins by "collecting" the data, then determining the state set based on the temporal logic relationships between propositions. This set includes initial states, satisfied states, violated states, and intermediate transition states. State transition rules are set for each proposition (e.g., transitioning from "initial state" to "satisfied state" requires satisfying the temporal constraints of all atomic propositions, while transitioning to "violation state" is triggered because the proposition is not true). When the real-time environmental state (e.g., whether an obstacle has been collided with, whether the target point has been reached) satisfies the corresponding rule, the system switches to the corresponding state to distinguish between satisfying and violating the LTL sub-formula, thereby calculating the corresponding reward function. In one possible implementation, the magnetically controlled robot "always avoids obstacles." Taking the task as an example, its state transition graph includes "normal state" (currently satisfying obstacle avoidance constraints) and "violation state" (currently colliding with an obstacle). The transition rule is: if the sensor detects no collision ( If the error is true, the system remains in the normal state. If a collision is detected (obstacle is true), the system transitions from the normal state to the violation state (irreversible). During detection, a sliding window is maintained to record the last 5 steps. The Boolean value is used to determine the normal state (satisfying the formula) if all values ​​in the window are true, and the state is determined to be false if any value is false, thus triggering the constraint violation penalty.

[0066] The total reward function provided in this embodiment is the sum of the three core rewards and intermediate rewards, as shown in the following formula:

[0067] R = R final +R penalty +R env +R intermediate

[0068] Among them, R final As the final state task reward, R penalty To constrain violations and penalties, r env Rewards for environmental interactions.

[0069] In step (3), sampling data from the experience replay buffer for iterative optimization of the network specifically includes storing state transition data tuples through a circular buffer with a capacity of N = 500,000. A random sampling strategy (based on TD error to determine sample weights, prioritizing samples more important for parameter updates and correcting for bias) is adopted to ensure sample diversity. Simultaneously, a priority sampling strategy is introduced to improve the utilization rate of key samples. The priority sampling probability formula is defined as:

[0070]

[0071] Where p(i) is the probability that the i-th sample is sampled; δ i is the Temporal Difference Error (TD) of the i-th sample, reflecting the importance of this sample to the network parameter update; || is the absolute value operator; β∈(0,1] is the bias correction coefficient, used to balance priority and sample diversity, and avoid training bias caused by over-focusing on high-priority samples.

[0072] To further ensure that the state transition process conforms to the timing constraints of LTL (avoiding training bias towards unreasonable paths due to logically violated actions in random sampling or high-priority samples), a logically corrected state transition model is proposed, whose formula is defined as:

[0073] p L (s′|s,a,h)=p(s′|s,a)·exp(λ3·h·I(a conforms to LTL))

[0074] Where, p L(s′|s,a,h) represents the probability of transitioning to the next state s′ after performing action a from the current environmental state s, considering LTL constraints; p(s′|s,a) represents the original state transition probability (based only on environmental dynamics, without considering LTL constraints); h represents the temporal feature vector output by LTL encoding; λ3 is a weighting coefficient used to adjust the strength of the influence of temporal features, such as a value of 0.8; I(a conforms to LTL) is an indicator function, which takes a value of 1 when action a satisfies the current LTL sub-formula constraint (such as "do not perform delivery action if not collected"), otherwise it is 0; exp(·) is an exponential function, which ensures that the correction term is positive, so as to weight and increase the transition probability of compliant actions.

[0075] The logically modified state transition model is used to filter actions that meet LTL constraints using an indicator function I(·), and then weights and enhances their transition probabilities by combining temporal features h, thereby shifting the state transition process towards satisfying temporal logic. For example, in the task of a magnetically controlled robot to "collect samples first and then deliver medicine," the action of "performing delivery without collecting samples" (violating the time-series logic) is modified. For the constraint), I(·) = 0, the transition probability remains the original value after correction; while for the compliant action of "collection and delivery", I(·) = 1, the positive feature of "collection completed" in the logical feature h will amplify its transition probability through the exponential term, ultimately making the training process pay more attention to the state transition that conforms to the temporal constraints, reducing the path exploration that violates the logic, and improving the temporal consistency and reliability of the magnetic microrobot task execution.

[0076] The above descriptions are merely preferred embodiments of this application, and the present invention is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included within the protection scope of the present invention.

Claims

1. A path planning method for a magnetic microrobot in a complex environment, characterized in that, The method includes: Linear Temporal Logic (LTL) is used to formally encode the requirements of multi-objective tasks, clarifying task priorities, execution order, and safety rules. A decision-making module based on a DQN neural network is constructed. The decision-making module selects actions based on the current environmental state and an ε-greedy policy. It combines the real-time environmental state and the results of the action execution with the input of an LTL-encoded multi-level reward function to generate reward and penalty signals, which are then fed back to the DQN neural network to guide path optimization decisions. The state transition data generated by the DQN neural network is stored in the experience replay buffer. Data is sampled from the experience replay buffer to iteratively optimize the network. During the optimization process, a logical correction state transition model is introduced to weight and increase the probability of compliant actions, so that the state transition process of the magnetic microrobot shifts in the direction that satisfies the LTL constraint. The state transition data includes the current environment state, the action performed, the immediate reward, and the next environment state. Environmental conditions include the position of the magnetic microrobot, the distribution of obstacles, and the target position; The action refers to the position that the magnetic microrobot can move to in the next moment.

2. The path planning method for magnetic microrobots in complex environments according to claim 1, characterized in that, The formal encoding of multi-objective task requirements using Linear Temporal Logic (LTL) includes: The multi-objective task requirements are broken down into basic LTL sub-formula combinations and encoded mutual exclusion constraints and cooperation rules between magnetic microrobots using LTL logical operators. The encoded LTL formula sequence is subjected to forward and backward temporal processing, and the forward and backward hidden states are concatenated to obtain a temporal feature vector, which is then passed to the DQN neural network.

3. The path planning method for magnetic microrobots in complex environments according to claim 2, characterized in that, The forward and backward timing processing of the encoded LTL formula sequence includes: A bidirectional GRU is employed to synchronously capture the temporal features in the LTL formula sequence through forward and backward information flows. The forward hidden state update mechanism is as follows: historical constraint information is filtered through a forward reset gate, and the current LTL sub-formula input and the previous forward hidden state jointly determine which historical constraint information related to the current LTL sub-formula should be retained; based on the filtered historical constraint information and the current LTL sub-formula, forward candidate hidden states are calculated to fuse the temporal logic of the current LTL sub-formula; the fusion ratio of new information and historical constraint information is adjusted through a forward update gate to determine the weights of retaining historical constraint information and current new features; finally, the historical constraint information and the new features of the current LTL sub-formula are fused proportionally to form the forward hidden state at step t, thereby storing all historical temporal constraints up to the present, providing historical logic support for subsequent decisions; The computational logic of the backward hidden state update mechanism is symmetrical to that of the forward hidden state update mechanism, except that the temporal direction is reversed, thereby enabling the early capture of future target constraints in the LTL formula sequence.

4. The path planning method for magnetic microrobots in complex environments according to claim 1, characterized in that, The DQN neural network adopts a three-layer fully connected structure, wherein: The input layer receives the current environment state and the temporal feature vector h output by LTL encoding. t The resulting joint feature vector; The hidden layer uses the ReLU activation function to handle non-linear mappings to alleviate dimensionality; The output layer dimension is equal to the action space size, and the Q value of each action is directly predicted; The DQN neural network, combined with a target network soft update mechanism, alleviates target drift during training and ensures that the network parameters θ satisfy h through gradient backpropagation. t The implicit logic is updated in the direction of the training process. The weight matrix and bias in the network structure are updated through the Adam optimizer. The learning rate adopts a cosine decay strategy to stabilize the training process.

5. The path planning method for magnetic microrobots in complex environments according to claim 4, characterized in that, A logistic feature regularization term is introduced into the loss function calculation part of the DQN neural network. The mandatory decision-making process follows the LTL constraints described above; Where λ1 is the coefficient of the regularization term, This represents the gradient of the Q-value function with respect to the network parameter θ.

6. The path planning method for a magnetic microrobot in a complex environment according to claim 1, characterized in that, The design of the LTL-encoded multi-level reward function includes: The satisfaction level of the LTL formula sequence generated by the encoding is detected in real time through a state transition diagram, and the reward and penalty signal is composed of the values ​​of each reward function. The reward function includes: Final state task reward: When the final state constraints in the LTL formula sequence are satisfied, a first reward value is applied to quantify the degree of task completion; Constraint violation reward: When the basic temporal constraints in the LTL formula sequence are violated, a first penalty value is applied to ensure decision compliance; when the real-time environmental status feedback indicates proximity to the target position, a second penalty value close to 0 is applied to optimize path efficiency. Environmental interaction rewards apply corresponding rewards and / or penalties based on real-time environmental status feedback to maintain basic exploration motivation.

7. The path planning method for magnetic microrobots in complex environments according to claim 6, characterized in that, The LTL-encoded multi-level reward function also includes an intermediate reward based on feature similarity, expressed as: R intermediate =λ2·cos(h t ,h target ) Among them, h t h is the temporal feature vector output by LTL encoding. target λ2 is the baseline logical feature vector corresponding to the final state constraint in the LTL formula sequence, λ2 is the weight used to control the influence strength of the intermediate reward on the overall reward and punishment signal, and cos(·) represents the calculation of h. t With the h target The cosine similarity is used to quantify the semantic distance between the state and the task target.

8. The path planning method for a magnetic microrobot in a complex environment according to claim 6, characterized in that, The real-time detection of the satisfaction of the LTL formula sequence generated by the state transition graph includes: Atomic propositions are extracted from the LTL formula sequence, and a set of states is determined based on the temporal logical relationship between the propositions, including initial states, satisfied states, violated states, and intermediate transition states. State transition rules are set for each proposition. When the real-time environment state satisfies the corresponding rule, it switches to the corresponding state to distinguish whether it satisfies or violates the LTL sub-formula, thereby calculating the corresponding reward function.

9. The path planning method for a magnetic microrobot in a complex environment according to claim 1, characterized in that, The optimization process introduces a logically modified state transition model to weightedly increase the probability of compliant actions, including: The formula for the logically modified state transition model is defined as follows: p L (s′|s,a,h)=p(s′|s,a)·exp(λ3·h·I(a conforms to LTL)) Where, p L (s′|s,a,h) represents the probability of transitioning to the next state s′ after performing action a from the current environment state s, considering the LTL constraints; p(s′|s,a) represents the original state transition probability; h represents the temporal feature vector output by the LTL encoding; λ3 represents the weighting coefficient used to adjust the intensity of the temporal feature influence; I(a conforms to LTL) is an indicator function, which takes a value of 1 when action a satisfies the current LTL sub-formula constraint, and 0 otherwise; The logically modified state transition model is used to filter out actions that meet the LTL constraints through the indicator function I(·), and then to weight and enhance the transition probability by combining the temporal feature h, so that the state transition process shifts in the direction that satisfies the temporal logic.

10. The path planning method for a magnetic microrobot in a complex environment according to claim 1, characterized in that, Sampling data from the experience replay buffer to iteratively optimize the network includes: A random sampling strategy is employed to ensure sample diversity, while a priority sampling strategy is introduced to improve the utilization rate of key samples. The probability formula for priority sampling is defined as follows: Where p(i) is the probability that the i-th sample is sampled; δ i is the TD error of the i-th sample, reflecting the importance of this sample to the network parameter update; ‖‖ is the absolute value operator; β∈(0,1] is the bias correction coefficient, used to balance priority and sample diversity.

Images