A magnetic control method and system for in vivo microrobots based on deep reinforcement learning and MPC

Abstract

This invention discloses a magnetic control method and system for in vivo microrobots based on deep reinforcement learning and MPC. The method includes: determining the control parameters of the magnetic control system; establishing an adaptive control optimization model for the in vivo microrobot magnetic control system; using soft actor-commentator reinforcement learning to complete offline policy training; and combining this with rolling time-domain optimization of model predictive control to complete online policy fusion and solution, thereby obtaining the optimal magnetic control operation strategy. This invention also provides a magnetic control system for implementing the above method, employing a three-layer hierarchical collaborative control architecture: inner-loop PID current control, middle-layer MPC magnetic field optimization, and outer-layer SAC policy decision-making. Through the deep fusion of deep reinforcement learning and model predictive control, this invention achieves high-precision, robust, safe, and controllable adaptive magnetic control for in vivo microrobots in complex environments with confined cavities, effectively meeting the practical needs for precise manipulation of in vivo microrobots.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a magnetic control method and system for an in vivo microrobot based on deep reinforcement learning and MPC. Background Technology

[0002] With the deep integration of minimally invasive medicine and robotics, in vivo microrobots, with their advantages of being non-invasive and painless, capable of autonomous movement, and covering the entire digestive tract, have become core equipment for early screening and precise diagnosis and treatment of digestive tract diseases. The core of the motion control of in vivo microrobots is an external magnetic control system, which achieves multi-degree-of-freedom movements such as forward movement, turning, levitation, and stationary position through the magnetic interaction between an external controllable magnetic field and the robot's built-in permanent magnets.

[0003] Current control methods for magnetic control systems of in vivo microrobots mainly suffer from the following technical shortcomings:

[0004] 1. Traditional control methods such as PID control and sliding mode control, although simple in structure and easy to implement, are difficult to adapt to the strong nonlinear characteristics of magnetic-force-motion coupling in microrobots in vivo. They have weak ability to suppress external disturbances such as gastrointestinal peristalsis and changes in mucus friction, resulting in low control accuracy and lag in dynamic response, which cannot meet the needs of complex diagnosis and treatment scenarios.

[0005] 2. Although Model Predictive Control (MPC) has excellent multi-constraint handling capabilities and local optimization performance, and can achieve stable trajectory tracking, it is highly dependent on the accuracy of the system dynamics model. When there is a sudden disturbance in the digestive environment or parameter perturbation in the model, the control performance will drop sharply. Moreover, its global optimization capability is insufficient and it cannot adapt to environmental changes caused by individual differences.

[0006] 3. Single deep reinforcement learning (DRL) has strong environmental adaptability and global optimization potential, and can make autonomous decisions in complex scenarios without the need for precise mathematical models. However, it has problems such as slow training convergence, poor online real-time performance, and large fluctuations in control commands. In addition, it is easy to break through the physical safety constraints of the system, resulting in robot motion instability. Therefore, it cannot be directly applied to clinical diagnosis and treatment scenarios with extremely high safety requirements.

[0007] 4. Existing DRL and MPC collaborative control technologies have not yet formed a mature application solution in the field of magnetic control of in vivo microrobots. The existing collaborative mechanism cannot adapt to the strong coupling nonlinearity, multiple engineering constraints, high real-time performance and high safety requirements of magnetic control systems, and cannot solve the core pain point that it is difficult to balance "control accuracy, dynamic response and robust safety". Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a magnetic control method and system for in vivo microrobots based on deep reinforcement learning and MPC, which addresses the shortcomings of existing technologies. By achieving complementary advantages of DRL and MPC through a hierarchical collaborative control architecture, this invention solves the technical problem that existing magnetic control methods cannot simultaneously achieve accuracy, response, robustness and safety, and realizes high-precision, high-stability and adaptive magnetic control of in vivo microrobots in the complex environment of the digestive tract.

[0009] To solve the above problems, the present invention adopts the following technical solution:

[0010] First, this invention proposes a magnetic control method for in vivo microrobots based on deep reinforcement learning and MPC, comprising the following steps:

[0011] S1. Determine the parameters of the magnetic control of the in vivo microrobot, including: external magnetic control array hardware parameters, in vivo microrobot dynamic parameters, soft actor-commentator SAC reinforcement learning training parameters, model predictive control (MPC) optimization parameters, control constraint thresholds, cost function weight coefficients, and number of scheduling periods.

[0012] S2. Establish an adaptive control optimization model for the magnetic control system of the in vivo microrobot. The variables of the optimization model are the magnetic field control sequence and coil current command of the in vivo microrobot within the scheduling cycle. The optimization objective is to achieve the best overall performance of the in vivo microrobot with the highest trajectory tracking accuracy, the lowest motion risk, and the smoothest control action.

[0013] S3. Based on the high-fidelity simulation of the motion conditions of the in vivo microrobot, the offline strategy training is completed by using soft actor-commentator SAC reinforcement learning, and the online strategy fusion solution is completed by combining MPC rolling time domain optimization to obtain the optimal magnetic control operation strategy of the in vivo microrobot.

[0014] Preferably, the adaptive control optimization model of the in vivo microrobot magnetic control system in step S2 includes a multi-dimensional composite cost function, specifically including:

[0015] (1) Posture Cost The expression for constraining the pose tracking error of a microrobot within the body is:

[0016]

[0017] In the formula, for The actual pose of the microrobot inside the body at any given time. For the target pose, The pose error threshold, This is the traction weighting coefficient;

[0018] (2) Risk Costs The expression is:

[0019]

[0020] In the formula, The total number of risk states. These are risk state quantities, including limits on attitude angle, limits on magnetic field strength, and collision risk values. As a risk threshold, This is the risk penalty weighting coefficient;

[0021] (3) Smoothing cost The expression is:

[0022]

[0023] In the formula, For discrete time step index, To control the total time step of the sequence, for Control your actions at all times. for Control your actions at all times. To control the incremental input vector (specifically referred to in this formula) ), For vectors transpose, It is a positive semidefinite weight matrix. This is the result of the multiplication operation between the weight matrix and the input vector. This is the weight matrix;

[0024] (4) Target Cost This is used to stimulate microrobots within the body to reach the target site, and the expression is:

[0025]

[0026] In the formula, For the target to reach the threshold, The target reward weighting coefficient;

[0027] (5) Total cost function Let be the weighted sum of the costs of each sub-cost, expressed as:

[0028]

[0029] In the formula, , , , These are the weighting coefficients for attitude cost, risk cost, smoothing cost, and target cost, respectively.

[0030] The constraints in the adaptive control optimization model of the in vivo microrobot magnetic control system are as follows:

[0031] Magnetron array current constraint:

[0032]

[0033] In the formula, for The coil current at any given moment, , These are the minimum and maximum amplitude values ​​of the coil current. The rate of change of current, This represents the upper limit of the rate of change of current.

[0034] Magnetic field strength constraint:

[0035]

[0036] In the formula, for Magnetic induction intensity in the working area at any time. This represents the safe upper limit for magnetic flux density.

[0037] Motion state constraints of in vivo microrobots:

[0038]

[0039] In the formula, for The speed of movement of the microrobot inside the body at all times. , For velocity constraint boundaries, For attitude angle, , This serves as the safety constraint boundary for attitude angles;

[0040] Charge state constraints for in vivo microrobots:

[0041]

[0042] In the formula, for The state of charge of the battery in the micro-robot inside the body at all times. , These are the minimum and maximum allowable states of charge.

[0043] Preferably, the offline training phase in step S3 includes:

[0044] Set the SAC reinforcement learning training parameters and initialize the SAC training framework, which includes a policy network, a value network and a corresponding target network, as well as an experience replay pool.

[0045] The following training process is iteratively executed until a preset termination condition is reached: Based on the current policy network output control action, action selection is performed using path integral method or model predictive control to obtain the optimal control action; it is determined whether the optimal control action satisfies the constraint conditions. If not, a preset penalty fitness value is assigned; if so, the action is executed, the next state and reward value from the simulation environment feedback are obtained, and the experience samples are stored in the experience replay pool; a batch of samples are sampled from the experience replay pool to update the parameters of the policy network and the value network; based on the updated policy network, the fitness of the current policy is calculated, and the fitness is determined based on the total cost function and constraint violation cases.

[0046] Repeat the above training process until the preset number of training rounds is completed or the convergence condition is met, and use the policy network with the best fitness as the output of the optimal SAC policy.

[0047] Preferably, the specific steps for initializing the SAC training framework are as follows:

[0048] Construct an Ensemble SAC policy network, which contains N parallel Actor subnetworks. Each subnetwork adopts a fully connected neural network structure. The input is the motion state of the in vivo microrobot and environmental parameters, and the output is the Gaussian distribution mean and variance of the control action.

[0049] Construct a Critic dual Q network, which consists of two sets of Q networks with identical structures and a corresponding target Q network. The input is a state-action pair, and the output is the action value. The parameters of the target Q network are initialized to be the same as those of the main Q network.

[0050] Initialize the experience replay pool, set the maximum capacity of the replay pool, and clear historical sample data;

[0051] Initialize the initial variance and attenuation coefficient of the exploration noise, and set the noise attenuation rules during training.

[0052] Preferably, step S3 combines MPC rolling time-domain optimization to complete the online strategy fusion solution, and the specific steps are as follows:

[0053] Initialize the scheduling period index =1, determine the initial motion state of the in vivo microrobot and the initial parameters of the magnetocontrol array;

[0054] Real-time acquisition of the current motion state and environmental parameters of the microrobot within the body is fed into the trained SAC policy network, which outputs a Gaussian distribution of the control actions. ;in, To enhance the state quantity of learning, To increase the amount of physical activity involved in learning, The mean of the SAC strategy, The variance of the SAC strategy;

[0055] Using the MPC optimization module, with the current state of the in vivo microrobot as the initial value, and based on the magnetic-force-motion coupled forward dynamics model, the robot's trajectory in the future prediction time domain is predicted. Under the given constraints, rolling time domain optimization is completed, and the prior Gaussian distribution of the control action is output. ,in The mean of the MPC strategy, The variance of the MPC strategy;

[0056] The mixed policy distribution is obtained through Bayesian posterior fusion. ,in:

[0057]

[0058]

[0059] The final control action is sampled from the hybrid strategy distribution, calculated as the current command of each coil of the magnetic control array, and driven by the inner loop PID closed-loop control to update the motion state of the micro robot inside the body.

[0060] make = +1, judgment If the number of scheduling periods is less than the total number of scheduling periods, repeat the above steps; otherwise, end the scheduling cycle and output the optimal magnetic control strategy for the entire period.

[0061] Preferably, the objective function for the MPC rolling time-domain optimization is:

[0062]

[0063] In the formula, To predict the time domain, To control the time domain, To predict pose, For reference trajectory pose, To control the quantity, Here is the pose error weight matrix. To control the incremental weight matrix.

[0064] Preferably, the steps for establishing the magnetic-force-motion coupled forward dynamics model of the in vivo microrobot are as follows:

[0065] A magnetic interaction model was established to calculate the magnetic gradient force and magnetic torque of the external magnetic field on the built-in permanent magnet of the microrobot.

[0066] A force model of the in vivo microrobot was established, and the net external force and net external torque of the robot were obtained by integrating magnetic gradient force, gravity, intestinal friction, fluid resistance and intestinal peristalsis disturbance.

[0067] A 6-DOF kinematic and dynamic model of an in vivo microrobot is established based on the Newton-Euler equations, serving as the forward prediction model for MPC and the simulation environment model for SAC training.

[0068] Meanwhile, this invention proposes an in vivo microrobot magnetic control system based on deep reinforcement learning and model predictive control collaboration, comprising an external magnetic control array module, an in vivo microrobot body, a state perception module, and a hierarchical collaborative control module, wherein:

[0069] The external magnetic control array module adopts a multi-degree-of-freedom Helmholtz coil array structure, receives current commands from the hierarchical collaborative control module, and generates a controllable gradient magnetic field and a rotating magnetic field to provide driving force and steering torque for the in vivo microrobot.

[0070] The in vivo microrobot body has a built-in permanent magnet, micro-sensing unit, image acquisition unit and wireless communication unit, which are used to complete multi-degree-of-freedom movement in the digestive tract under the drive of an external magnetic field.

[0071] The state perception module consists of a built-in micro-sensing unit and an external positioning unit. It is used to collect the position, attitude angle, movement speed, surrounding magnetic field strength and digestive tract environmental parameters of the in vivo micro-robot in real time, and transmit the state information to the hierarchical collaborative control module.

[0072] The hierarchical collaborative control module has a built-in SAC reinforcement learning submodule, MPC optimization submodule, and PID current closed-loop submodule, which are used to execute the method described in this invention and output control commands to drive the external magnetic control array module to work.

[0073] The hierarchical collaborative control module includes:

[0074] The inner-loop PID current control submodule uses an incremental PID algorithm combined with feedforward compensation to achieve fast and robust tracking of single-coil current and power drive, and to suppress hardware non-ideal characteristics and coupling disturbances between coils.

[0075] The mid-level MPC optimization submodule has a built-in forward dynamics model of magnetic-force-motion coupling of the in vivo microrobot, which is used for the inverse mapping from the coil group to the spatial magnetic field and gradient, and then to the torque of the in vivo microrobot, to complete the rolling time domain optimization under multiple engineering constraints.

[0076] The outer SAC reinforcement learning submodule has an embedded Ensemble SAC policy network, which contains multiple sets of parallel-trained Actor subnetworks for learning prior information about the digestive tract environment. It outputs reference actions and cost term weight correction values ​​online to realize policy injection into MPC.

[0077] The internal microrobot has a body size of Φ10mm-Φ12mm and a length of 25mm-30mm. The built-in permanent magnet is a neodymium iron boron cylindrical permanent magnet, and the magnetization direction is along the robot's axis. The positioning accuracy of the state perception module is ≤0.05mm, the attitude angle acquisition accuracy is ≤0.05°, and the sampling frequency is ≥100Hz.

[0078] The present invention adopts the above technical solution and has the following technical effects compared with the prior art:

[0079] (1) The present invention constructs a three-layer hierarchical collaborative control architecture of “inner loop PID-middle layer MPC-outer layer SAC”, realizes the target decoupling of coil current closed loop and magnetic field to torque, and implements high-speed current tracking, multi-constraint rolling optimization and environmental adaptive strategy decision-making in layers, which not only ensures the real-time performance and control accuracy of the system, but also greatly improves the adaptive capability to complex environments.

[0080] (2) This invention achieves the complementary advantages of the two control methods through the deep collaboration of SAC reinforcement learning and MPC: the outer SAC algorithm learns the prior information of the environment, provides global policy reference and weight scheduling for MPC, and makes up for the defects of MPC's insufficient global optimization and weak anti-disturbance ability; the middle MPC completes real-time constraint correction through rolling optimization, which solves the problems of large fluctuations in DRL control commands and easy breach of safety constraints, so that the system has both the environmental adaptability of DRL and the constraint controllability of MPC.

[0081] (3) The present invention designs a multi-dimensional composite cost function that takes into account pose tracking accuracy, motion safety, control stability and target arrival efficiency. Through the segmented cost design, the reinforcement learning strategy is guided to converge to the optimal control strategy that meets the needs of clinical diagnosis and treatment, which greatly improves the clinical applicability of the system.

[0082] (4) This invention uses Bayesian probability fusion to achieve the deployment of a hybrid strategy of SAC and MPC. The optimal weighting of the two strategies is achieved through Gaussian distribution posterior fusion, which not only retains the global optimization of the SAC strategy, but also ensures the constraint satisfaction of the MPC strategy. It has strong robustness to model error and external disturbance, and can be adapted to the differences in digestive tract anatomy and complex diagnosis and treatment scenarios of different patients.

[0083] (5) The control method and system of the present invention can realize high-precision, low-delay and strong robust control of the 6 degrees of freedom of the in vivo microrobot. The trajectory tracking error is significantly reduced compared with the traditional control method, the dynamic response speed is greatly improved, and it can still maintain stable operation under complex working conditions such as gastrointestinal peristalsis and fluid disturbance. It provides core technical support for the in vivo microrobot to move from the laboratory to clinical application. Attached Figure Description

[0084] Figure 1 This invention relates to a schematic diagram of the overall process of a magnetic control method for in vivo microrobots based on deep reinforcement learning and MPC.

[0085] Figure 2 This is a schematic diagram of the SAC reinforcement learning offline training process involved in the present invention.

[0086] Figure 3 This is a schematic diagram of the online hybrid strategy deployment process of SAC and MPC involved in this invention. Detailed Implementation

[0087] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0088] Example 1: This example proposes a magnetic control method for in vivo microrobots based on deep reinforcement learning and MPC, which is divided into the following... Figure 1 The three core steps shown are described in detail below:

[0089] S1. Determine the parameters of the magnetic control of the in vivo microrobot, including: external magnetic control array hardware parameters, in vivo microrobot dynamic parameters, soft actor-commentator SAC reinforcement learning training parameters, model predictive control (MPC) optimization parameters, control constraint thresholds, cost function weight coefficients, and scheduling period number; core hardware parameters are shown in Table 1; basic control parameters are shown in Table 2; and constraint and cost core parameters are shown in Table 3.

[0090] Table 1 Core Hardware Parameters

[0091]

[0092] Table 2 Basic Control Parameters

[0093]

[0094] Table 3. Constraints and Costs: Core Parameters

[0095]

[0096] S2. Establish an adaptive control optimization model for the magnetic control system of the in vivo microrobot. The variables of the optimization model are the magnetic field control sequence and coil current command of the in vivo microrobot within the scheduling cycle. The optimization objective is to achieve the best overall performance of the in vivo microrobot with the highest trajectory tracking accuracy, the lowest motion risk, and the smoothest control action.

[0097] S3. Based on the high-fidelity simulation of the motion conditions of the in vivo microrobot, the offline strategy training is completed by using soft actor-commentator SAC reinforcement learning, and the online strategy fusion solution is completed by combining MPC rolling time domain optimization to obtain the optimal magnetic control operation strategy of the in vivo microrobot.

[0098] Specifically, the adaptive control optimization model of the in vivo microrobot magnetic control system described in step S2 includes a multi-dimensional composite cost function, specifically including:

[0099] (1) Posture Cost The expression for constraining the pose tracking error of a microrobot within the body is:

[0100]

[0101] In the formula, for The actual pose of the microrobot inside the body at any given time. For the target pose, The pose error threshold, This is the traction weighting coefficient;

[0102] (2) Risk Costs The expression is:

[0103]

[0104] In the formula, The total number of risk states. These are risk state quantities, including limits on attitude angle, limits on magnetic field strength, and collision risk values. As a risk threshold, This is the risk penalty weighting coefficient;

[0105] (3) Smoothing cost The expression is:

[0106]

[0107] In the formula, For discrete time step index, To control the total time step of the sequence, for Control your actions at all times. for Control your actions at all times. To control the incremental input vector (specifically referred to in this formula) ), For vectors transpose, It is a positive semidefinite weight matrix. This is the result of the multiplication operation between the weight matrix and the input vector. This is the weight matrix;

[0108] (4) Target Cost This is used to stimulate microrobots within the body to reach the target site, and the expression is:

[0109]

[0110] In the formula, For the target to reach the threshold, The target reward weighting coefficient;

[0111] (5) Total cost function Let be the weighted sum of the costs of each sub-cost, expressed as:

[0112]

[0113] In the formula, , , , These are the weighting coefficients for attitude cost, risk cost, smoothing cost, and target cost, respectively.

[0114] The constraints in the adaptive control optimization model of the in vivo microrobot magnetic control system are as follows:

[0115] Magnetron array current constraint:

[0116]

[0117] In the formula, for The coil current at any given moment, , These are the minimum and maximum amplitude values ​​of the coil current. The rate of change of current, This represents the upper limit of the rate of change of current.

[0118] Magnetic field strength constraint:

[0119]

[0120] In the formula, for Magnetic induction intensity in the working area at any time. This represents the safe upper limit for magnetic flux density.

[0121] Motion state constraints of in vivo microrobots:

[0122]

[0123] In the formula, for The speed of movement of the microrobot inside the body at all times. , For velocity constraint boundaries, For attitude angle, , This serves as the safety constraint boundary for attitude angles;

[0124] Charge state constraints for in vivo microrobots:

[0125]

[0126] In the formula, for The state of charge of the battery in the micro-robot inside the body at all times. , These are the minimum and maximum allowable states of charge.

[0127] like Figure 2 As shown, the specific steps for solving the adaptive control optimization model of the in vivo microrobot magnetic control system described in step S3 are as follows:

[0128] S3.1: Set the training parameters for SAC deep reinforcement learning and MPPI, including: experience replay pool size, learning rate, discount factor γ, soft update coefficient τ, maximum training epochs, number of samples, and temperature parameter λ; S3.2: Initialize the SAC training framework, including the Actor policy network, the Critic double-Q network, and the corresponding target Actor network and target Critic network, and initialize the parameters of each network and the experience replay pool; S3.3: Training epoch index. Initialize to 0, that is, let =0; The core parameters for SAC-MPPI training are shown in Table 4:

[0129] Table 4 Core parameters for SAC-MPPI training

[0130]

[0131] S3.4: Order = +1, calculate the... Policy fitness of generational networks, environment state index Initialize to 1, that is, set =1;

[0132] S3.5: For the current simulation environment, the first... each state The Actor network outputs multiple path distributions. Based on environmental parameters, a large number of control sequences are sampled using MPPI. Each sequence corresponds to a predicted trajectory. The optimal control action is obtained by weighted selection using path integrals. ;

[0133] S3.6: Determine the action Does the control action meet the internal safety constraints? If the control action does not meet the safety constraints, then calculate the policy fitness corresponding to that action according to the formula. Then proceed to step S3.8; if the control action satisfies the safety constraints, execute the action. The simulation environment provides feedback on the next state. The reward value R is derived from the total cost function. Continue with step S3.7;

[0134] S3.7: Sample Store the data in the experience replay pool, randomly sample a batch of samples from the experience replay pool, minimize the Q-value prediction error to update the Critic double Q network parameters, update the Actor network parameters by maximizing the action value, and update the target Actor network and target Critic network parameters using a soft update method.

[0135] S3.8: Based on the updated policy network, simulate and verify the control policy under the current state, calculate the trajectory tracking accuracy and constraint satisfaction rate, and calculate the first... The policy fitness corresponding to each state:

[0136]

[0137] In the formula, The penalty coefficient for exceeding the limit is given in advance. To constrain the probability of exceeding the limit;

[0138] S3.9: Determine whether the fitness calculation for all environmental states in the current training round has been completed, i.e., determine... Is it equal to the maximum number of states? If it is less than the maximum number of states, then let = +1, and jump to step S3.5; otherwise, continue to the next step S3.10;

[0139] S3.10: Determine if the maximum number of training rounds has been reached, i.e., determine the training round index. Is it equal to the maximum number of training rounds? If the number of training rounds is equal to the maximum, proceed to step S3.11; otherwise, update the network's noise exploration parameters based on fitness and proceed to step S3.4.

[0140] S3.11: The policy network with the highest fitness after training is selected as the optimal SAC policy output and enters the online running phase.

[0141] The SAC training framework initialization steps described in step S3.2 are as follows:

[0142] S3.2.1: Construct an Ensemble SAC policy network, which contains N parallel Actor subnetworks. Each subnetwork adopts a fully connected neural network structure. The input is the motion state and environmental parameters of the in vivo microrobot, and the output is the Gaussian distribution mean and variance of the control action.

[0143] S3.2.2: Construct a Critic dual Q network, which includes two sets of Q networks with identical structures and a corresponding target Q network. The input is a state-action pair, and the output is the action value. The parameters of the target Q network are initialized to be the same as those of the main Q network.

[0144] S3.2.3: Initialize the experience replay pool, set the maximum capacity of the replay pool, and clear historical sample data;

[0145] S3.2.4: Initialize the initial variance and attenuation coefficient of the exploration noise, and set the noise attenuation rules during training.

[0146] like Figure 3 As shown, in step S3, the online operation phase, the steps for solving the optimal control strategy based on the collaboration of SAC and MPC are as follows:

[0147] S3.11: Initialize the scheduling period index =1, determine the initial motion state of the in vivo microrobot and the initial parameters of the magnetocontrol array;

[0148] S3.12: The state perception module collects the current motion state and environmental parameters of the microrobot in real time, inputs them into the trained SAC policy network, and outputs a Gaussian distribution of the control actions. ,in The mean of the SAC strategy, The variance of the SAC strategy;

[0149] S3.13: The MPC optimization module uses the current state of the in vivo microrobot as the initial value. Based on the magneto-force-motion coupled forward dynamics model, it predicts the robot's trajectory in the future prediction time domain. Under constraints of magnetic array current, magnetic field strength, motion state, and charge state, it completes the rolling time domain optimization and outputs the prior Gaussian distribution of the control action. ,in The mean of the MPC strategy, Let be the MPC policy variance; s be the state variables in reinforcement learning; and a be the action variables in reinforcement learning. The mean of the SAC strategy, The variance of the SAC strategy;

[0150] S3.14: Obtain the mixed policy distribution through Bayesian posterior fusion ,in:

[0151]

[0152]

[0153] S3.15: The final control action is sampled from the hybrid strategy distribution, calculated as the current command of each coil of the magnetic control array, and driven by the magnetic control array through the inner loop PID closed-loop control to update the motion state of the micro robot in the body;

[0154] S3.16: Order = +1, judgment If the number of scheduling periods is less than the total number of scheduling periods, proceed to step S3.12; otherwise, end the scheduling cycle and output the optimal magnetic control strategy for the entire time period.

[0155] The objective function for the MPC rolling time-domain optimization described in step S3.13 is:

[0156]

[0157] In the formula, To predict the time domain, To control the time domain, To predict pose, For reference trajectory pose, To control the quantity, Here is the pose error weight matrix. To control the incremental weight matrix.

[0158] The in vivo microrobot magnetic-force-motion coupled forward dynamics model described in step S3.13 is established as follows:

[0159] S3.13.1: Establish a magnetic interaction model to calculate the magnetic gradient force and magnetic torque of the external magnetic field on the built-in permanent magnet of the microrobot:

[0160]

[0161] In the formula, It has a built-in permanent magnet magnetic moment. The magnetic flux density of the external magnetic field;

[0162] S3.13.2: Establish a force model for the in vivo microrobot, and obtain the net external force and net external torque of the robot by integrating magnetic gradient force, gravity, intestinal friction, fluid resistance and intestinal peristalsis disturbance force;

[0163] S3.13.3: Based on the Newton-Euler equations, a 6-DOF kinematic and dynamic model of the in vivo microrobot is established as the forward prediction model of MPC and the simulation environment model for SAC training.

[0164] The hierarchical collaborative control module described in this embodiment adopts a three-layer architecture: "inner-loop PID current control—middle-layer MPC magnetic field optimization—outer-layer SAC strategy decision-making." The functions of each layer cooperate to achieve high-precision, high-stability, and adaptive control of the in vivo microrobot magnetic control system. The inner-loop PID current control submodule enables rapid tracking of single-coil current and suppresses hardware disturbances; the middle-layer MPC optimization submodule performs magnetic field optimization under multiple constraints, ensuring system operational safety; and the outer-layer SAC reinforcement learning submodule learns prior information about the environment, improving the system's adaptability to the complex digestive tract environment.

[0165] This embodiment achieves adaptive optimization control of the in vivo microrobot magnetic control system through deep collaboration between SAC reinforcement learning and MPC. Under the premise of meeting the safety constraints of clinical diagnosis and treatment, it significantly improves the trajectory tracking accuracy, dynamic response speed and environmental robustness of the in vivo microrobot.

[0166] Example 2: This invention also provides an in vivo microrobot magnetic control system based on deep reinforcement learning and model predictive control, comprising an external magnetic control array module, an in vivo microrobot body, a state perception module, and a hierarchical collaborative control module, wherein:

[0167] The external magnetic control array module adopts a multi-degree-of-freedom Helmholtz coil array structure, receives current commands from the hierarchical collaborative control module, and generates a controllable gradient magnetic field and a rotating magnetic field to provide driving force and steering torque for the in vivo microrobot.

[0168] The in vivo microrobot body has a built-in permanent magnet, micro-sensing unit, image acquisition unit and wireless communication unit, which are used to complete multi-degree-of-freedom movement in the digestive tract under the drive of an external magnetic field.

[0169] The state perception module consists of a built-in micro-sensing unit and an external positioning unit. It is used to collect the position, attitude angle, movement speed, surrounding magnetic field strength and digestive tract environmental parameters of the in vivo micro-robot in real time, and transmit the state information to the hierarchical collaborative control module.

[0170] The hierarchical collaborative control module has a built-in SAC reinforcement learning submodule, MPC optimization submodule, and PID current closed-loop submodule, which are used to execute the method described in this invention and output control commands to drive the external magnetic control array module to work.

[0171] The hierarchical collaborative control module includes:

[0172] The inner-loop PID current control submodule uses an incremental PID algorithm combined with feedforward compensation to achieve fast and robust tracking of single-coil current and power drive, and to suppress hardware non-ideal characteristics and coupling disturbances between coils.

[0173] The mid-level MPC optimization submodule has a built-in forward dynamics model of magnetic-force-motion coupling of the in vivo microrobot, which is used for the inverse mapping from the coil group to the spatial magnetic field and gradient, and then to the torque of the in vivo microrobot, to complete the rolling time domain optimization under multiple engineering constraints.

[0174] The outer SAC reinforcement learning submodule has an embedded Ensemble SAC policy network, which contains multiple sets of parallel-trained Actor subnetworks for learning prior information about the digestive tract environment. It outputs reference actions and cost term weight correction values ​​online to realize policy injection into MPC.

[0175] The internal microrobot has a body size of Φ10mm-Φ12mm and a length of 25mm-30mm. The built-in permanent magnet is a neodymium iron boron cylindrical permanent magnet, and the magnetization direction is along the robot's axis. The positioning accuracy of the state perception module is ≤0.05mm, the attitude angle acquisition accuracy is ≤0.05°, and the sampling frequency is ≥100Hz.

[0176] It should be noted that the processing flow of Embodiment 2 corresponds to the specific steps of the method provided in Embodiment 1 of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in Embodiment 1 of the present invention.

[0177] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0178] The specific implementation schemes described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific implementation schemes of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of the present invention should fall within the scope of protection of the present invention.

Claims

1. A magnetic control method for in vivo microrobots based on deep reinforcement learning and MPC, characterized in that, Includes the following steps: S1. Determine the parameters of the magnetic control of the in vivo microrobot, including: external magnetic control array hardware parameters, in vivo microrobot dynamic parameters, soft actor-commentator SAC reinforcement learning training parameters, model predictive control (MPC) optimization parameters, control constraint thresholds, cost function weight coefficients, and number of scheduling periods. S2. Establish an adaptive control optimization model for the magnetic control system of the in vivo microrobot. The variables of the optimization model are the magnetic field control sequence and coil current command of the in vivo microrobot within the scheduling cycle. The optimization objective is to achieve the best overall performance of the in vivo microrobot with the highest trajectory tracking accuracy, the lowest motion risk, and the smoothest control action. S3. Based on the high-fidelity simulation of the motion conditions of the in vivo microrobot, the offline policy training is completed by using soft actor-commentator SAC reinforcement learning, and the online policy fusion solution is completed by combining MPC rolling time domain optimization to obtain the optimal magnetic control operation strategy of the in vivo microrobot.

2. The method according to claim 1, characterized in that, The adaptive control optimization model of the in vivo microrobot magnetic control system described in step S2 includes a multi-dimensional composite cost function, specifically including: (1) Posture Cost The expression is: ； In the formula, for The actual pose of the microrobot inside the body at any given time. For the target pose, The pose error threshold, This is the traction weighting coefficient; (2) Risk Costs The expression is: ； In the formula, The total number of risk states. These are risk state quantities, including limits on attitude angle, limits on magnetic field strength, and collision risk values. As a risk threshold, This is the risk penalty weighting coefficient; (3) Smoothing cost The expression is: ； In the formula, For discrete time step index, To control the total time step of the sequence, for Control your actions at all times. for Control your actions at all times. To control the incremental input vector (specifically referred to in this formula) ), For vectors transpose, It is a positive semidefinite weight matrix. This is the result of the multiplication operation between the weight matrix and the input vector. This is the weight matrix; (4) Target Cost The expression is: ； In the formula, For the target to reach the threshold, The target reward weighting coefficient; (5) Total cost function The expression is: ； In the formula, , , , These are the weighting coefficients for attitude cost, risk cost, smoothing cost, and target cost, respectively. The constraints in the adaptive control optimization model of the in vivo microrobot magnetic control system are as follows: Magnetron array current constraint: ； In the formula, for The coil current at any given moment, , The minimum and maximum amplitude values ​​of the coil current. The rate of change of current, This represents the upper limit of the rate of change of current. Magnetic field strength constraint: ； In the formula, for Magnetic induction intensity in the working area at any time. This represents the safe upper limit for magnetic flux density. Motion state constraints of in vivo microrobots: ； In the formula, for The speed of movement of the microrobot inside the body at all times. , For velocity constraint boundaries, For attitude angle, , This serves as the safety constraint boundary for attitude angles; Charge state constraints for in vivo microrobots: ； In the formula, for The state of charge of the battery in the micro-robot inside the body at all times. , These represent the minimum and maximum allowable states of charge.

3. The method according to claim 2, characterized in that, The offline training phase described in step S3 includes: Set the SAC reinforcement learning training parameters and initialize the SAC training framework, which includes a policy network, a value network and a corresponding target network, as well as an experience replay pool. The following training process is iteratively executed until a preset termination condition is reached: Based on the current policy network output control action, action selection is performed using path integral method or model predictive control to obtain the optimal control action; it is determined whether the optimal control action satisfies the constraint conditions. If not, a preset penalty fitness value is assigned; if so, the action is executed, the next state and reward value from the simulation environment are obtained, and the experience samples are stored in the experience replay pool; a batch of samples are sampled from the experience replay pool to update the parameters of the policy network and the value network; based on the updated policy network, the fitness of the current policy is calculated, and the fitness is determined based on the total cost function and constraint violation cases. Repeat the above training process until the preset number of training rounds is completed or the convergence condition is met, and output the policy network with the best fitness as the optimal SAC policy.

4. The method according to claim 3, characterized in that, The specific steps for initializing the SAC training framework are as follows: Construct an Ensemble SAC policy network, which contains N parallel Actor subnetworks. Each subnetwork adopts a fully connected neural network structure. The input is the motion state of the in vivo microrobot and environmental parameters, and the output is the Gaussian distribution mean and variance of the control action. Construct a Critic dual-Q network, which consists of two sets of Q networks with identical structures and a corresponding target Q network. The input is a state-action pair, and the output is the action value. The parameters of the target Q network are initialized to be the same as those of the main Q network. Initialize the experience replay pool, set the maximum capacity of the replay pool, and clear historical sample data; Initialize the initial variance and attenuation coefficient of the exploration noise, and set the noise attenuation rules during training.

5. The method according to claim 3, characterized in that, Step S3 combines MPC rolling time-domain optimization to complete the online strategy fusion solution. The specific steps are as follows: Initialize the scheduling period index =1, determine the initial motion state of the in vivo microrobot and the initial parameters of the magnetocontrol array; Real-time acquisition of the current motion state and environmental parameters of the microrobot within the body is fed into the trained SAC policy network, which outputs a Gaussian distribution of the control actions. ;in, To enhance the state quantity of learning, To enhance the amount of physical activity involved in learning, The mean of the SAC strategy, The variance of the SAC strategy; Using the MPC optimization module, with the current state of the in vivo microrobot as the initial value, and based on the magnetic-force-motion coupled forward dynamics model, the robot's trajectory in the future prediction time domain is predicted. Under the given constraints, rolling time domain optimization is completed, and the prior Gaussian distribution of the control action is output. ,in The mean of the MPC strategy, The variance of the MPC strategy; The mixed policy distribution is obtained through Bayesian posterior fusion. ,in: ； ； The final control action is sampled from the hybrid strategy distribution, calculated as the current command of each coil of the magnetic control array, and driven by the inner loop PID closed-loop control to update the motion state of the micro robot inside the body. make = +1, judgment If the number of scheduling periods is less than the total number of scheduling periods, repeat the above steps; otherwise, end the scheduling cycle and output the optimal magnetic control strategy for the entire period.

6. The method according to claim 5, characterized in that, The objective function for the MPC rolling time-domain optimization is: ； In the formula, To predict the time domain, To control the time domain, To predict pose, For reference trajectory pose, To control the quantity, Here is the pose error weight matrix. To control the incremental weight matrix.

7. The method according to claim 5, characterized in that, The steps for establishing the magnetic-force-motion coupled forward dynamics model of the in vivo microrobot are as follows: A magnetic interaction model was established to calculate the magnetic gradient force and magnetic torque of the external magnetic field on the built-in permanent magnet of the microrobot. A force model of the in vivo microrobot was established, and the net external force and net external torque of the robot were obtained by integrating magnetic gradient force, gravity, intestinal friction, fluid resistance and intestinal peristalsis disturbance. A 6-DOF kinematic and dynamic model of an in vivo microrobot is established based on the Newton-Euler equations, serving as the forward prediction model for MPC and the simulation environment model for SAC training.

8. A magnetic control system for an in vivo microrobot based on deep reinforcement learning and MPC, characterized in that, It includes an external magnetic control array module, an internal microrobot body, a state perception module, and a hierarchical collaborative control module, among which: The external magnetic control array module adopts a multi-degree-of-freedom Helmholtz coil array structure, receives current commands from the hierarchical collaborative control module, and generates a controllable gradient magnetic field and a rotating magnetic field to provide driving force and steering torque for the in vivo microrobot. The in vivo microrobot body has a built-in permanent magnet, micro-sensing unit, image acquisition unit and wireless communication unit, which are used to complete multi-degree-of-freedom movement in the digestive tract under the drive of an external magnetic field. The state perception module consists of a built-in micro-sensing unit and an external positioning unit. It is used to collect the position, attitude angle, movement speed, surrounding magnetic field strength and digestive tract environmental parameters of the in vivo micro-robot in real time, and transmit the state information to the hierarchical collaborative control module. The hierarchical collaborative control module has a built-in SAC reinforcement learning submodule, MPC optimization submodule, and PID current closed-loop submodule, which are used to execute the method described in any one of claims 1-7 and output control commands to drive the external magnetic control array module to work.

9. The system according to claim 8, characterized in that, The hierarchical collaborative control module includes: The inner-loop PID current control submodule uses an incremental PID algorithm combined with feedforward compensation to achieve fast and robust tracking of single-coil current and power drive, and to suppress hardware non-ideal characteristics and coupling disturbances between coils. The mid-level MPC optimization submodule has a built-in forward dynamics model of magnetic-force-motion coupling of the in vivo microrobot, which is used for the inverse mapping from the coil group to the spatial magnetic field and gradient, and then to the torque of the in vivo microrobot, to complete the rolling time domain optimization under multiple engineering constraints. The outer SAC reinforcement learning submodule has an embedded Ensemble SAC policy network, which contains multiple sets of parallel-trained Actor subnetworks for learning prior information about the digestive tract environment. It outputs reference actions and cost term weight correction values ​​online to realize policy injection into MPC.

10. The system according to claim 8, characterized in that, The internal microrobot has a body size of Φ10mm-Φ12mm and a length of 25mm-30mm. The built-in permanent magnet is a neodymium iron boron cylindrical permanent magnet, and the magnetization direction is along the robot's axis. The positioning accuracy of the state perception module is ≤0.05mm, the attitude angle acquisition accuracy is ≤0.05°, and the sampling frequency is ≥100Hz.

Images