Robot cell microinjection operation method based on dynamic equilibrium game mechanism reinforcement learning

Abstract

The application discloses a robot cell microinjection operation method based on dynamic balance game mechanism reinforcement learning. The method comprises the following steps: training an injection operation decision model using a total loss function; the total loss function comprises a loss of a PPO reinforcement learning network, an entropy regularization term and a state prediction loss of a state prediction network; during the robot cell microinjection operation process, injection environment images are collected in real time, input into an instance segmentation network model, and environment mask images are obtained; the injection needle tip real-time coordinates and cell mask images are obtained according to the environment mask images; a feature extraction network receives the cell mask images, a PPO reinforcement learning network receives the feature extraction results and the injection needle tip real-time coordinates, and the expected position of the injection needle tip is obtained as feedback information to control the robot. The application can perform safe and high-precision zebrafish embryo cell microinjection operation in a real scene, and provides technical support for the fields of gene editing and cell breeding.

Description

Technical Field

[0001] This invention relates to a robotic cell micromanipulation method, which relates to the fields of robotics and biological cell manipulation, and specifically to a robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism. Background Technology

[0002] In the intersection of robotics and biological cell manipulation, cell microinjection is a core technology in life science research (such as gene editing, embryo engineering, and drug screening). It requires the precise injection of minute amounts of biological reagents (such as DNA, RNA, and drug molecules) into cells while ensuring the preservation of cell structure and activity. Currently, while robotic cell microinjection technology has gradually replaced traditional manual operations, significant technical bottlenecks remain when facing complex application scenarios. Traditional vision systems often use monocular cameras to acquire two-dimensional images, only obtaining planar position information of cells and failing to reconstruct the depth distribution of cells in three-dimensional space (such as the equatorial position of the cell nucleus). This makes it difficult to precisely control the insertion depth of the injection needle (too deep or too shallow an insertion will cause the unconstrained cell nucleus to rotate and damage it). Although reinforcement learning is effective in addressing the high trial-and-error costs of such real-world experiments, existing methods often encounter difficulties in aligning the physical parameters of real cells with those of simulated cells when handling cell injection scenarios, leading to problems such as "Sim to Real" difficulties or significant discrepancies in results. Therefore, it is necessary to develop a new reinforcement learning algorithm to reduce the modeling error between real and simulated environments, thereby improving the success rate of cell injection operations in real-world scenarios. Summary of the Invention

[0003] To address the problems existing in the background technology, this invention provides a robotic cell microinjection method based on reinforcement learning using a dynamic equilibrium game mechanism. This invention aims to use the YOLO-segmentation model to reduce the dimensionality of cell image features and, by constructing a cell state prediction module, fully utilize the operational dataset of the simulation environment to reduce the gap in the model's transfer from simulation to real-world scenarios, effectively improving the success rate of cell microinjection.

[0004] The technical solution adopted in this invention is: The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism provided by this invention includes the following steps: Step S1: Construct and train an instance segmentation network model; the input of the instance segmentation network model is the injection environment image, and the output is the corresponding environment mask image.

[0005] In step S1, the real cell puncture dataset and the simulated cell puncture dataset are combined into a sample dataset, and the sample dataset is used to train the instance segmentation network; in the sample dataset, each group of sample data mainly consists of an injection environment image and a corresponding environment mask image.

[0006] The simulated cell puncture dataset was obtained through simulation using a cell microinjection simulation model. The robotic cell microinjection simulation model includes a cell model, an injection needle model, and a forceps model. The cell model includes a cell nucleus and a cell membrane. The cell nucleus consists of an animal pole and a vegetative pole, which are rigidly connected and subjected to buoyancy within the cell membrane, resulting in vertical force balance. The injection needle model is configured to move freely along the X, Y, and Z axes. The needle tip diameter is set to 5-10 μm, the needle tube length to 400-500 μm, and the material parameters match the physical properties of quartz glass. The forceps model is configured to clamp and fix the cell model, and the cell nucleus is configured to undergo damped rotation when punctured by the injection needle model.

[0007] Preferably, the cell to be punctured is a zebrafish embryonic cell. In the robotic cell microinjection simulation model, the cell model is a zebrafish embryonic cell model. The rotational damping coefficient between the cell vegetative pole and the cell membrane is set to 0.05~0.1 N·s / m, the lateral friction coefficient is set to 20.0 N·s / m, the rotational damping coefficient is set to 0.05 N·s / m, and the contact stiffness and damping are set to 3000 N / m and 300 N·s / m, respectively.

[0008] Preferably, when training the instance segmentation network model, the weighted calculation result of the bounding box loss, class loss, and distribution focus loss is used as the total loss. The total loss function of the instance segmentation network model is expressed as: In the formula, The total loss of the instance segmentation network model. , , These are the bounding box loss, class loss, and distribution focus loss, respectively. , , These are the bounding box loss, class loss, and distribution focus loss relative to the final loss, respectively. The weights; For complete intersection and union, This is a parameter used to measure the consistency of the aspect ratio of the predicted bounding box. This is the balance coefficient; and These are the true category label and the predicted category label, respectively. It predicts the probability distribution of the bounding box and the ground truth bounding box across four coordinates. It is the focusing parameter.

[0009] Step S2: Construct and train the injection operation decision model. The injection operation decision model includes a feature extraction network, a PPO reinforcement learning network, and a state prediction network.

[0010] In step S2, the injection operation decision model is trained using the simulated cell puncture dataset, and each set of simulated cell puncture data also includes real cell state parameters.

[0011] Specifically, in each round of training of the injection operation decision model, the feature extraction network receives the cell mask image and performs feature extraction, the PPO reinforcement learning network processes the feature extraction results and the real-time coordinates of the injection needle tip to obtain the expected position of the injection needle tip, and the state prediction network processes the feature extraction results to obtain the predicted cell state parameters.

[0012] Specifically, the total loss function of the injection operation decision model includes the loss and entropy regularization term of the PPO reinforcement learning network and the state prediction loss of the state prediction network.

[0013] The cell state parameters include the cell's state angle, rotational angular velocity, and rotational angular acceleration; the state estimation loss is expressed as: In the formula, , , These are the predicted values ​​for the state angle, rotational angular velocity, and rotational angular acceleration, respectively. , , These are the true values ​​of the state angle, rotational angular velocity, and rotational angular acceleration, respectively, in the real cell state parameters. This represents the mean square error between the predicted and true values.

[0014] The entropy regularization term includes an entropy regularization coefficient. During the training of the injection operation decision model, the entropy regularization coefficient is adjusted according to the following formula: Where T represents the total number of training rounds, It is the total task loss in the t-th round of training. It is the initial total task loss for each round of training.

[0015] Furthermore, an entropy conservation constraint is introduced during the training process. Specifically, this constraint states that the rate of change of policy entropy is proportional to the rate of change of the total task loss, expressed as: In the formula, Represents policy entropy The rate of change of the target over time. This represents the rate of change of total task loss. This indicates the loss of the PPO reinforcement learning network. This represents the state prediction loss. These are the weights of the state prediction loss. This represents the constraint coefficient.

[0016] In each round of training of the injection operation decision model, after introducing the entropy conservation constraint, the target rate of change of policy entropy over time is calculated based on the loss of the PPO reinforcement learning network and the state prediction loss. Based on the actual rate of change of policy entropy over time and the target rate of change of policy entropy over time, the action exploration range of the next round of training is adjusted: if the actual rate of change of policy entropy over time is less than the target rate of change, the action exploration range of the next round of training is increased; if the actual rate of change of policy entropy over time is greater than the target rate of change, the action exploration range of the next round of training is decreased.

[0017] Step S3: During the robotic cell microinjection operation, real-time images of the injection environment are acquired and input into the instance segmentation network model to obtain an environment mask image. Based on the environment mask image, the real-time coordinates of the injection needle tip and the cell mask image are obtained. The feature extraction network of the injection operation decision model receives the cell mask image and performs feature extraction to obtain the feature extraction result. The PPO reinforcement learning network receives the feature extraction result and the real-time coordinates of the injection needle tip to obtain the expected position of the injection needle tip at the next moment. The robot controller uses the expected position of the injection needle tip as feedback information and moves the injection needle tip by controlling the movement of the robot, thereby realizing closed-loop control of the robotic cell microinjection operation.

[0018] Specifically, the environmental mask image includes annotation information for the cell membrane, the animal pole of the cell, the plant pole, as well as the injection needle and tweezers. In step S3, the real-time coordinates of the injection needle tip are obtained based on the annotation information of the injection needle in the environmental mask image.

[0019] Specifically, the cell mask image includes annotation information for the cell membrane, the animal pole of the cell, and the vegetative pole of the cell.

[0020] In summary, this invention trains two networks—state prediction and policy value learning—in parallel through a dynamic equilibrium game mechanism. This guides the model to learn policies while narrowing the gap between the simulation and real domains, overcoming the limitations of traditional monocular vision. It solves the problems of needle tip z-axis alignment difficulties and cell nucleus rotation damage in cell microinjection, achieving high puncture success rate (excellent performance in zebrafish embryo injection tasks) and low cell damage. It effectively reduces the migration error from simulation to reality, enabling safe and high-precision zebrafish embryo cell microinjection operations in real-world scenarios, providing technical support for gene editing, cell breeding, and other fields.

[0021] The beneficial effects of this invention are: 1. The method proposed in this invention extracts features of the operation object through an instance segmentation network, which reduces the information dimension of the input state prediction network, effectively improves the prediction accuracy of the state prediction network, and provides a stability guarantee for subsequent reinforcement learning.

[0022] 2. This invention learns the state parameters and operation strategies of the operation object simultaneously by constructing a state prediction network and a multi-task learning mechanism. Compared with the method of relying on rule calculation and separate training, it can have better robustness and learning efficiency, and can ensure good results for different operation objects, thereby improving the stability of subsequent reinforcement learning.

[0023] 3. The method of this invention reduces the difference in visual observation between simulation and real-world scenarios through instance segmentation networks, effectively improving the model transfer error from simulation to real-world scenarios and ensuring that strategies learned in the simulation environment can be easily transferred to real-world scenarios.

[0024] In summary, by introducing a dynamic equilibrium game mechanism, the problems of excessive discrepancies between simulation and reality and high strategy deployment costs in microinjection tasks are solved. This algorithm demonstrates a high success rate in zebrafish embryo cell injection tasks and effectively reduces the internal deflection of the cell nucleus during injection. This invention enables safe and low-damage cell injection operations in real-world scenarios and is expected to further advance the field of cell breeding. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the method of the present invention; Figure 2This is a schematic diagram of the overall structure of the robot cell injection model of the present invention. The upper part is the training process in the simulation scenario: the simulation image is masked by YOLO-segmentation, and then after passing through the feature extraction module, the policy head outputs the action, the value head outputs the value, and the pose head outputs the pose. Finally, the training is completed through the co-optimization of PPO loss and pose loss. The lower part is the inference process in the actual scenario: after the real scene image is masked by YOLO-segmentation, the action instructions are generated by feature extraction and each output head, and then the robot is driven to perform the operation by RobotSDK.

[0026] Figure 3 This is a schematic diagram of a robot microinjection simulation scenario constructed during a specific implementation of the present invention; Figure 4 This is a diagram illustrating the changes in the loss function and evaluation metrics during the training process of the YOLO-segmentation module. Figure 5 This is a schematic diagram illustrating the changes in the loss function during the training process, where, Figure 5 (a) is a schematic diagram showing the change in task success rate with training rounds during model training and testing. Figure 5 (b) is a schematic diagram showing the change of the average reward value obtained by the strategy during the model training and testing process with the number of training rounds; Figure 6 This is a schematic diagram illustrating the process of controlling a robot to perform cell injection during a specific implementation of the present invention. Detailed Implementation

[0027] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] This invention provides a robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism.

[0029] like Figure 1 As shown, the robotic cell microinjection method provided by this invention includes the following steps: Step S1: Construct and train the instance segmentation network model. The input to the instance segmentation network model is an injection environment image, i.e., a microscopic image containing cells, injection needles, and tweezers, and the output is the corresponding environment mask image. The environment mask image contains annotation information for the cell membrane, animal pole, vegetative pole, injection needle, and tweezers.

[0030] In step S1, the real cell puncture dataset and the simulated cell puncture dataset are combined into a sample dataset, and the sample dataset is used to train the instance segmentation network. In the sample dataset, each group of sample data mainly consists of an injection environment image and a corresponding environment mask image.

[0031] The simulated cell puncture dataset was obtained through simulation using a cell microinjection simulation model. Specifically, the Pybullet physics engine was used to construct a robotic cell microinjection simulation environment. This environment includes a robotic cell microinjection simulation model, which mainly consists of a cell model, an injection needle model, and a forceps model. The cell model includes a nucleus and a cell membrane. The nucleus is composed of an animal pole and a vegetative pole, which are rigidly connected. It experiences buoyancy within the cell membrane, is in vertical equilibrium, and maintains a relatively stable center of mass. The injection needle model is configured to move freely along the X, Y, and Z axes. The needle tip diameter is set to 5-10 μm, the needle length to 400-500 μm, and the material parameters are matched to the physical properties of quartz glass. The forceps model is configured to hold and fix the cell model, and the cell nucleus is configured to undergo damped rotation when punctured by the injection needle model.

[0032] Preferably, the cells to be punctured are zebrafish embryonic cells. In the robotic cell microinjection simulation model, the cell model is a zebrafish embryonic cell model. To simulate the cell nucleus rotation phenomenon that occurs during puncture, the rotational damping coefficient between the cell plant pole and the cell membrane is set to 0.05~0.1 N·s / m, the lateral friction coefficient is set to 20.0 N·s / m to ensure sufficient tangential force is generated upon contact, the rotational damping coefficient is set to 0.05 N·s / m to simulate easy rotation characteristics, and the contact stiffness and damping are set to 3000 N / m and 300 N·s / m, respectively, to simulate rigid contact.

[0033] In step S1, when training the instance segmentation network model, the weighted calculation result of the bounding box loss, class loss, and distribution focus loss is used as the total loss. The total loss function of the instance segmentation network model is expressed as: In the formula, The total loss of the instance segmentation network model. , , These are the bounding box loss, class loss, and distribution focus loss, respectively. , , These are the bounding box loss, class loss, and distribution focus loss relative to the final loss, respectively. The weights; The perfect intersection-over-union (CIU) ratio is used to evaluate the matching degree between the predicted bounding box and the ground truth bounding box. This is a parameter used to measure the consistency of the aspect ratio of the predicted bounding box. This is the balance coefficient; and These are the true category label and the predicted category label, respectively. It predicts the probability distribution of the bounding box and the ground truth bounding box across four coordinates. It is the focusing parameter.

[0034] Step S2: Construct and train the injection operation decision model. For example... Figure 2 As shown, the injection operation decision model includes a feature extraction network, a PPO reinforcement learning network, and a state prediction network. In each training iteration of the injection operation decision model, the feature extraction network receives the cell mask image and performs feature extraction. The PPO reinforcement learning network processes the feature extraction results and the real-time coordinates of the injection needle tip to obtain the desired position of the injection needle tip. The state prediction network performs cell state prediction processing on the feature extraction results to obtain predicted cell state parameters. The total loss function of the injection operation decision model includes the loss and entropy regularization term of the PPO reinforcement learning network, as well as the state prediction loss of the state prediction network.

[0035] In step S2, the injection operation decision model is trained using a simulated cell puncture dataset. Each set of simulated cell puncture data in the simulated cell puncture dataset also includes real cell state parameters.

[0036] Cell state parameters include the cell's state angle, rotational angular velocity, and rotational angular acceleration; the state estimation loss is expressed as: In the formula, , , These are the predicted values ​​for the state angle, rotational angular velocity, and rotational angular acceleration, respectively. , , These are the true values ​​of the state angle, rotational angular velocity, and rotational angular acceleration obtained from the simulation environment, respectively, representing the real cell state parameters. This represents the mean square error between the predicted and true values.

[0037] Furthermore, an entropy conservation constraint is introduced during the training process in step S2. Specifically, the entropy conservation constraint states that the rate of change of policy entropy is proportional to the rate of change of the total task loss, expressed as: In the formula, the left side Represents policy entropy The rate of change of the target over time represents the "speed of change in the degree of exploration", right side. This represents the rate of change of the total task loss (i.e., the total loss excluding entropy regularization). This indicates the loss of the PPO reinforcement learning network. This represents the state prediction loss. These are the weights of the state prediction loss, representing the "speed of overall task optimization". This represents the constraint coefficient, which is a fixed value set in advance based on experience.

[0038] In each round of training of the injection operation decision model, the target rate of change of policy entropy over time is calculated based on the loss and state prediction loss of the PPO reinforcement learning network. The action exploration amplitude for the next round of training is adjusted based on the actual rate of change of policy entropy over time and the target rate of change of policy entropy over time in the current round. Specifically, if the actual rate of change of policy entropy over time is less than the target rate of change, it indicates that the current exploration level is insufficient, and the action exploration amplitude for the next round of training (i.e., the standard deviation σ of the Gaussian distribution in the PPO reinforcement learning network) is increased. If the actual rate of change of policy entropy over time is greater than the target rate of change, it indicates that the current exploration level is excessive, and the action exploration amplitude for the next round of training (i.e., the standard deviation σ of the Gaussian distribution in the PPO reinforcement learning network) is decreased.

[0039] The entropy regularization term includes the entropy regularization coefficient; during the training of the injection operation decision model, the entropy regularization coefficient is adjusted according to the following formula: Where T represents the total number of training rounds, in this embodiment of the invention, T=300. It is the total task loss in the t-th round of training, 1 300, It is the initial total task loss for each round of training.

[0040] Step S3: During the robotic cell microinjection operation, real-time images of the injection environment are acquired and input into the trained instance segmentation network model to obtain an environment mask image. Based on the annotation information of the injection needle in the environment mask image, the real-time coordinates of the injection needle tip are obtained. The annotation information of the cell membrane, animal pole, and vegetative pole are combined to form a cell mask image. For example... Figure 3As shown, the feature extraction network of the injection operation decision model receives the cell mask image and performs feature extraction to obtain the feature extraction result. The PPO reinforcement learning network receives the feature extraction result and the real-time coordinates of the injection needle tip to obtain the expected position of the injection needle tip at the next moment. The robot controller uses the expected position of the injection needle tip as feedback information and moves the injection needle tip by controlling the movement of the robot, thereby realizing the closed-loop control of the robot cell microinjection operation.

[0041] Specific embodiments of the present invention are as follows: Example This embodiment specifically includes the following steps: Step 1: Construct a robotic cell microinjection simulation environment using the Pybullet physics engine. The simulation environment includes a zebrafish embryonic cell model (containing a nucleus and cell membrane, with the nucleus consisting of an animal pole and a vegetal pole), an injection needle model, and a forceps model. The injection needle model is configured to move freely along the X, Y, and Z axes, the forceps model is configured to hold and fix the zebrafish embryonic cell model, and the nucleus, which is composed of the animal and vegetal poles, is configured to rotate with damping when punctured by the injection needle model.

[0042] In step one, the zebrafish embryonic cell puncture model is constructed based on real-world zebrafish embryonic cells. The cell membrane and forceps are fixed in place to prevent slippage during cell puncture. The animal pole and vegetative pole models of the cell are rigidly connected to the zebrafish embryonic cell membrane. To simulate the nuclear rotation phenomenon during puncture, the rotational damping coefficient between the vegetative pole and the zebrafish embryonic cell membrane is set to 0.05~0.1 N·s / m. The lateral friction coefficient is 20.0 N·s / m to ensure sufficient tangential force upon contact, and the rotational damping coefficient is 0.05 N·s / m to simulate easy rotation. The contact stiffness and damping are set to 3000 N / m and 300 N·s / m, respectively, to simulate rigid contact. The needle tip diameter of the injection needle model is set to 5~10 μm, the needle tube length is set to 400~500 μm, and the material parameters match the physical properties of quartz glass. Step 2: Collect real cell puncture datasets and simulated cell puncture datasets. Use these two datasets as training samples to train the YOLO-segmentation instance segmentation network. The real cell puncture dataset contains microscopic images of real zebrafish embryonic cells and corresponding annotations (i.e., mask images) of the zebrafish embryonic cell membrane, animal pole, vegetative pole, injection needle, and tweezers. The simulated cell puncture dataset contains images of zebrafish embryonic cell models collected in the simulation environment constructed in Step 1 and corresponding annotations (i.e., mask images) of the animal and vegetative pole models. After training, save the YOLO-segmentation instance segmentation network parameters for subsequent instance segmentation operations on the animal and vegetative poles of the cells.

[0043] In step two, at least 1,000 images of real cell puncture datasets and at least 5,000 images of simulated cell puncture datasets are collected. An instance segmentation model is used for segmentation task training, and the training data is configured based on the COCO format (including categories such as animal pole, plant pole, needle, and tweezers). The AdamW optimizer is used during training, with the initial learning rate set to 0.01 and the final learning rate decaying to 0.01 times the initial value. The learning rate is scheduled using the cosine annealing method.

[0044] Specific loss function It consists of three parts: bounding box loss, class loss, and distribution focus loss. in, , , These are the bounding box loss, class loss, and distribution focus loss, respectively. , , It is their relative to the final loss Weight; The perfect intersection-union ratio (MIU) is used to evaluate the degree of matching between the predicted bounding box and the ground truth bounding box. Used to measure the consistency of the aspect ratio of the frame. This is the balance coefficient; and These are the true category label and the predicted category label; It represents the probability distribution of the predicted bounding box and the ground truth bounding box across the four coordinates. It is the focusing parameter; The training rounds were set to 300, with an early stop mechanism enabled (a patience value of 100 rounds, meaning training was terminated if the validation metrics did not improve within 100 rounds). Mosaic data augmentation was disabled in the last 10 rounds for fine-tuning. The input image size was 1280×1280, the batch size was 16, and the model was initialized using pre-trained weights to accelerate convergence.

[0045] Step 3: Construct a “state prediction-reinforcement learning” collaborative training framework: Using a simulated cell puncture set with nucleus orientation ground truth data, the state prediction network and the PPO reinforcement learning agent share a feature extraction module through a multi-task loss function, and the state prediction accuracy and injection strategy performance are simultaneously optimized through a policy entropy regularization term, ultimately constructing an injection needle injection strategy in a simulated environment.

[0046] Specifically, the model is divided into three modules: perception, prediction, and decision-making. Each module sequentially processes data to complete inference, and the overall system can quickly output results to meet real-time operation requirements. The YOLO-segmentation instance segmentation network trained in step two serves as the perception module. It receives real-time video frames at 1280×1280 resolution, extracts multi-scale features from the image through the YOLO-segmentation network, identifies and outputs the positions and mask information of the animal pole, plant pole, cell membrane, injection needle, and tweezers, and then filters these information using a threshold to obtain accurate results. The prediction module takes the cell membrane and pole masks output by the perception module as input, and performs feature extraction through a multi-layer convolutional neural network (CNN) and a multi-layer perceptron (MLP) network to obtain the feature extraction results.

[0047] The state prediction network calculates the cell's angle, rotational angular velocity, and rotational angular acceleration using a multilayer perceptron and a fully connected network. It also optimizes parameters through error correction to ensure accurate prediction. The decision module combines feature extraction results with the real-time coordinates of the injection needle tip and constructs a policy network head and a value network head through a multilayer perceptron (MLP) network. The policy network head generates displacement commands for the injection needle in the X, Y, and Z directions, while the value network head outputs auxiliary calculations of the loss to optimize the policy. Finally, the output displacement commands are processed to generate robot control commands.

[0048] The decision-making module adopts a multimodal network architecture, which includes a PPO network and a state prediction network. The PPO network includes two parallel sub-networks: Policy-Head and Value-Head.

[0049] The feature extraction network mainly consists of 5 convolutional layers, 5 batch normalization (BN) layers, and 5 activation layers. The input of the first convolutional layer is a mask image (dimensions C×H×W, where C is the number of channels, H=1280, W=1280). The first convolutional layer receives and processes the mask image, with 64 kernels, a kernel size of 3×3, a stride of 1, and SAME padding. The second convolutional layer receives and processes the output of the first convolutional layer, with 128 kernels, a kernel size of 3×3, a stride of 2, and SAME padding. The first activation layer receives and processes the output of the second convolutional layer, and the first normalization layer receives and processes the output of the first activation layer. The third convolutional layer receives and processes the output of the first normalization layer, with 256 kernels, a kernel size of 3×3, a stride of 1, and SAME padding. The second activation layer receives and processes the mask image. The output of the third convolutional layer is processed, and the second normalization layer receives and processes the output of the second activation layer. The fourth convolutional layer receives and processes the output of the second normalization layer, with 512 kernels, a kernel size of 3×3, a stride of 2, and SAME padding. The third activation layer receives and processes the output of the fourth convolutional layer, and the third normalization layer receives and processes the output of the third activation layer. The fifth convolutional layer receives and processes the output of the third normalization layer, with 1024 kernels, a kernel size of 3×3, a stride of 1, and SAME padding. The fourth activation layer receives and processes the output of the fifth convolutional layer, and the fourth normalization layer receives and processes the output of the fourth activation layer. The fifth activation layer receives the output of the fourth normalization layer, and after global average pooling (GAP) to compress the spatial dimension, a 512-dimensional image feature vector is output as the feature extraction result.

[0050] The activation layers all use the GELU activation function, the momentum parameter of the batch normalization layer is set to 0.9, and the weight decay coefficient is set to 1e-5.

[0051] The PPO reinforcement learning network comprises a policy sub-network and a value sub-network. The policy sub-network adopts a concatenated architecture of "feature mapping-branch output," with its input connected to the output of the fifth activation layer of the feature extraction network, receiving a 512-dimensional image feature vector. This feature vector undergoes high-order semantic mapping and dimensionality transformation via the policy branch (Policy Net). This policy branch consists of two fully connected layers (linear layers) alternately connected with activation layers. The first fully connected layer has 256 neurons, receiving the 512-dimensional input features and mapping them to a 256-dimensional feature vector. After introducing a non-linear expression through the GELU activation function, it is input to the second fully connected layer (128 neurons), further transforming it into a 128-dimensional Latent Policy feature. The policy output layer uses a dual-branch fully connected layer design. The first branch is a linear layer with 3 neurons, and the output is constrained to [-0.1, 0.1] mm by the Tanh activation function. The first branch accurately outputs the mean parameters of the displacement motion in the X, Y, and Z directions; the second branch is also configured with a linear layer of 3 neurons, using the Softplus activation function to ensure the non-negativity of the output, corresponding to the standard deviation parameters of the displacement motion in the X, Y, and Z directions, providing a probability distribution basis for motion sampling.

[0052] The value subnetwork and policy subnetwork share a feature input link method. Its input end receives a 512-dimensional image feature vector output by the feature extraction network. The value branch performs feature depth mining and representation learning. The value branch adopts a bilinear layer structure consistent with the policy branch. The first fully connected layer (256 neurons) maps the 512-dimensional input features into a 256-dimensional vector. After processing by the GELU activation function, it is fed into the second fully connected layer (128 neurons), which outputs 128-dimensional latent value features. The value output layer adopts a single-neuron fully connected layer (linear layer) structure. Without introducing an additional activation function, it directly outputs the value estimate scalar of the current state, providing a value benchmark for the calculation of the advantage function and policy optimization of the PPO algorithm.

[0053] The input of the state prediction network is connected to the output of the fifth activation layer of the feature extraction network, receiving a 512-dimensional image feature vector. The main body of the network consists of three fully connected layers (linear layers), three GELU activation layers, and two batch normalization layers, alternately connected in series. The first fully connected layer has 256 neurons, mapping the 512-dimensional input features to a 256-dimensional intermediate feature vector. After introducing a non-linear expression through the GELU activation function, it is fed into the first batch normalization layer (momentum parameter 0.9, weight decay coefficient 1e-5) for feature distribution normalization. The normalized feature vector is then input into the second fully connected layer (128 neurons). The features are further converted into 128-dimensional feature vectors. After processing by the GELU activation function, the feature stability is optimized by a second batch normalization layer (momentum parameter 0.9, weight decay coefficient 1e-5). The output of the third batch normalization layer is connected to a third fully connected layer (64 neurons), which compresses the 128-dimensional features into 64-dimensional higher-order features. After the feature representation ability is enhanced by the GELU activation function, the features are input to the prediction output layer. The prediction output layer is a single-branch fully connected layer (linear layer) with 3 neurons. It has no additional activation function and directly outputs 3 continuous values, which correspond to the predicted values ​​of cell state angles. Predicted rotational angular velocity Predicted value of rotational angular acceleration Participate in multi-task collaborative optimization.

[0054] The multi-task loss function consists of policy loss, value loss, and state prediction loss. Figure 2 It consists of Pose-loss and policy entropy regularization term.

[0055] In the multi-task loss function, state prediction loss Represented as: In the formula, , , These are the predicted values ​​for state angle, rotational angular velocity, and rotational angular acceleration. , , These are the truth values ​​obtained through the simulation environment. The mean squared error was calculated directly for the predicted and true values.

[0056] In the multi-task loss function, the policy entropy regularization term includes the policy entropy. And the entropy regularization coefficient α. Policy entropy. Represented as: in, The entropy represents the exploration strategy; the negative sign is used to maximize the entropy to encourage exploration.

[0057] During training, the rate of change of policy entropy is forced to be proportional to the rate of change of total task loss to ensure dynamic matching between the "exploration level" and the "task optimization progress." The linkage formula between the "entropy conservation constraint" and the rate of change of task loss is expressed as follows: In the formula, the left side Represents policy entropy The rate of change of the target over time represents the "speed of change in the degree of exploration", right side. This represents the rate of change of the total task loss (i.e., the total loss excluding entropy regularization). This represents the loss of the PPO reinforcement learning network. Figure 2 (PPO-loss in the middle) This represents the state prediction loss. These are the weights of the state prediction loss, representing the "speed of overall task optimization". This represents the constraint coefficient, which is a fixed value set in advance based on experience.

[0058] The multi-task loss function is expressed as: in, This represents the loss of the PPO reinforcement learning network. , It is the weight of value loss in the overall loss of the PPO reinforcement learning network. is the entropy regularity coefficient.

[0059] In this embodiment, the initial value of α is set to 0.01, and during the training process, as the loss decreases, it eventually becomes 0.2.

[0060] In this embodiment, the variation law of α is set as follows: Among them, the maximum value of the entropy regularization coefficient minimum value The total number of training rounds is T=300. It is the total task loss in each round of training ( For the current training round, 1 300), It is the initial total task loss for each round of training.

[0061] The prediction module and the decision module together form the injection operation decision model. The training process uses the AdamW optimizer, and the initial learning rate is set to... The learning rate eventually decays to 0.05 times the initial value, and the learning rate is scheduled using the cosine annealing algorithm. The policy ratio clipping threshold is set to 0.2, the value loss weight coefficient is 1.0, the initial entropy regularization coefficient is set to 0.01 to balance exploration and exploitation, and the gradient clipping threshold is set to 1.0 to prevent gradient explosion.

[0062] Step 4: The robot operates autonomously, using real-time acquired video frames as input. The YOLO-segmentation network is used to obtain mask information for zebrafish embryonic cell membranes, animal poles, vegetative poles, injection needles, and tweezers. The needle tip position is obtained through the needle mask information. The feature extraction results corresponding to the mask information of the zebrafish embryonic cell membrane, animal pole, and vegetative pole are then processed through a state prediction network to obtain the current cell state parameters. Based on the current cell state parameters and the injection strategy, the robot further obtains the displacement commands of the injection needle in the x, y, and z directions, and generates robot control commands to execute the operation. In step five, when the robot performs the injection operation, a trajectory smoothing strategy is adopted to smooth the desired movement position commands of the injection needle generated by the reinforcement learning agent along the X-axis, Y-axis, and Z-axis directions. Adjust the position command to a smoothed state. The expression is adjusted to: in, For smoothing coefficients, These represent the actual positions of the injection needle along the X, Y, and Z axes within the previous control time step; the robot controller receives the smoothed position command ( After that, the actuator is driven by the position closed-loop control algorithm (PID control) to make the injection needle move precisely to the target position.

[0063] like Figure 3 As shown, the dynamic scenario of robotic cell microinjection used in this embodiment mainly consists of cells and injection needles. The cells are composed of the outer cell layer (cell membrane) corresponding to label 1 and the internal cell structures (animal pole body 2 and plant pole body 3) corresponding to labels 2 and 3, exhibiting a typical spherical biological sample morphology. The animal pole and plant pole are relatively fixed, while the whole cell can rotate freely, which is used to simulate the rotation phenomenon of the cell nucleus when touched by the injection needle in a real scene. The injection needle corresponding to label 4 is a micromanipulation tool that can move along the x, y, and z directions in the simulation environment.

[0064] In this embodiment, the hardware structure of the robotic system mainly consists of an inverted microscope, a microscope-operating robotic arm, a camera, a micro-gripper, and an injection needle. The inverted microscope is an IXplore Standard model, equipped with an XY motorized stage, and its maximum magnification, achieved through the eyepiece and objective lenses, reaches 400x. The robotic arm is a TransferMan model with three degrees of freedom and a motion resolution of 0.05µm. A W160 camera is mounted on the inverted microscope to capture image information at a rate of 60 frames per second. For software control, the system's software and control components are integrated into the Robot Operating System (ROS) Melodic, running on Ubuntu 18.04. The computer configuration is an Intel Core i9-10980Xe CPU and an NVIDIA TITAN XP.

[0065] In this embodiment, zebrafish embryonic cells were used as experimental subjects. These cells were placed in culture dishes with added nutrient solution to maintain their activity, thereby ensuring that the cells could maintain a normal physiological state during the experimental operation, which would facilitate precise cell microinjection experiments.

[0066] The entire cell microinjection process mainly consists of two steps: cell grasping and cell injection. Cell grasping is a prerequisite for cell injection, requiring the use of end effector tweezers to grasp the cell membrane; the entire process is achieved through visual servoing. The cell injection step requires adaptive adjustment of the cell nucleus orientation to ensure that the injection needle penetrates from the vegetative pole and reaches the animal pole, which places high demands on the intelligence of the needle position adjustment during injection.

[0067] During the cell puncture dataset collection phase, cell puncture image data from both real-world and simulated scenarios needed to be collected. Real-world data collection was achieved through a teleoperation system. While an expert remotely operated the cell injection task, a microscope recorded images of the entire process, including cells, the injection needle, and the end effector forceps. The entire process was recorded using the Robot Operating System (ROS) platform, ultimately collecting 500 images of the injection process. Simulated scenario data collection was simpler. The position of the end effector forceps was essentially fixed in the scenario. By randomizing the orientation of the animal and vegetative poles in the cell, as well as the position of the injection needle, through code, automated image collection was achieved, ultimately collecting 1000 images of the injection process.

[0068] The YOLO-segmentation module constructed a cell injection dataset using 500 real cell injection images and 1000 simulated cell injection images, and then performed supervised training. The loss function changes during the training process of the YOLO-segmentation module as follows: Figure 4 As shown. By Figure 4 As can be seen, the YOLO-segmentation module has three types of losses: bounding box loss. Category loss and distribution focus loss As the number of training epochs increases, the loss value gradually decreases and converges. Simultaneously, the evaluation metrics, precision and recall, also significantly increase with each training epoch, converging and stabilizing to reach a precision of 99.4% and a recall of 99.8%, demonstrating the ability to accurately detect the animal and plant pole regions of cells in both simulated and real-world images.

[0069] After training to obtain the YOLO-segmentation module, based on Figure 2 The network structure in the example is used for multi-task reinforcement learning training. The changes in task success rate and average reward value obtained by the policy during the training process are as follows: Figure 5 As shown in (a) and (b); by Figure 5 (a) It can be seen that as the strategy continues to learn, the success rate of the injection task during training and testing continues to improve, indicating that the current strategy can complete the cell injection task in the simulation environment and has a high task success rate of 95.0%.

[0070] To verify the effectiveness of the automated cell injection strategy trained in the simulation environment in a real-world scenario, experiments were conducted. While the robot operated autonomously according to the learned strategy, video frames were acquired in real time, and the YOLO-segmentation feature extraction and optimal motion generation process was repeated. The normalized motion was then transferred to the robot controller, transforming it into the actual motion of the robotic arm's end effector.

[0071] Figure 6 The demonstration showed that the robot controlled the injection needle to complete four automated injections of zebrafish embryonic cells. The robot successfully completed all four injections, leaving the cells intact with minimal cell membrane damage, and the cells hatched successfully. This proves the safety and effectiveness of the robotic cell microinjection method based on reinforcement learning and dynamic equilibrium game theory.

[0072] Finally, it should be noted that the above embodiments and descriptions are only used to illustrate the technical solutions of the present invention and not to limit it. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the disclosure of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the protection scope of the claims of the present invention.

Claims

1. A robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism, characterized in that, Includes the following steps: Step S1: Construct and train an instance segmentation network model; the input of the instance segmentation network model is the injection environment image, and the output is the corresponding environment mask image; Step S2: Construct and train an injection operation decision model; the injection operation decision model includes a feature extraction network, a PPO reinforcement learning network, and a state prediction network; in each round of training of the injection operation decision model, the feature extraction network receives a cell mask image and performs feature extraction, the PPO reinforcement learning network processes the feature extraction results and the real-time coordinates of the injection needle tip to obtain the expected position of the injection needle tip, and the state prediction network processes the feature extraction results to obtain predicted cell state parameters; The total loss function includes the loss and entropy regularization term of the PPO reinforcement learning network and the state prediction loss of the state prediction network. Step S3: During the robotic cell microinjection operation, real-time images of the injection environment are acquired and input into the instance segmentation network model to obtain an environment mask image. Based on the environment mask image, the real-time coordinates of the injection needle tip and the cell mask image are obtained. The feature extraction network of the injection operation decision model receives the cell mask image and performs feature extraction. The PPO reinforcement learning network receives the feature extraction results and the real-time coordinates of the injection needle tip to obtain the desired position of the injection needle tip. The robot controller uses the desired position of the injection needle tip as feedback information and controls the robot to move the injection needle tip.

2. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 1, characterized in that: The environmental mask image contains annotation information for the cell membrane, the animal pole, the vegetal pole, and the injection needle and tweezers. In step S3, the real-time coordinates of the injection needle tip are obtained based on the annotation information of the injection needle in the environmental mask image. The cell mask image contains annotation information for the cell membrane, the animal pole, and the vegetal pole.

3. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 1, characterized in that: Step S2 further includes an entropy conservation constraint; specifically, the entropy conservation constraint states that the rate of change of policy entropy is proportional to the rate of change of total task loss, expressed as: In the formula, Represents policy entropy The rate of change of the target over time. This represents the rate of change of total task loss. This indicates the loss of the PPO reinforcement learning network. This represents the state prediction loss. These are the weights of the state prediction loss. This represents the constraint coefficient.

4. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 3, characterized in that: In each round of training of the injection operation decision model, the target rate of change of policy entropy over time is calculated based on the loss of the PPO reinforcement learning network and the state prediction loss. Based on the actual rate of change of policy entropy over time and the target rate of change of policy entropy over time, the action exploration range of the next round of training is adjusted: if the actual rate of change of policy entropy over time is less than the target rate of change, the action exploration range of the next round of training is increased; if the actual rate of change of policy entropy over time is greater than the target rate of change, the action exploration range of the next round of training is decreased.

5. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 1, characterized in that: The entropy regularization term includes an entropy regularization coefficient; during the training of the injection operation decision model, the entropy regularization coefficient is adjusted according to the following formula: Where T represents the total number of training rounds, It is the total task loss in the t-th round of training. It is the initial total task loss for each round of training.

6. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 1, characterized in that: In step S1, the real cell puncture dataset and the simulated cell puncture dataset are combined into a sample dataset, and the sample dataset is used to train the instance segmentation network; in the sample dataset, each set of sample data mainly consists of an injection environment image and a corresponding environment mask image; in step S2, the simulated cell puncture dataset is used to train the injection operation decision model, and each set of simulated cell puncture data also includes real cell state parameters.

7. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 2, characterized in that: The simulated cell puncture dataset was obtained through simulation using a robotic cell microinjection simulation model. This model includes a cell model, an injection needle model, and a forceps model. The cell model comprises a nucleus and a cell membrane. The nucleus consists of an animal pole and a vegetative pole, which are rigidly connected and subjected to buoyancy within the cell membrane, resulting in vertical force balance. The injection needle model is configured to move freely along the X, Y, and Z axes. The needle tip diameter is set to 5-10 μm, the needle tube length to 400-500 μm, and the material parameters match the physical properties of quartz glass. The forceps model is configured to clamp and fix the cell model, and the cell nucleus is configured to undergo damped rotation when punctured by the injection needle model.

8. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 7, characterized in that: The cells to be punctured are zebrafish embryonic cells. In the robotic cell microinjection simulation model, the cell model is a zebrafish embryonic cell model. The rotational damping coefficient between the cell vegetative pole and the cell membrane is set to 0.05~0.1 N·s / m, the lateral friction coefficient is set to 20.0 N·s / m, the rotational damping coefficient is set to 0.05 N·s / m, and the contact stiffness and damping are set to 3000 N / m and 300 N·s / m, respectively.

9. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 1 or 6, characterized in that: In step S1, when training the instance segmentation network model, the weighted calculation result of the bounding box loss, class loss, and distribution focus loss is used as the total loss, and the loss function is expressed as: In the formula, The total loss of the instance segmentation network model. , , These are the bounding box loss, class loss, and distribution focus loss, respectively. , , These are the bounding box loss, class loss, and distribution focus loss relative to the final loss, respectively. The weights; For complete intersection and union, This is a parameter used to measure the consistency of the aspect ratio of the predicted bounding box. This is the balance coefficient; and These are the true category label and the predicted category label, respectively. It predicts the probability distribution of the bounding box and the ground truth bounding box across four coordinates. It is the focusing parameter.

10. The robotic cell microinjection method based on reinforcement learning with a dynamic equilibrium game mechanism according to claim 1, characterized in that: The cell state parameters include the cell's state angle, rotational angular velocity, and rotational angular acceleration; the state estimation loss is expressed as: In the formula, , , These are the predicted values ​​for the state angle, rotational angular velocity, and rotational angular acceleration, respectively. , , These are the true values ​​of the state angle, rotational angular velocity, and rotational angular acceleration, respectively, in the real cell state parameters. This represents the mean square error between the predicted and true values.

Images