Operation method of cell operation platform combining piezoelectric actuator and six-axis mechanical arm

Abstract

The invention discloses an operation method of a cell operation platform combining a piezoelectric actuator and a six-axis mechanical arm, which integrates the large-stroke positioning capability of the mechanical arm and the nano-scale fine operation capability of the piezoelectric actuator, and combines the large-stroke positioning capability of the mechanical arm and the nano-scale fine operation capability of the piezoelectric actuator on the basis of macroscopic mechanical arm movement. The piezoelectric driving unit can accurately approach and control target cells in various postures, so that cell-level fine operation tasks such as cell surface treatment, single cell grabbing and cell injection are completed at high precision, and reliable technical support is provided for single cell analysis and operation.

Description

Technical Field

[0001] The present invention relates to the technical field of cell manipulation, and in particular to an operation method of a cell manipulation platform combining a piezoelectric actuator with a six-axis mechanical arm. Background Art

[0002] With the continuous development of micromanipulation technology, micromanipulation platforms based on the principle of macro-micro dual drive have been widely used in the fields of biomedicine, new materials, etc., such as cell surface treatment, single cell grabbing, cell injection and other cell-level fine operation tasks. This type of macro-micro operation platform usually consists of two parts: a six-degree-of-freedom parallel workbench responsible for macroscopic movement and a micromanipulator unit driven by nano-scale piezoelectric ceramics. Among them, the six-degree-of-freedom parallel workbench is used to complete the large-scale positioning of cell positions at the macro scale; while the piezoelectric actuator realizes high-precision micro-displacement adjustment of cell operations. To achieve precise control at the single-cell level, the macro-micro operation platform needs to have nano-level displacement resolution and efficient driving capabilities. Traditional macro-micro operation robots are often used for single-cell operations due to their high resolution and fast response speed. However, the displacement mechanism itself at the macro scale has problems such as small displacement range and limited operation dimension, which makes it impossible to approach cells in multiple postures for different operations. In addition, the macro-micro operation platform also puts forward higher requirements for machine vision guidance, motion control, automation and other aspects. How to effectively integrate heterogeneous systems, achieve multi-level precision control, and improve the level of intelligent operation has become a key technical problem that needs to be solved urgently. Summary of the invention

[0003] In view of the problems existing in the prior art, the purpose of the present invention is to provide an operating method for a cell manipulation platform combining a piezoelectric actuator with a six-axis robotic arm, which integrates the large-stroke positioning capability of the robotic arm and the nanometer-level fine manipulation capability of the piezoelectric drive. On the basis of the macroscopic movement of the robotic arm, the piezoelectric drive unit can accurately approach and manipulate the target cells in a variety of postures, thereby achieving high-precision completion of cell-level fine manipulation tasks such as cell surface treatment, single cell grasping, and cell injection, providing reliable technical support for single-cell analysis and manipulation.

[0004] To solve the above problems, the present invention adopts the following technical solutions.

[0005] A method for operating a cell manipulation platform combining a piezoelectric actuator with a six-axis robotic arm comprises the following steps:

[0006] 1) Use Zhang Zhengyou's camera calibration method to calibrate the binocular camera and the microscope camera to obtain the internal parameters of the two cameras;

[0007] 2) Install the binocular camera at the connecting rod between the fifth and sixth axes of the robot arm, complete the hand-eye calibration of the robot arm and the binocular camera, and determine their spatial coordinate relationship;

[0008] 3) Obtain 3D reconstruction of the operating table through a binocular camera, and use visual servo control to achieve large-scale coarse positioning of the robotic arm;

[0009] 4) Under an inverted microscope, the segmentation detection algorithm YOLO-SF is used to obtain the precise location and contour information of the target cells;

[0010] 5) Calibrate the microscope image coordinate system and the robot base coordinate system to establish a spatial coordinate conversion relationship between the two;

[0011] 6) Based on the cell position information, multi-posture operation of the six-axis robot for cell micromanipulation.

[0012] In step 3), a three-dimensional reconstruction of the operating table is obtained by a binocular camera, and a large-scale rough positioning of the robot arm is achieved by visual servo control, including the following steps:

[0013] A1. Image Capture

[0014] The binocular camera is installed at the end of the robotic arm to capture images on the operating table. The images of the operating table captured by the left and right cameras are I l and I r ;

[0015] For a spatial point P, its image I imaged by the left and right cameras l and I r The corresponding points in are P L and P R , the horizontal coordinate on the imaging plane is X L and X R , assuming that the distance between the object and the binocular camera is Z, the focal length of the camera is f, and the optical centers of the two cameras are O L and O R The distance between them is b, where b is the baseline distance. According to the principle of similar triangles, the following relationship can be obtained:

[0016]

[0017] By simplifying, we can get the triangle distance measurement formula:

[0018]

[0019] Where parallax d = X L -X R , thus, the distance Z from the spatial point P to the binocular camera is calculated by the parallax d;

[0020] A2. Image preprocessing and 3D reconstruction

[0021] Through the image preprocessing algorithm, the noise is filtered out and the edges are enhanced. The preprocessed left and right images I l ′ and I r ′, and the edge image E l and E r The disparity map D(x, y) is generated by the stereo matching algorithm, and the three-dimensional model M(V, F, T) of the operating table is reconstructed through the disparity map. V is a vertex set, which represents points in the three-dimensional space; F is a face set, which represents the connection relationship between vertices; T is a texture set, which represents the texture information of the surface of the three-dimensional model;

[0022] In terms of multi-scale feature extraction, for the left and right images I l ′ and I r ′ Perform Gaussian pyramid decomposition respectively to obtain feature maps of different scales Where s represents the scale;

[0023] Based on the adaptive weighted matching, stereo matching is performed at each scale s to generate a disparity map {D s (x,y)}. For disparity maps of different scales, according to the local matching confidence C s (x,y), the lower the weight, the greater the matching uncertainty in the area, and the weight w is assigned s (x,y), and fuse the disparity information of each scale:

[0024]

[0025] Among them, the weight w s (x,y) is defined as the negative exponential function of the matching cost:

[0026]

[0027] λ is a parameter that controls the sensitivity of matching weight;

[0028] The optimized disparity map D(x,y) is used to calculate the depth value of the three-dimensional space point:

[0029]

[0030] In scenes with complex lighting or uneven texture, the introduction of multi-scale fusion and adaptive weights can effectively alleviate the limitations of traditional stereo matching algorithms and significantly improve the robustness and accuracy of 3D reconstruction.

[0031] A3. Feature extraction and matching

[0032] In the feature extraction and matching process, key feature points near the microscope are first extracted from the 3D reconstruction model. Common feature points include the edge of the microscope, marking points, and areas with strong texture contrast. These feature points are extracted through edge detection, corner detection, or SIFT and SURF algorithms. Then a descriptor is generated for each feature point. The descriptor can represent the local information of the feature point and has certain scale and rotation invariance. Common descriptors include SIFT, SURF, and ORB.

[0033] Next, the feature points extracted from the 3D model are matched with the feature points in the pre-stored microscope feature library through the feature matching algorithm. In order to improve the matching efficiency and accuracy, the feature library adopts the efficient index structure KD-Tree, and uses metrics such as Euclidean distance or Hamming distance to find the most similar feature point pairs. The mismatched points in the matching process will be eliminated through RANSAC to ensure that the final matching results have high accuracy and robustness. Through the RANSAC algorithm, the system can find the optimal transformation model among a large number of matching points, thereby filtering out mismatches and retaining the correct matching point pairs;

[0034] After completing feature matching, the desired position T0 of the microscope in three-dimensional space can be calculated based on the position of the feature points obtained by matching. This position is the target point that the end effector of the robot arm needs to reach. Using the inverse kinematics method, the system can calculate the angle θ0 of each joint of the robot arm based on the desired position T0, thereby generating corresponding control instructions to drive the robot arm to move accurately to the target position and perform subsequent operations.

[0035] A4. Generate control instructions and move operations

[0036] According to the position of the desired point in three-dimensional space, the target position T0 and posture of the end effector of the robot arm are calculated. The control instructions are generated through the forward kinematics calculation of the robot arm to drive the robot arm to move to the desired point;

[0037] The robot moves according to the generated control instructions and accurately reaches the desired point near the inverted microscope. After reaching the desired point, the robot can perform subsequent cell micromanipulation.

[0038] In step 4), the YOLO-SF network structure is divided into four parts: input, backbone network, neck and YOLO head;

[0039] B1. Input

[0040] The YOLO-SF model receives RGB images as input. These images are in common JPEG or PNG formats. Before the images are fed into YOLO-SF, they need to be preprocessed to improve the performance of the model and the accuracy of the detection results. After the preprocessing step is completed, the processed images will be used in the YOLO-SF model to perform cell detection tasks.

[0041] B2. Backbone network

[0042] The backbone network of YOLO-SF consists of convolutional layers, spatial pyramid pooling, and fully connected layers. In order to more efficiently detect cells of different sizes and shapes and expand the receptive field of the model, MobileViTv2 is used as the backbone network. MobileViTv2 is a lightweight visual Transformer network, and its structure is divided into two parts: a backbone network based on MobileNetV3 and a head network based on Vision Transformer.

[0043] The backbone network extracts image features through lightweight depth-separable convolution modules and inverse residual modules, aiming to achieve a design with high computational efficiency and low memory usage. The main network adopts the framework of Vision Transformer, and encodes and decodes features through the Transformer encoder to capture global features and establish associations between features. To further reduce the amount of computation, MobileViTv2 introduces a multi-scale mechanism that can prioritize features of different resolutions, improve model efficiency while maintaining accuracy;

[0044] The size of the input feature map is W × H × C. First, the features are processed by the local representation module, using n × n convolution to encode local information, and then mapped to a high-dimensional space through 1 × 1 point-by-point convolution. Subsequently, the features are passed to the global representation module, where the global information is encoded through the Transformer, and then the mapped feature information is processed again using 1 × 1 point-by-point convolution.

[0045] Finally, the features enter the fusion module, and the number of channels is increased from C to 2C through 1×1 point-by-point convolution, and then normalized through another 1×1 convolution to complete the feature optimization process;

[0046] B3. Neck network

[0047] The neck network of YOLO-SF combines the residual structure and the squeeze and excitation attention mechanism to enhance the feature extraction capability. In the neck network, the ELAN-C5 module improves the feature fusion efficiency by stacking convolution blocks and ensures the shortest gradient path. However, this module is insufficient in integrating the feature information of the upper and lower layers, and is prone to ignoring small targets, which limits the ability to capture global information. To solve this problem, the CBAM attention mechanism is introduced to process channel and spatial feature information, and CBAM is integrated into the ELAN-C5 module to form an improved ELAN-C5-CBAM architecture.

[0048] The CBAM attention module contains the channel attention module and the spatial attention module. When the feature map F∈R C×H×W After being generated by the convolutional network, CBAM sequentially calculates the one-dimensional channel attention map M c ∈R C×1×1 and the 2D spatial attention map M c ∈R 1×H×W , the calculation process is shown in the following formula:

[0049]

[0050] Among them, the operator represents element-by-element multiplication. First, the input feature map is processed using the channel attention mechanism to obtain F′. Then, the processed feature map F′F′F′ is subjected to spatial attention and finally outputs F″.

[0051] Each channel in the feature map is regarded as a feature detector. In order to efficiently extract channel features, a combination of maximum pooling and average pooling is used to compress the spatial dimensions of the feature map into two spatial descriptions: and Then, the channel attention map M is calculated through a shared multi-layer perceptron network c ∈R C×1×1 , the calculation process is shown in the following formula:

[0052] M c (F)=α(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0053]

[0054] in, and are different weight matrices. Unlike channel attention, spatial attention focuses on position information. To calculate spatial features, the channel dimension generates two feature descriptions through maximum pooling and average pooling operations, which are expressed as and Subsequently, the two feature descriptions are fused using a dense connection layer and a convolution operation is performed to generate a spatial attention map M. s (F)∈R H×W , the calculation process is shown in the following formula:

[0055] M s (F) = α(f 7×7 ([AvgPool(F); MaxPool(F)]))

[0056]

[0057] B4.YOLO Head

[0058] In cell detection, occlusion and interference of various objects are very common. The original YOLOv7 algorithm only provides a bounding box surrounding the entire object area, which cannot accurately describe the object boundary at the pixel level. Especially in the case of occlusion or overlap, the positioning accuracy of the cell area is low. To solve these problems, this study improved the detection head of YOLOv7 and replaced it with the segmentation detection head of YOLOR to achieve the fusion of object detection and pixel-level segmentation;

[0059] YOLOR's segmentation detection head is a key component that combines object detection and segmentation tasks. It defines the "IDetect" class for object detection and extends the functionality through the "ISegate" class to support segmentation tasks. The "ISegate" class inherits and extends the "IDetect" class to achieve shared functionality for different visual tasks and avoid duplication of design.

[0060] The "IDetect" class uses image features extracted by the backbone and neck networks to generate candidate bounding boxes, while predicting object categories and confidence levels. Subsequently, the "ISegate" class generates a segmentation mask for each candidate box based on shared features, dividing pixels into foreground or background. The segmentation mask can make pixel-level predictions of cells and impurities, and accurately segment cell targets from the image. The target detection results are combined with the segmentation mask to generate the location information and pixel-level segmentation results for each cell. The improved detection head design supports the simultaneous processing of cell target positioning and pixel-level segmentation, achieving a more comprehensive and accurate analysis of cell location;

[0061] B5. Loss of zoom

[0062] The variable focal loss introduces a dynamic focus factor that can be dynamically adjusted according to the complexity, prediction probability, and confidence of the cell sample, thereby achieving a more accurate and balanced focus on the cell category. At the same time, the boundary-aware loss function significantly improves the positioning accuracy of the bounding box by prioritizing the optimization of the boundary area of ​​the object. By adding additional loss terms in the boundary area, this method further optimizes the regression effect of the bounding box.

[0063] The mathematical expression of focal loss is:

[0064]

[0065] Mathematical expression of zoom loss:

[0066] VFL(p t )=-α(1-p t ) γ log(p t_hat )

[0067] = -α(1-p t_hat ) β log(p t_hat )

[0068] Among them, p t Indicates the probability that the model predicts that the sample is a positive class, p t_hat It indicates the probability that the model predicts that the sample is positive and the prediction is correct. The parameter α is a balance factor, which is used to adjust the weights of positive and negative samples. When the predicted probability is close to 1, the focusing factor gradually decreases; when the predicted probability is close to 0, the focusing factor increases. The parameters γ and β are used to adjust the weights of samples that are easy to classify and accurate. The combination of these parameters improves the focusing accuracy and promotes the balance between positive and negative samples, thereby optimizing the performance of the model in detecting and segmenting cell targets.

[0069] In step 5), the microscope image coordinate system and the robot base coordinate system are calibrated to establish a spatial coordinate conversion relationship between the two;

[0070] The robot arm and microscope camera are configured with the eye outside the hand. The purpose of calibration is to obtain the transformation matrix To lay the foundation for subsequent operations, the calibration process is also called the process of solving the equation AX=XB. The calculation process for the eye-outside-hand configuration is as follows:

[0071] The robot arm and microscope camera adopt the eye-outside-hand configuration. The purpose of calibration is to determine the transformation matrix from the camera coordinate system to the robot arm base coordinate system. Provide basic support for subsequent operations. The calibration process is actually to solve A i X=XB i The process of the equation specifically involves the following steps:

[0072] C1. Motion data collection

[0073] Through multiple movements of the robot arm, the motion matrix in the robot arm base coordinate system is recorded and the corresponding motion matrix in the camera coordinate system Where i (i = 1, 2, 3 ... n) is the transformation matrix under the i-th data collection;

[0074] C2. Constructing the system of equations

[0075] According to A i X=XB i The constraint relationship is used to construct a set of equations using all the collected motion data, where X is the transformation matrix to be solved. The set of equations is as follows:

[0076]

[0077] C3. Solve the matrix X

[0078] Use optimization algorithms or matrix decomposition methods to solve the optimal transformation matrix X that satisfies the equation, that is, the transformation matrix from camera to base

[0079] Through this calibration process, the geometric relationship between the robotic arm and the microscope camera is accurately established, laying the foundation for other subsequent visual tasks.

[0080] In step 6), according to the cell position information, the six-axis robot arm multi-posture operation for cell micromanipulation includes the following steps:

[0081] D1. Set the current position as the initial position: According to the DH parameter table below and the parameters of the microscope camera calibrated in 1 above, determine the coordinate transformation relationship between the camera and the robot. Set the joint angle θ0 of the robot at the current position and the end effector position T0.

[0082] The order of parameter changes between adjacent joint coordinate systems is: α, a, θ, d,

[0083] The rotation matrix is ​​as follows:

[0084]

[0085]

[0086] D2. Collect cell images I through a microscope and use image processing algorithms to extract the location p of the target cells. c =(x c ,y c ,z c ),

[0087] Convert the cell position from the image coordinate system to the robot base coordinate system:

[0088]

[0089] in, is the transformation matrix from the camera to the robot base, is the transformation matrix from image to camera;

[0090] D3. According to the requirements of cell operation and the location of target cells c , at the target position p c On the basis of , the attitude angle (α, β, γ) is introduced to represent the rotation attitude of the end effector. α represents the angle of rotation around the x-axis, β represents the angle of rotation around the y-axis, and γ represents the angle of rotation around the z-axis. In this way, the target posture can be expressed as:

[0091] T goal =T(p c )·R x (α)·R y (β)·R z (γ)

[0092] Among them, T(p c ) is p c is the translation matrix of the origin, R x (α), R y (β), R z (γ) are rotation matrices rotating around three axes. By introducing the attitude angle, the robot arm has the ability to operate cells in multiple attitudes, which can not only expand the operating range and flexibility of the robot arm, but also improve the accuracy and success rate of the operation;

[0093] After determining the target position T of the end effector goal Finally, considering the special constraints of micromanipulation, the plan is made from the current pose T0 to the target pose T goal The movement path,

[0094] The path can be discretized into a series of intermediate poses T1, T2, ..., T n , each pose corresponds to a time point t i , the pose change can be expressed by exponential mapping:

[0095]

[0096] D4. For each intermediate pose T i , solve the corresponding joint angle θ i , use numerical iteration to solve the inverse kinematics equations:

[0097] f(θ i )=T i -T1(θ i1 )·T2(θ i2 )·…·T6(θ i6 )=0

[0098] Assuming there are multiple solutions, choose the one that matches the current joint angle θ i-1 The closest solution ensures the continuity of motion.

[0099] D5. For the discrete joint angle sequence θ1,θ2,…,θ n Smoothing and optimization are performed, and spline interpolation is used to generate a continuous joint angle trajectory θ(t), and the continuity constraints of velocity and acceleration are satisfied. The dynamic characteristics of the robot arm are considered during optimization to minimize the joint torque or energy consumption:

[0100]

[0101] D6. Calculate the joint velocity based on the optimized joint angle trajectory θ(t) and acceleration Design a trajectory tracking controller to achieve precise motion control, and the torque control is as follows:

[0102]

[0103] At the same time, the cell position is detected in real time through visual feedback to compensate for motion errors. If the error exceeds the threshold, path planning and trajectory optimization are re-performed;

[0104] D7. When the end effector reaches the target position T goal When the microneedle is inserted into the cell, the piezoelectric ceramic driver is activated to precisely control the micro-movement of the microneedle near the cell, achieving sub-micron-level precise alignment and operation. During the operation, the contact force with the cell is monitored by the force sensor to avoid damaging the cell. If the contact force exceeds the safety threshold, the end position is adjusted in time. After the operation is completed, the robot arm is controlled to return to the initial position T0 to prepare for the next operation.

[0105] D8. During the entire operation, monitor the system status and performance indicators, such as joint angle, speed, torque, etc. If an error or abnormality is detected, stop the movement in time and execute the recovery procedure. The recovery procedure may include: returning to a safe position, recalibrating sensors, adjusting control parameters, etc. If necessary, send an audible alarm to the operator to request manual intervention and processing.

[0106] Beneficial effects of the present invention

[0107] Compared with the prior art, the advantages of the present invention are:

[0108] The operating dimension and degree of freedom are higher. By combining a six-degree-of-freedom serial robotic arm with a piezoelectric actuator, the robotic arm not only has the ability to move with high degrees of freedom in three-dimensional space, but can also use the piezoelectric actuator to achieve fine operations on a micro scale, thereby greatly increasing the dimension and flexibility of operation, and being able to efficiently approach and operate cells from multiple angles and postures.

[0109] The macro-displacement range is larger. Compared with the parallel mechanism, the serial robot arm has a larger working space range and can cover a larger operating range. It is suitable for batch and high-throughput cell manipulation tasks and significantly improves the operating efficiency.

[0110] The three-dimensional visual guidance capability is stronger. The dual-vision system that integrates a depth camera and an inverted microscope can not only obtain two-dimensional plane images of cells, but also reconstruct the three-dimensional geometric information of cells and their surrounding environment, providing reliable perception support for high-precision three-dimensional visual servo control.

[0111] It has a higher level of automation and intelligence, combining machine vision, robotics and artificial intelligence technology to achieve automated operation control under vision guidance, and incorporates intelligent decision-making functions such as machine learning, which greatly improves the system's level of automation and intelligence and reduces the need for human intervention.

[0112] The comprehensive utilization of multiple feedback information integrates multiple perception information such as visual feedback, displacement feedback and force feedback. Through the comprehensive application of closed-loop control algorithms, fine perception and control of the entire process of cell operation are achieved, further improving the accuracy and reliability of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 This is a system structure diagram of the operating platform of the present invention.

[0114] Figure 2 This is a control block diagram of the piezoelectric drive system of the present invention.

[0115] Figure 3 The overall flow chart of the camera calibration of the present invention.

[0116] Figure 4 This is a schematic diagram of the binocular stereo vision principle of the present invention.

[0117] Figure 5 This is the YOLO-SF network structure diagram of the present invention.

[0118] Figure 6 This is the structural diagram of MobileViTv2 of the present invention.

[0119] Figure 7 This is a detailed diagram of the backbone network improvement of the present invention.

[0120] Figure 8 This is a detailed diagram of the ELAN-C5-CBAM neck network improvement of the present invention.

[0121] Fig. 9 This is a structural diagram of the CBAM of the present invention.

[0122] Fig.10This is a structural diagram of the channel attention and spatial attention of the present invention.

[0123] Fig.11 This is the YOLO-SF head image of the present invention.

[0124] Fig.12 This is a model diagram of the eye outside the hand according to the present invention.

[0125] Fig.13 It is a structural diagram of the mechanical arm of the present invention.

[0126] Fig.14 It is a schematic diagram of the operating platform of the present invention. DETAILED DESCRIPTION

[0127] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments, and all other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work are within the scope of protection of the present invention.

[0128] like Figure 1 and Figure 2 As shown, the operation steps of the cell manipulation platform combining a piezoelectric actuator under visual perception with a six-axis robotic arm are as follows:

[0129] 1. Use Zhang Zhengyou's camera calibration method to calibrate the binocular camera and the microscope camera to obtain the internal parameters of the two cameras. The specific process is as follows: Figure 3 shown.

[0130] 2. Install the binocular camera on the connecting rod between the fifth and sixth axes of the robotic arm, complete the hand-eye calibration of the robotic arm and the binocular camera, and determine their spatial coordinate relationship.

[0131] 3. Obtain the three-dimensional reconstruction of the operating table through the binocular camera, and use visual servo control to achieve large-scale coarse positioning of the robotic arm, including the following steps:

[0132] A1. Image Capture

[0133] The binocular camera is installed at the end of the robotic arm to capture images on the operating table. The images of the operating table captured by the left and right cameras are I l and I r .

[0134] For a spatial point P, its image I imaged by the left and right cameras l and I r The corresponding points in are P L and P R , the horizontal coordinate on the imaging plane is X L and XR Assume that the distance between the object and the binocular camera is Z, the focal length of the camera is f, and the optical centers of the two cameras are O L and O R The distance between them is b (baseline distance). According to the principle of similar triangles, the following relationship can be obtained:

[0135]

[0136] By simplifying, we can get the triangle distance measurement formula:

[0137]

[0138] Where parallax d = X L -X R Therefore, the distance Z from the spatial point P to the binocular camera is calculated by the disparity d.

[0139] A2. Image preprocessing and 3D reconstruction

[0140] Through the image preprocessing algorithm, the noise is filtered out and the edges are enhanced. The preprocessed left and right images I l ′ and I r ′, and the edge image E l and E r The disparity map D(x, y) is generated by the stereo matching algorithm, and the three-dimensional model M(V, F, T) of the operating table is reconstructed through the disparity map. V is a vertex set, which represents points in three-dimensional space; F is a face set, which represents the connection relationship between vertices; T is a texture set, which represents the texture information on the surface of the three-dimensional model.

[0141] In the process of 3D reconstruction, the traditional stereo matching algorithm is easily affected by illumination changes or uneven texture, resulting in noise or mismatch in the disparity map D(x,y). To this end, the present invention introduces a multi-scale stereo matching method with adaptive weights, which improves the matching accuracy by constructing a multi-scale feature map and assigning dynamic weights to the matching results.

[0142] In terms of multi-scale feature extraction, for the left and right images I l ′ and I r ′ Perform Gaussian pyramid decomposition respectively to obtain feature maps of different scales Where s represents the scale.

[0143] Based on the adaptive weighted matching, stereo matching is performed at each scale s to generate a disparity map {D s (x,y)}. For disparity maps of different scales, according to the local matching confidence C s (x,y), the lower the weight, the greater the uncertainty of the matching in this area. Assign weight w s (x,y), and fuse the disparity information of each scale:

[0144]

[0145] Among them, the weight w s (x,y) is defined as the negative exponential function of the matching cost:

[0146]

[0147] λ is a parameter that controls the sensitivity of the matching weight.

[0148] The optimized disparity map D(x,y) is used to calculate the depth value of the three-dimensional space point:

[0149]

[0150] In scenes with complex lighting or uneven texture, the introduction of multi-scale fusion and adaptive weights can effectively alleviate the limitations of traditional stereo matching algorithms and significantly improve the robustness and accuracy of 3D reconstruction.

[0151] In scenes with complex lighting, local areas of the image may have significant brightness differences due to lighting changes, which can seriously affect the accuracy of stereo matching. By introducing multi-scale feature extraction, the system can capture overall structural features on lower-resolution feature maps while retaining more detail information in high-resolution feature maps, thereby effectively reducing the interference caused by lighting changes. In addition, combined with the adaptive weight mechanism, the matching confidence of local areas can be dynamically evaluated, and the contribution weight of areas with greater lighting influence can be automatically reduced, thereby improving the robustness of global matching.

[0152] In areas with uneven or weak textures, traditional stereo matching algorithms often produce mismatches due to the lack of obvious features, thereby increasing the noise in the disparity map. After the introduction of multi-scale fusion, low-scale global features can effectively compensate for the shortcomings of areas with weak textures, while high-resolution features can retain key texture information. Through adaptive weight adjustment, the fusion of features at different scales can effectively suppress the noise in areas with uneven textures, thereby enhancing the matching effect and accuracy.

[0153] For surfaces with blurred edges or strong reflections that may exist in actual scenes, multi-scale fusion can enhance the ability to capture complex scenes by extracting features at different scales. In highly reflective areas such as around microscopes, low-scale feature maps help avoid reflection errors, while high-resolution feature maps can accurately capture the clear edges of microscope markers, thereby effectively improving matching accuracy, especially in difficult visual scenes, which can significantly improve the robustness of 3D reconstruction.

[0154] A3. Feature extraction and matching

[0155] In the feature extraction and matching process, key feature points near the microscope are first extracted from the 3D reconstruction model. Common feature points include the edge of the microscope, marker points, and areas with strong texture contrast. These feature points are extracted through edge detection, corner detection, or algorithms such as SIFT and SURF, and then a descriptor is generated for each feature point. The descriptor can represent the local information of the feature point and has certain scale and rotation invariance. Common descriptors include SIFT, SURF, ORB, etc.

[0156] Next, the feature points extracted from the 3D model are matched with the feature points in the pre-stored microscope feature library through the feature matching algorithm. In order to improve the matching efficiency and accuracy, the feature library adopts the efficient index structure KD-Tree, and uses metrics such as Euclidean distance or Hamming distance to find the most similar feature point pairs. The mismatched points in the matching process will be eliminated through RANSAC (random sampling consensus algorithm) to ensure that the final matching results have high accuracy and robustness. Through the RANSAC algorithm, the system can find the optimal transformation model among a large number of matching points, thereby filtering out mismatches and retaining the correct matching point pairs.

[0157] After completing feature matching, the desired position T0 of the microscope in three-dimensional space can be calculated based on the position of the feature points obtained by matching. This position is the target point that the end effector of the robot arm needs to reach. Using the inverse kinematics method, the system can calculate the angle θ0 of each joint of the robot arm based on the desired position T0, thereby generating corresponding control instructions to drive the robot arm to move accurately to the target position and perform subsequent operations.

[0158] A4. Generate control instructions and move operations

[0159] According to the position of the desired point in three-dimensional space, the target position T0 and posture of the end effector of the robot arm are calculated. The control instructions are generated through the forward kinematics calculation of the robot arm to drive the robot arm to move to the desired point.

[0160] The robot moves according to the generated control instructions and accurately reaches the desired point near the inverted microscope. After reaching the desired point, the robot can perform subsequent cell micromanipulation.

[0161] 4. Under an inverted microscope, the segmentation detection algorithm YOLO-SF is used to obtain the precise position and contour information of the target cells.

[0162] The present invention proposes a segmentation detection algorithm YOLO-SF, which combines instance segmentation technology with the YOLOv7 target detection algorithm to effectively improve the detection accuracy. We constructed a cell segmentation dataset (CSD) containing a variety of cell images and adopted the segmentation detection head of YOLOR to enhance the segmentation accuracy and detail expression ability of the model. By introducing the MobileViTv2 module to build the backbone network, the model parameters are significantly reduced while maintaining the feature extraction efficiency. In addition, the convolutional block attention module (CBAM) is integrated in the effective layer aggregation network (ELAN) of the neck network, which not only expands the receptive field of the model, but also enhances the processing ability of the channel and spatial information of the cell image. The FPS of the model reached 52.24, which meets the needs of real-time detection. The improved algorithm shows excellent generalization ability and robustness.

[0163] The YOLO-SF network structure is divided into four parts: input, backbone network, neck and YOLO head. The network structure of YOLO-SF is as follows Figure 5 shown.

[0164] B1. Input

[0165] The YOLO-SF model receives images in RGB format as input, which are usually in the common JPEG or PNG format. Before the image is sent to YOLO-SF, it needs to be preprocessed to improve the performance of the model and the accuracy of the detection results. After the preprocessing step is completed, the processed image will be used in the YOLO-SF model to perform the cell detection task.

[0166] B2. Backbone network

[0167] The backbone network of YOLO-SF consists of convolutional layers, spatial pyramid pooling (SPP), and fully connected layers. In order to more efficiently detect cells of different sizes and shapes and expand the receptive field of the model, MobileViTv2 is used as the backbone network. MobileViTv2 is a lightweight visual Transformer network whose structure is divided into two parts: a backbone network based on MobileNetV3 and a head network based on Vision Transformer (ViT).

[0168] The backbone network extracts image features through lightweight depth-separable convolutional modules and inverse residual modules, aiming to achieve a design with high computational efficiency and low memory usage. The main network adopts the framework of Vision Transformer (ViT), and encodes and decodes features through the Transformer encoder to capture global features and establish associations between features. To further reduce the amount of computation, MobileViTv2 introduces a multi-scale mechanism that can prioritize features of different resolutions, improving model efficiency while maintaining accuracy.

[0169] The structure of MobileViTv2 is as follows Figure 6 shown.

[0170] The size of the input feature map is W×H×C. First, the features are processed by the local representation module, using n×n convolution to encode local information, and then mapped to a high-dimensional space through 1×1 point-by-point convolution. Subsequently, the features are passed to the global representation module, where the global information is encoded through the Transformer. Next, the mapped feature information is processed again using 1×1 point-by-point convolution.

[0171] Finally, the features enter the fusion module, undergo a 1×1 point-by-point convolution to increase the number of channels from C to 2C, and are normalized by another 1×1 convolution to complete the feature optimization process. Figure 7 The detailed process of backbone network integration with MobileViTv2 is demonstrated.

[0172] B3. Neck network

[0173] The neck network of YOLO-SF combines residual structure and squeeze and excitation (SE) attention mechanism to enhance feature extraction capability. In the neck network, the ELAN-C5 module improves feature fusion efficiency by stacking convolution blocks and ensures the shortest gradient path. However, this module is insufficient in integrating feature information from upper and lower layers, and is prone to ignoring small targets, limiting the ability to capture global information. To address this problem, the CBAM attention mechanism is introduced to process channel and spatial feature information. CBAM is integrated into the ELAN-C5 module to form an improved ELAN-C5-CBAM architecture.

[0174] The CBAM attention module is incorporated into the C5 module of the neck network as follows Figure 8 shown.

[0175] The CBAM attention module consists of a channel attention module (CAM) and a spatial attention module (SAM). C×H×W After being generated by the convolutional network, CBAM sequentially calculates the one-dimensional channel attention map M c ∈R C×1×1 and the 2D spatial attention map Mc ∈R 1 ×H×W The calculation process is shown in the following formula:

[0176]

[0177] Among them, the operator Denotes element-wise multiplication. First, the input feature map is processed using the channel attention mechanism to obtain F′. Then, the processed feature map F′F′F′ is subjected to spatial attention and finally outputs F″. Fig. 9 Demonstrates the operation of the CBAM attention mechanism.

[0178] Each channel in the feature map is regarded as a feature detector. In order to efficiently extract channel features, a combination of maximum pooling and average pooling is used to compress the spatial dimensions of the feature map into two spatial descriptions: and Then, the channel attention map M is calculated through a shared multi-layer perceptron (MLP) network. c ∈R C×1×1 The calculation process is shown in the following formula.

[0179] M c (F)=α(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0180]

[0181] in, and are different weight matrices. Unlike channel attention, spatial attention focuses on location information. To calculate spatial features, the channel dimension generates two feature descriptions through maximum pooling and average pooling operations, which are represented as and Subsequently, the two feature descriptions are fused using a dense connection layer and a convolution operation is performed to generate a spatial attention map M. s (F)∈R H×W The calculation process is shown in the following formula.

[0182] M s (F) = α(f 7×7 ([AvgPool(F); MaxPool(F)]))

[0183]

[0184] The working principle of channel attention and spatial attention is as follows Fig.10 shown.

[0185] B4.YOLO Head

[0186] like Fig.11 As shown in the figure, occlusion and interference of various objects are very common in cell detection. The original YOLOv7 algorithm only provides a bounding box surrounding the entire object area, which cannot accurately describe the object boundary at the pixel level. Especially in the case of occlusion or overlap, the positioning accuracy of the cell area is low. To solve these problems, this study improved the detection head of YOLOv7 and replaced it with the segmentation detection head of YOLOR to achieve the fusion of object detection and pixel-level segmentation.

[0187] YOLOR's segmentation detection head is a key component that combines object detection and segmentation tasks. It defines the "IDetect" class for object detection and extends the functionality through the "ISegate" class to support segmentation tasks. The "ISegate" class inherits and extends the "IDetect" class to achieve shared functionality for different visual tasks and avoid duplication of design.

[0188] The "IDetect" class uses image features extracted by the backbone and neck networks to generate candidate bounding boxes, while predicting object categories and confidence. Subsequently, the "ISegate" class generates a segmentation mask for each candidate box based on shared features, dividing pixels into foreground or background. The segmentation mask can make pixel-level predictions of cells and impurities, and accurately segment cell targets from the image. The target detection results are combined with the segmentation mask to generate the location information and pixel-level segmentation results for each cell. The improved detection head design supports the simultaneous processing of cell target positioning and pixel-level segmentation, achieving a more comprehensive and accurate analysis of cell location.

[0189] B5. Loss of zoom

[0190] The variable focal loss introduces a dynamic focus factor that can be dynamically adjusted according to the complexity, prediction probability, and confidence of the cell sample, thereby achieving a more accurate and balanced focus on the cell category. At the same time, the boundary-aware loss function significantly improves the positioning accuracy of the bounding box by prioritizing the optimization of the boundary area of ​​the object. By adding additional loss terms in the boundary area, this method further optimizes the regression effect of the bounding box.

[0191] The mathematical expression of focal loss is:

[0192] FL(p t )=-(1-p t ) γ log(p t ),

[0193] Mathematical expression of zoom loss:

[0194] VFL(pt )=-α(1-p t ) γ log(p t_hat )

[0195] = -α(1-p t_hat ) β log(p t_hat )

[0196] Among them, p t Indicates the probability that the model predicts that the sample is a positive class, p t_hat It indicates the probability that the model predicts that the sample is positive and the prediction is correct. Parameter α is a balancing factor, which is used to adjust the weights of positive and negative samples. When the predicted probability is close to 1, the focusing factor gradually decreases; when the predicted probability is close to 0, the focusing factor increases. Parameters γ and β are used to adjust the weights of samples that are easy to classify and accurate. The combination of these parameters improves the focusing accuracy and promotes the balance between positive and negative samples, thereby optimizing the performance of the model in detecting and segmenting cell targets.

[0197] 5. Calibrate the microscope image coordinate system and the robot base coordinate system to establish the spatial coordinate transformation relationship between the two.

[0198] The robot arm and microscope camera are configured with the eye outside the hand. The purpose of calibration is to obtain the transformation matrix This lays the foundation for subsequent operations. The calibration process is also called the process of solving the equation AX=XB. Fig.12 , the calculation process for the eye-outside-hand configuration is as follows:

[0199] The robot arm and microscope camera adopt the eye-outside-hand configuration. The purpose of calibration is to determine the transformation matrix from the camera coordinate system to the robot arm base coordinate system. Provide basic support for subsequent operations. The calibration process is actually to solve A i X=XB i The process of the equation specifically involves the following steps:

[0200] C1. Motion data collection

[0201] Through multiple movements of the robot arm, the motion matrix in the robot arm base coordinate system is recorded and the corresponding motion matrix in the camera coordinate system Here, i (i=1, 2, 3…n) is the transformation matrix under the i-th data collection.

[0202] C2. Constructing the system of equations

[0203] According to A i X=XB iThe constraint relationship is used to construct a set of equations using all the collected motion data, where X is the transformation matrix to be solved. The set of equations is as follows:

[0204]

[0205] C3. Solve the matrix X

[0206] Use optimization algorithms or matrix decomposition methods to solve the optimal transformation matrix X that satisfies the equation, that is, the transformation matrix from camera to base

[0207] Through this calibration process, the geometric relationship between the robotic arm and the microscope camera is accurately established, laying the foundation for other subsequent visual tasks.

[0208] 6.Reference Fig.13 ,According to the cell position information, the six-axis robot arm multi-posture operation for cell micromanipulation includes the following steps:

[0209] D1. Set the current position as the initial position: According to the DH parameter table below and the parameters of the microscope camera calibrated using the above 1, determine the coordinate transformation relationship between the camera and the robot. Set the joint angle θ0 of the robot at the current position and the end effector position T0.

[0210] MDH table of table robot

[0211]

[0212] The order of parameter changes between adjacent joint coordinate systems is: α, a, θ, d

[0213] The rotation matrices can be obtained from the DH table defined above, as follows:

[0214]

[0215] D2. Collect cell images I through the above 4 microscopes and use image processing algorithms to extract the position p of the target cells. c =(x c ,y c ,z c ).

[0216] Convert the cell position from the image coordinate system to the robot base coordinate system:

[0217]

[0218] in, is the transformation matrix from the camera to the robot base, is the image-to-camera transformation matrix.

[0219] D3. According to the requirements of cell manipulation (injection or extraction) and the location of target cells c , at the target position p c Based on this, the attitude angle (α, β, γ) is introduced to represent the rotation attitude of the end effector. α represents the angle of rotation around the x-axis, β represents the angle of rotation around the y-axis, and γ represents the angle of rotation around the z-axis. In this way, the target posture can be expressed as:

[0220] T goal =T(p c )·R x (α)·R y (β)·R z (γ)

[0221] Among them, T(p c ) is p c is the translation matrix of the origin, R x (α), R y (β), R z (γ) are the rotation matrices around the three axes. By introducing the attitude angle, we can enable the robot arm to have the ability to operate cells in multiple attitudes. This can not only expand the operating range and flexibility of the robot arm, but also improve the accuracy and success rate of the operation.

[0222] After determining the target position T of the end effector goal Finally, considering the special constraints of micromanipulation (avoiding collision and maintaining posture), the plan is made from the current posture T0 to the target posture T goal motion path.

[0223] The path can be discretized into a series of intermediate poses T1, T2, ..., T n , each pose corresponds to a time point t i The pose change can be represented by an exponential mapping:

[0224]

[0225] D4. For each intermediate pose T i , solve the corresponding joint angle θ i . The inverse kinematics equations are solved using a numerical iteration method:

[0226] f(θ i )=T i -T1(θ i1 )·T2(θ i2 )…T6(θ i6 )=0

[0227] If there are multiple solutions, choose the one that matches the current joint angle θ i-1 The closest solution ensures the continuity of motion.

[0228] D5. For the discrete joint angle sequence θ1,θ2,…,θ n Smoothing and optimization. Use spline interpolation (such as cubic spline) to generate continuous joint angle trajectory θ(t) and satisfy the continuity constraints of velocity and acceleration. Consider the dynamic characteristics of the robot arm during optimization to minimize joint torque or energy consumption:

[0229]

[0230] D6. Calculate the joint velocity based on the optimized joint angle trajectory θ(t) and acceleration Design a trajectory tracking controller to achieve precise motion control. The torque control is as follows:

[0231]

[0232] At the same time, the cell position is detected in real time through visual feedback to compensate for motion errors. If the error exceeds the threshold, path planning and trajectory optimization are re-performed.

[0233] D7. When the end effector reaches the target position T goal When the microneedle is inserted into the cell, the piezoelectric ceramic driver is activated to finely control the micro-movement of the microneedle near the cell, achieving precise alignment and operation at the submicron level. During the operation, the contact force with the cell is monitored by the force sensor to avoid damaging the cell. If the contact force exceeds the safety threshold, the end position is adjusted in time. After the operation is completed, the robot arm is controlled to return to the initial position T0 to prepare for the next operation.

[0234] D8. During the entire operation, monitor the system status and performance indicators, such as joint angle, speed, torque, etc. If an error or abnormality is detected (such as a singular position, joint overrun, excessive torque), stop the movement in time and execute the recovery procedure. The recovery procedure may include: returning to a safe position, recalibrating sensors, adjusting control parameters, etc. If necessary, send a sound alarm to the operator to request manual intervention and processing.

[0235] The operating platform used in the present invention is as follows Fig.14 As shown, it includes the following structures: 1. seismic isolation table 2. picoliter pump 3. robotic arm 4. depth camera 5. piezoelectric actuator 6. cell microneedle 7. micromanipulation stage 8. inverted microscope camera.

[0236] The above is only a preferred specific implementation of the present invention; however, the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved conception within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for operating a cell manipulation platform combining a piezoelectric actuator and a six-axis robotic arm, characterized in that: The following steps are involved: 1) Use Zhang Zhengyou's camera calibration method to calibrate the binocular camera and the microscope camera to obtain the internal parameters of the two cameras; 2) Install the binocular camera at the connecting rod between the fifth and sixth axes of the robot arm, complete the hand-eye calibration of the robot arm and the binocular camera, and determine their spatial coordinate relationship; 3) Obtain 3D reconstruction of the operating table through a binocular camera, and use visual servo control to achieve large-scale coarse positioning of the robotic arm; 4) Under an inverted microscope, the segmentation detection algorithm YOLO-SF is used to obtain the precise location and contour information of the target cells; 5) Calibrate the microscope image coordinate system and the robot base coordinate system to establish a spatial coordinate conversion relationship between the two; 6) Based on the cell position information, multi-posture operation of the six-axis robot for cell micromanipulation.

2. The method for operating a cell manipulation platform combining a piezoelectric actuator and a six-axis robotic arm according to claim 1, characterized in that: In step 3), a three-dimensional reconstruction of the operating table is obtained by a binocular camera, and a large-scale rough positioning of the robot arm is achieved by visual servo control, including the following steps: A1. Image Capture The binocular camera is installed at the end of the robotic arm to capture images on the operating table. The images of the operating table captured by the left and right cameras are I l and I r ; For a spatial point P, its image I imaged by the left and right cameras l and I r The corresponding points in are P L and P R , the horizontal coordinate on the imaging plane is X L and X R , assuming that the distance between the object and the binocular camera is Z, the focal length of the camera is f, and the optical centers of the two cameras are O L and O R The distance between them is b, where b is the baseline distance. According to the principle of similar triangles, the following relationship can be obtained: By simplifying, we can get the triangle distance measurement formula: Where disparity d = X L -X R , thus, the distance Z from the spatial point P to the binocular camera is calculated by the parallax d; A2. Image preprocessing and 3D reconstruction Through the image preprocessing algorithm, the noise is filtered out and the edges are enhanced. The preprocessed left and right images I l ′ and I r ′, and the edge image E l and E r The disparity map D(x, y) is generated by the stereo matching algorithm, and the three-dimensional model M(V, F, T) of the operating table is reconstructed through the disparity map. V is a vertex set, which represents points in the three-dimensional space; F is a face set, which represents the connection relationship between vertices; T is a texture set, which represents the texture information of the surface of the three-dimensional model; In terms of multi-scale feature extraction, for the left and right images I l ′ and I r ′ Perform Gaussian pyramid decomposition respectively to obtain feature maps of different scales Where s represents the scale; Based on the adaptive weighted matching, stereo matching is performed at each scale s to generate a disparity map {D s (x,y)}. For disparity maps of different scales, according to the local matching confidence C s (x,y), the lower the weight, the greater the matching uncertainty in the area, and the weight w is assigned s (x,y), and fuse the disparity information of each scale: Among them, the weight w s (x,y) is defined as a negative exponential function of the matching cost: λ is a parameter that controls the sensitivity of matching weight; The optimized disparity map D(x,y) is used to calculate the depth value of the three-dimensional space point: In scenes with complex lighting or uneven texture, the introduction of multi-scale fusion and adaptive weights can effectively alleviate the limitations of traditional stereo matching algorithms and significantly improve the robustness and accuracy of 3D reconstruction. A3. Feature extraction and matching In the feature extraction and matching process, key feature points near the microscope are first extracted from the 3D reconstruction model. Common feature points include the edge of the microscope, marking points, and areas with strong texture contrast. These feature points are extracted through edge detection, corner detection, or SIFT and SURF algorithms. Then a descriptor is generated for each feature point. The descriptor can represent the local information of the feature point and has certain scale and rotation invariance. Common descriptors include SIFT, SURF, and ORB. Next, the feature points extracted from the 3D model are matched with the feature points in the pre-stored microscope feature library through the feature matching algorithm. In order to improve the matching efficiency and accuracy, the feature library adopts the efficient index structure KD-Tree, and uses metrics such as Euclidean distance or Hamming distance to find the most similar feature point pairs. The mismatched points in the matching process will be eliminated through RANSAC to ensure that the final matching results have high accuracy and robustness. Through the RANSAC algorithm, the system can find the optimal transformation model among a large number of matching points, thereby filtering out mismatches and retaining the correct matching point pairs; After completing feature matching, the desired position T0 of the microscope in three-dimensional space can be calculated based on the position of the feature points obtained by matching. This position is the target point that the end effector of the robot arm needs to reach. Using the inverse kinematics method, the system can calculate the angle θ0 of each joint of the robot arm based on the desired position T0, thereby generating corresponding control instructions to drive the robot arm to move accurately to the target position and perform subsequent operations. A4. Generate control instructions and move operations According to the position of the desired point in three-dimensional space, the target position T0 and posture of the end effector of the robot arm are calculated. The control instructions are generated through the forward kinematics calculation of the robot arm to drive the robot arm to move to the desired point; The robot moves according to the generated control instructions and accurately reaches the desired point near the inverted microscope. After reaching the desired point, the robot can perform subsequent cell micromanipulation.

3. The method for operating a cell manipulation platform combining a piezoelectric actuator and a six-axis robotic arm according to claim 2, characterized in that: In step 4), the YOLO-SF network structure is divided into four parts: input, backbone network, neck and YOLO head; B1. Input The YOLO-SF model receives RGB images as input. These images are in common JPEG or PNG formats. Before the images are fed into YOLO-SF, they need to be preprocessed to improve the performance of the model and the accuracy of the detection results. After the preprocessing step is completed, the processed images will be used in the YOLO-SF model to perform cell detection tasks. B2. Backbone network The backbone network of YOLO-SF consists of convolutional layers, spatial pyramid pooling, and fully connected layers. In order to more efficiently detect cells of different sizes and shapes and expand the receptive field of the model, MobileViTv2 is used as the backbone network. MobileViTv2 is a lightweight visual Transformer network, and its structure is divided into two parts: a backbone network based on MobileNetV3 and a head network based on Vision Transformer. The backbone network extracts image features through lightweight depth-separable convolution modules and inverse residual modules, aiming to achieve a design with high computational efficiency and low memory usage. The main network adopts the framework of Vision Transformer, and encodes and decodes features through the Transformer encoder to capture global features and establish associations between features. To further reduce the amount of computation, MobileViTv2 introduces a multi-scale mechanism that can prioritize features of different resolutions, improve model efficiency while maintaining accuracy; The size of the input feature map is W × H × C. First, the features are processed by the local representation module, using n × n convolution to encode local information, and then mapped to a high-dimensional space through 1 × 1 point-by-point convolution. Subsequently, the features are passed to the global representation module, where the global information is encoded through the Transformer, and then the mapped feature information is processed again using 1 × 1 point-by-point convolution. Finally, the features enter the fusion module, and the number of channels is increased from C to 2C through 1×1 point-by-point convolution, and then normalized through another 1×1 convolution to complete the feature optimization process; B3. Neck network The neck network of YOLO-SF combines the residual structure and the squeeze and excitation attention mechanism to enhance the feature extraction capability. In the neck network, the ELAN-C5 module improves the feature fusion efficiency by stacking convolution blocks and ensures the shortest gradient path. However, this module is insufficient in integrating the feature information of the upper and lower layers, and is prone to ignoring small targets, which limits the ability to capture global information. To solve this problem, the CBAM attention mechanism is introduced to process channel and spatial feature information, and CBAM is integrated into the ELAN-C5 module to form an improved ELAN-C5-CBAM architecture. The CBAM attention module contains the channel attention module and the spatial attention module. When the feature map F∈R C×H×W After being generated by the convolutional network, CBAM sequentially calculates the one-dimensional channel attention map M c ∈R C×1×1 and the 2D spatial attention map M c ∈R 1×H×W , the calculation process is shown in the following formula: Among them, the operator Represents element-wise multiplication. First, the input feature map is processed using the channel attention mechanism to obtain F ′ ,Then, perform spatial attention processing on the processed feature map F′F′F′, and finally output F″; Each channel in the feature map is regarded as a feature detector. In order to efficiently extract channel features, a combination of maximum pooling and average pooling is used to compress the spatial dimensions of the feature map into two spatial descriptions: and Then, the channel attention map M is calculated through a shared multi-layer perceptron network. c ∈R C×1×1 , the calculation process is shown in the following formula: M c (F)=α(MLP(AvgPool(F))+MLP(MaxPool(F))) in, and are different weight matrices. Unlike channel attention, spatial attention focuses on position information. To calculate spatial features, the channel dimension generates two feature descriptions through maximum pooling and average pooling operations, which are expressed as and Subsequently, the two feature descriptions are fused using a dense connection layer and a convolution operation is performed to generate a spatial attention map M. s (F)∈R H×W , the calculation process is shown in the following formula: M s (F)=α(f 7×7 ([AvgPool(F);MaxPool(F)])) B4.YOLO Head In cell detection, occlusion and interference of various objects are very common. The original YOLOv7 algorithm only provides a bounding box surrounding the entire object area, which cannot accurately describe the object boundary at the pixel level. Especially in the case of occlusion or overlap, the positioning accuracy of the cell area is low. To solve these problems, this study improved the detection head of YOLOv7 and replaced it with the segmentation detection head of YOLOR to achieve the fusion of object detection and pixel-level segmentation; YOLOR's segmentation detection head is a key component that combines object detection and segmentation tasks. It defines the "IDetect" class for object detection and extends the functionality through the "ISegate" class to support segmentation tasks. The "ISegate" class inherits and extends the "IDetect" class to achieve shared functionality for different visual tasks and avoid duplication of design. The "IDetect" class uses image features extracted by the backbone and neck networks to generate candidate bounding boxes, while predicting object categories and confidence. Subsequently, the "ISegate" class generates a segmentation mask for each candidate box based on shared features, classifying pixels as foreground or background. The segmentation mask enables pixel-level prediction of cells and impurities, accurately segmenting cell targets from the image. The target detection results are combined with the segmentation mask to generate the location information and pixel-level segmentation results for each cell. The improved detection head design supports the simultaneous processing of cell target positioning and pixel-level segmentation, achieving a more comprehensive and accurate analysis of cell location; B5. Loss of zoom The variable focal loss introduces a dynamic focus factor that can be dynamically adjusted according to the complexity, prediction probability, and confidence of the cell sample, thereby achieving a more accurate and balanced focus on the cell category. At the same time, the boundary-aware loss function significantly improves the positioning accuracy of the bounding box by prioritizing the optimization of the boundary area of ​​the object. By adding additional loss terms in the boundary area, this method further optimizes the regression effect of the bounding box. The mathematical expression of focal loss is: Mathematical expression of zoom loss: VFL(p t )=-α(1-p t ) γ log(p t_hat ) =-α(1-p t_hat ) β log(p t_hat ) Among them, p t Indicates the probability that the model predicts that the sample is a positive class, p t_hat It indicates the probability that the model predicts that the sample is positive and the prediction is correct. The parameter α is a balance factor, which is used to adjust the weights of positive and negative samples. When the predicted probability is close to 1, the focusing factor gradually decreases; when the predicted probability is close to 0, the focusing factor increases. The parameters γ and β are used to adjust the weights of samples that are easy to classify and accurate. The combination of these parameters improves the focusing accuracy and promotes the balance between positive and negative samples, thereby optimizing the performance of the model in detecting and segmenting cell targets.

4. The method for operating a cell manipulation platform combining a piezoelectric actuator and a six-axis robotic arm according to claim 3, characterized in that: In step 5), the microscope image coordinate system and the robot base coordinate system are calibrated to establish a spatial coordinate conversion relationship between the two; The robot arm and microscope camera are configured with the eye outside the hand. The purpose of calibration is to obtain the transformation matrix To lay the foundation for subsequent operations, the calibration process is also called the process of solving the equation AX=XB. The calculation process for the eye-outside-hand configuration is as follows: The robot arm and microscope camera adopt the configuration of eye outside the hand. The purpose of calibration is to determine the transformation matrix from the camera coordinate system to the robot arm base coordinate system. Provide basic support for subsequent operations. The calibration process is actually to solve A i X=XB i The process of the equation specifically involves the following steps: C1. Motion data collection Through multiple movements of the robot arm, the motion matrix in the robot arm base coordinate system is recorded and the corresponding motion matrix in the camera coordinate system Where i (i = 1, 2, 3 ... n) is the transformation matrix under the i-th data collection; C2. Constructing the system of equations According to A i X=XB i The constraint relationship is used to construct a set of equations using all the collected motion data, where X is the transformation matrix to be solved. The set of equations is as follows: C3. Solve the matrix X Use optimization algorithms or matrix decomposition methods to solve the optimal transformation matrix X that satisfies the equation, that is, the transformation matrix from camera to base Through this calibration process, the geometric relationship between the robotic arm and the microscope camera is accurately established, laying the foundation for other subsequent visual tasks.

5. The method for operating a cell manipulation platform combining a piezoelectric actuator and a six-axis robotic arm according to claim 4, characterized in that: In step 6), according to the cell position information, the six-axis robot arm multi-posture operation for cell micromanipulation includes the following steps: D1. Set the current position as the initial position: According to the DH parameter table below and the parameters of the microscope camera calibrated in 1 above, determine the coordinate transformation relationship between the camera and the robot. Set the joint angle θ0 of the robot at the current position and the end effector position T0. The order of parameter changes between adjacent joint coordinate systems is: α, a, θ, d, The rotation matrix is ​​as follows: D2. Collect cell images I through a microscope and use image processing algorithms to extract the location p of the target cells. c =(x c ,y c ,z c ), Convert the cell position from the image coordinate system to the robot base coordinate system: in, is the transformation matrix from camera to robot base, is the transformation matrix from image to camera; D3. According to the requirements of cell operation and the location of target cells c , at the target position p c On the basis of , the attitude angle (α, β, γ) is introduced to represent the rotation attitude of the end effector. α represents the angle of rotation around the x-axis, β represents the angle of rotation around the y-axis, and γ represents the angle of rotation around the z-axis. In this way, the target posture can be expressed as: T goal =T(p c )·R x (a)·R y (b)·R z (c) Among them, T(p c ) is p c is the translation matrix of the origin, R x (α), R y (β), R z (γ) are rotation matrices rotating around three axes. By introducing the attitude angle, the robot arm has the ability to operate cells in multiple attitudes, which can not only expand the operating range and flexibility of the robot arm, but also improve the accuracy and success rate of the operation; After determining the target position T of the end effector goal Finally, considering the special constraints of micromanipulation, the plan is made from the current pose T0 to the target pose T goal The movement path, The path can be discretized into a series of intermediate poses T1, T2, ..., T n , each posture corresponds to a time point t i , the pose change can be expressed by exponential mapping: D4. For each intermediate pose T i , solve the corresponding joint angle θ i , use numerical iteration to solve the inverse kinematics equations: f(θ i )=T i -T1(θ i1 )·T2(θ i2 )…T6(θ i6 )=0 Assuming there are multiple solutions, choose the one that matches the current joint angle θ i-1 The closest solution ensures the continuity of motion. D5. For the discrete joint angle sequence θ1,θ2,…,θ n Smoothing and optimization are performed, and spline interpolation is used to generate a continuous joint angle trajectory θ(t), and the continuity constraints of velocity and acceleration are satisfied. The dynamic characteristics of the robot arm are considered during optimization to minimize the joint torque or energy consumption: D6. Calculate the joint velocity based on the optimized joint angle trajectory θ(t) and acceleration Design a trajectory tracking controller to achieve precise motion control, and the torque control is as follows: At the same time, the cell position is detected in real time through visual feedback to compensate for motion errors. If the error exceeds the threshold, path planning and trajectory optimization are re-performed; D7. When the end effector reaches the target position T goal When the microneedle is inserted into the cell, the piezoelectric ceramic driver is activated to precisely control the micro-movement of the microneedle near the cell, achieving sub-micron-level precise alignment and operation. During the operation, the contact force with the cell is monitored by the force sensor to avoid damaging the cell. If the contact force exceeds the safety threshold, the end position is adjusted in time. After the operation is completed, the robot arm is controlled to return to the initial position T0 to prepare for the next operation. D8. During the entire operation, monitor the system status and performance indicators, such as joint angle, speed, torque, etc. If an error or abnormality is detected, stop the movement in time and execute the recovery procedure. The recovery procedure may include: returning to a safe position, recalibrating sensors, adjusting control parameters, etc. If necessary, send an audible alarm to the operator to request manual intervention and processing.

Images