An automatic initialization positioning device and method for the tip of an end effector of a micromanipulation system

Abstract

The present invention discloses an automatic initialization positioning device and method for an end effector tip of a micromanipulation system, which relates to the field of automated micromanipulation. The method comprises: S1, performing macroscopic end effector tip positioning, calculating camera posture using ArUco Tag, and realizing three-dimensional positioning of the end effector tip by triangulation, and moving the end effector to an area close to the center of a focal plane based on visual servo; S2, performing microscopic end effector tip positioning; S3, collecting calibration data based on visual servo, calibrating the end effector tip and a culture dish, and obtaining a left calibration matrix and a right calibration matrix reflecting the conversion relationship between left and right micromanipulator coordinates and image coordinates; and generating an intuitive interactive microscope interface based on a mouse and a keyboard according to the position information collected by S1-S3. The method mentioned in the present application has obvious advantages in terms of operation accuracy, operation efficiency and repeatability, is easy to deploy, has strong compatibility, and is adapted to various magnifications of microscopes.

Description

Technical Field

[0001] The present invention relates to the field of automated micromanipulation, in particular to an automatic initialization positioning device and method for the tip of an end effector of a micromanipulation system. Background Art

[0002] At present, the initialization positioning of the tip of the end effector of the micromanipulation system is usually completed manually by the operator. In manual initialization positioning, the operator needs to move the end effector, move its tip from the initial position away from the focal plane of the microscope to the field of view of the microscope and complete the focus. However, the above operation involves repeated conversion between macro and micro fields of view, which is cumbersome and time-consuming, and can easily cause accidental damage to the tip of the end effector. In addition, the end effector usually needs to be replaced frequently during the micromanipulation process, and the above initialization positioning operation needs to be repeated after each replacement.

[0003] In recent years, micromanipulation technology has been widely used in related research and practice in biomedical engineering fields such as assisted reproduction, pathological diagnosis, tissue engineering and regenerative medicine. However, current micromanipulation is still highly dependent on manual labor. In order to improve the accuracy, efficiency and consistency of micromanipulation and reduce the learning cost and labor intensity of operators, automated micromanipulation has become a research hotspot. Although relevant studies have achieved the automation of subtasks such as focusing, target tracking, and trajectory planning during micromanipulation, there is still a lack of convenient and effective solutions for the problem of automatic initialization positioning of the end effector tip of the micromanipulation system.

[0004] Before performing a specific operation, it is crucial to accurately position the tip of the end effector. This is because errors in the initial positioning process will continue to affect the accuracy of subsequent micromanipulations. However, the initial positioning of the end effector tip is still highly dependent on manual labor and has a steep learning curve. Because the diameter of the end effector tip is usually at the micro-nano level, which exceeds the perception range of the human eye, even a slight operational error may cause damage to the end effector tip due to collision.

[0005] To avoid damage to the tip of the end effector due to collision, the installation of the end effector is usually completed in an open space away from the culture dish. This means that the initial position of the tip of the end effector will be beyond the field of view of the microscope. Therefore, the operator needs to estimate the relative position of the tip of the end effector and the microscope through the macroscopic field of view of the human eye, and move it to the vicinity of the microscopic field of view of the microscope to complete the rough positioning. After that, the operator also needs to observe the microscopic field of view of the microscope while sweeping the end effector back and forth, and find the tip position and complete the focus based on the shadow produced by the end effector in the microscopic field of view.

[0006] Manual positioning operations involve repeated switching between macro and micro fields of view, which is tedious and time-consuming, and can easily cause accidental damage to the tip of the end effector. In addition, the end effector usually needs to be replaced frequently during micromanipulation, and the above initialization positioning operation needs to be repeated after each replacement.

[0007] There is limited research on the automatic initialization positioning of the end-effector tip of a micromanipulation system.

[0008] Liu et al. realized automatic tip positioning and focusing within a range of 4x4x4 mm using a detection algorithm based on motion history images and active contour models. They detected the tip of the end effector and moved it to the center of the microscope image field of view by estimating the position of the end effector tip and using an adaptive quadtree autofocus algorithm. This method requires the end effector tip to be manually moved to a specific area close to the microscope's microscopic field of view in advance, and different microscope magnifications (x4, x10, x20) need to be switched during the positioning process. The operation is complicated and is only applicable to commercial inverted microscopes with objective lens converters.

[0009] Su et al. proposed a method for positioning the tip of the end effector using an additional macro camera, which achieved two-dimensional positioning of the tip of the end effector and the microscope under the macroscopic field of view, and was able to move the tip of the end effector from the macroscopic field of view to the microscopic field of view and focus it. However, this method can only achieve tip positioning under the low-magnification objective of the microscope (4 times in the paper) because it cannot obtain the distance between the tip of the end effector and the depth direction of the focal plane of the microscope. Under the high-magnification objective, the shallow depth of field will make the target far from the focal plane difficult to identify or even disappear completely.

[0010] Therefore, in order to reduce human intervention and realize automated micromanipulation, an automated technology that can easily locate the tip of the end effector is urgently needed. Summary of the invention

[0011] To solve the above problems, the present application provides an end effector tip automatic initialization positioning device and method for a micromanipulation system, with the goal of improving accuracy and efficiency, aiming to achieve easy deployment and improved compatibility.

[0012] In the present application, there is provided an automatic initialization positioning device for the tip of an end effector for a micromanipulation system, comprising a micromanipulator, a macro-micro combined imaging system, and a computer / user interface;

[0013] The micromanipulator includes two sets of three-axis Cartesian manipulators, which are symmetrically distributed on the left and right sides of the microscope. The three-axis Cartesian manipulators are used to control the end effector and the culture dish respectively. The two sets of three-axis Cartesian manipulators are defined as the left arm and the right arm respectively.

[0014] The macro-micro combined imaging system includes a microscopic image part and a macroscopic image part. The microscopic image part is collected by a 20x objective lens digital camera, and the macroscopic image part is collected by a macroscopic camera. The macro and micro images will eventually be transmitted to a computer / user interface respectively.

[0015] In the present application, a method for automatic initialization positioning of the tip of an end effector of a micromanipulation system is also provided, comprising the following steps:

[0016] S1. Perform macro positioning of the end effector tip. The macro camera collects a macro image of the micro-operation space, moves the end effector to analyze the position of its tip in the image coordinate system, calculates the camera pose using the reference QR code ArUco Tag used for positioning, and realizes the three-dimensional positioning of the end effector tip through triangulation. The end effector is moved to an area close to the center of the focal plane based on visual servoing.

[0017] S2, positioning the tip of the micro end effector in a 4x2x2mm rectangular area, using the shadow of the end effector projected in the microscope image to locate the center axis and tip of the end effector, and moving the tip of the end effector to the focal plane;

[0018] S3, based on visual servoing, collecting calibration data, wherein the calibration data is calibration data including microscopic image position information and micromanipulator encoder position information, calibrating the tip of the end effector and the culture dish, and obtaining a left calibration matrix and a right calibration matrix reflecting the conversion relationship between the left and right micromanipulator coordinates and the image coordinates;

[0019] S4. Generate an intuitive human-computer interaction interface based on a mouse and keyboard according to the position information collected by S1-S3. The microscopic information in the interface is used for local precision microscopic operations, and the macroscopic information is used to help the operator grasp the global environment.

[0020] Preferably, the specific steps of S1 are as follows:

[0021] S101, based on ArUco Tag, select the region of interest (ROI) in the macro camera image, and locate the two-dimensional position of the end effector tip in the image coordinate system through micromanipulator movement;

[0022] S102, estimating the three-dimensional position of the tip of the end effector by a triangulation method based on the image coordinates of the tip before and after the movement and the corresponding camera posture;

[0023] S103, analyzing the position difference between the center of the microscope focal plane and the tip of the end effector, and moving the tip of the end effector to the target area through visual servoing.

[0024] Preferably, the specific contents of selecting a region of interest (ROI) in the macro camera image and locating the two-dimensional position of the tip of the end effector in the image coordinate system in S101 are as follows:

[0025] Automatically select the region of interest (ROI) containing the end effector tip based on the fiducial markers;

[0026] Based on the pre-calibrated monocular camera, the intrinsic parameters of the macro camera are obtained. and distortion coefficient :

[0027] ;

[0028] There are two ArUco Tags, namely Tag1 and Tag2, where Tag2 is bound to the end effector of the left arm and moves with the micromanipulator, while Tag1 is bound to the microscope and remains stationary during the micromanipulation process;

[0029] The coordinate system of Tag1 and Tag2 is obtained through the calibrated camera and the ArUco Tag of known size. With the macro camera coordinate system The conversion relationship between them;

[0030] The expressions of the coordinate system of Tag1 and Tag2 and the macro camera coordinate system are:

[0031] ;

[0032] in, is the Tag1 or Tag2 coordinate system, and are the homogeneous coordinates of a point in the three-dimensional space in the macro camera and QR code coordinate systems, respectively. is the transformation matrix, which is Rotation Matrix and Translation Vector constitute;

[0033] Based on Tag1 bound to the microscope, according to the travel of the micromanipulator, in the Tag1 coordinate system Next, select the rectangular area that contains the motion range of the end effector tip;

[0034] Calculate the coordinates of the projection points of the cuboid endpoints in the Tag1 coordinate system in the macro camera image;

[0035] The expression for calculating the coordinates of the projection points of the cuboid endpoints in the Tag1 coordinate system in the macro camera image is:

[0036] ;

[0037] in, represents the scaling factor, represents the pixel coordinates of the projection point, Represents the point in the tag1 coordinate system, Represents the projection matrix from Tag1 to the macro camera;

[0038] The circumscribed rectangle of the convex hull of the projection point is taken as the ROI area;

[0039] The end effector is controlled to move horizontally for a certain distance and the image matrix before and after the movement is collected. as well as And calculate the image difference;

[0040] According to the influence of light on the end effector, define The positive part of and negative part They correspond to the end effector before and after the movement respectively;

[0041] right Binarization is performed to obtain a silhouette image of the end effector and determine the maximum contour of the image;

[0042] The expression for calculating image difference is:

[0043] ;

[0044] The point at the lower right of the maximum contour of the image is taken as the position of the tip of the end effector before the movement;

[0045] exist The template is selected based on the tip position of the end effector, and the template matching method is used to Find the end effector tip position after the motion in .

[0046] Preferably, the specific content of estimating the three-dimensional position of the tip of the end effector by the triangulation method in S102 is:

[0047] Obtain the image coordinates of the end effector tip and calculate the three-dimensional position of the end effector tip in the Tag2 coordinate system by triangulation;

[0048] Combining the camera intrinsic parameter K and the distortion parameter D, we can obtain the pixel coordinates of the end effector tip before and after the movement in the dedistorted image. ;

[0049] The camera projection matrix corresponding to the pixel coordinates is obtained by keeping tag2 relatively still with the tip of the end effector ;

[0050] Define the homogeneous three-dimensional coordinates of the tip in the tag2 coordinate system as ;

[0051] The three-dimensional point With 2D image points The conversion relationship is: , where p is the pixel coordinate of the end effector tip before and after the movement in the dedistorted image, represents the projection matrix from Tag2 to the macro camera, They represent the rotation and projection matrices from Tag2 to the macro camera respectively, and K is the camera intrinsic parameter.

[0052] according to and The vector parallel relationship is obtained and The relationship between

[0053] and The relationship is:

[0054] ;

[0055] in, To constitute of row vector;

[0056] Based on the tip coordinates after movement and the corresponding camera projection matrix get and The relationship between

[0057] and The relationship is:

[0058] ;

[0059] Will and The relationship between and In the relationship between x and y, SVD is applied to the equation x to solve the target 3D point q in the case of noise;

[0060] The expression for solving the equation x by applying SVD is:

[0061] ;

[0062] Analyze the expression and get The target three-dimensional point q, ;

[0063] in, Representation and Matrix The smallest non-zero singular value of The associated eigenvectors, according to the definition of SVD decomposition, is a matrix The last column in ;

[0064] Calculate the normalized three-dimensional coordinates of the tip, and use the coordinate system conversion expression to obtain the coordinates of the end effector tip relative to tag1 after movement. ;

[0065] The coordinate system transformation expression is: ;

[0066] The coordinates obtained for the latest 5 frames of images The data is averaged to obtain a smoothed result .

[0067] Preferably, the specific contents of analyzing the position difference between the center of the focal plane of the microscope and the tip of the end effector in S103 and moving the tip of the end effector to the target area by visual servoing are:

[0068] Manually move the tip of the end effector to the center of the microscope screen and complete the focus. The position of the end effector tip obtained at this time As ;

[0069] Based on the pre-recorded focal plane center position , the system will select as the target location;

[0070] in, , including horizontal offset, which facilitates the end-effector tip positioning process based on sweeping in the next stage;

[0071] Target distance difference based on real-time feedback The scheme combined with the pseudocode to implement visual servoing will first control the tip of the manipulator to move above the target area, and then move it down to the target area, thereby avoiding contact with the outer wall of the culture dish, where .

[0072] Preferably, the specific content of performing microscopic end effector tip positioning in S2 is:

[0073] S201, collecting microscope images and completing the initialization of the actuator scanning detector;

[0074] S202, estimating the position of the tip of the end effector based on a scanning method, and moving it into the field of view of the microscope;

[0075] S203, moving the tip of the end effector to the focal plane through two rounds of automatic focusing.

[0076] Preferably, the specific contents of S201, collecting microscope images and completing the initialization of the actuator scanning detector are:

[0077] The actuator sweeps the detector to segment the microscope image into sub-areas, any of which can be used express, ;

[0078] During the initialization process, the detector collects 60 frame difference images, calculates the standard deviation of the average gray value of each sub-region, and sets the threshold according to the standard deviation;

[0079] The expression for calculating the standard deviation of the average gray value of each sub-region is:

[0080] ;

[0081] In the formula, represents the standard deviation of the corresponding area, Indicates the trigger coefficient preset according to the imaging conditions;

[0082] During the detection process, if the average grayscale of the frame difference image of the region exceeds the corresponding threshold, the sub-region is marked as triggered.

[0083] Preferably, the specific steps of S202, estimating the position of the tip of the end effector based on the scanning method and moving it into the field of view of the microscope are:

[0084] S2021, along Sweep the end effector in a certain direction to bring it into the picture;

[0085] S2022, record the end effector position in the current micromanipulator coordinate system, and continue moving along the original sweeping direction Ensure that the shadow of the end effector leaves the field of view and moves along Direction movement distance Complete the end effector retraction;

[0086] S2023, along The axis sweeps the end effector in the reverse direction. If the end effector is detected during the motion, S2022 is repeated. If the end effector is not detected, it is judged that the end effector has been retracted;

[0087] S2024, based on the last two records The average value of the direction position, the system along Axis moves the middle axis of the end effector into the picture and controls the end effector to move along Direction of movement , move the end effector tip into the microscope field of view.

[0088] In summary, the present application provides an automatic initialization positioning device and method for the tip of an end effector of a micromanipulation system. Compared with manual initialization positioning, this method has obvious advantages in terms of operation accuracy, operation efficiency and repeatability. In addition, this method is easy to deploy. It only needs to supplement the original micromanipulation system with a macro camera and a QR code for positioning to achieve automatic three-dimensional positioning of the tip of the end effector. In addition, the method mentioned in this application has strong compatibility and can adapt to a variety of magnifications of microscopes.

[0089] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 Schematic diagram of the initial positioning process of the end effector tip of a traditional micromanipulator;

[0091] Figure 2 It is a system overview diagram of an automatic initialization positioning device for the tip of an end effector of a micromanipulation system of the present invention;

[0092] Figure 3 A flow chart of a method for automatically initializing and positioning the tip of an end effector of a micromanipulation system according to the present invention;

[0093] Figure 4 A schematic diagram of macro positioning operation of an automatic initialization positioning method for the tip of an end effector of a micromanipulation system according to the present invention;

[0094] Figure 5 A schematic diagram of a microscopic positioning operation of an automatic initialization positioning method for the tip of an end effector of a micromanipulation system according to the present invention;

[0095] Figure 6 This is a diagram showing the experimental results of the macroscopic tip positioning accuracy of an automatic initialization positioning method for the tip of an end effector used in a micromanipulation system of the present invention;

[0096] Figure 7 This is a graph showing the experimental results of the automatic initialization execution time of a method for automatic initialization positioning of the tip of an end effector of a micromanipulation system according to the present invention. DETAILED DESCRIPTION

[0097] The technical solution of the present invention is further described below by means of the accompanying drawings and embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present application.

[0098] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0099] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0100] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0101] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0102] At present, the initialization positioning of the tip of the end effector of the micromanipulation system is usually completed manually by the operator. In the manual initialization positioning, the operator needs to move the end effector from an initial position away from the focal plane of the microscope to within the field of view of the microscope and complete the focus.

[0103] like Figure 1 As shown, Figure 1 (a) is the initial state without operation. Figure 1 (b) shows the rough positioning of the microscope. Figure 1 Middle (c) Microscopic field of view precise positioning state, such as Figure 1 To avoid damage to the end effector tip due to collision, the end effector is usually mounted in an open space away from the culture dish. This means that the initial position of the end effector tip will be beyond the field of view of the microscope. Figure 1 In (b), the operator needs to estimate the relative position of the end effector tip and the microscope through the macroscopic field of view of the human eye, and move it to the vicinity of the microscopic field of view of the microscope to complete the rough positioning. Figure 1In (c), the operator also needs to observe the microscope's microscopic field of view while sweeping the end effector back and forth, and find the tip position and complete the focus based on the shadow produced by the end effector in the microscopic field of view. However, the above operation involves repeated conversion between macro and micro fields of view, which is cumbersome and time-consuming, and can easily cause accidental damage to the tip of the end effector. In addition, the end effector usually needs to be replaced frequently during micromanipulation, and the above initialization positioning operation needs to be repeated after each replacement.

[0104] like Figure 2 As shown, in the present application, there is provided an automatic initialization positioning device for the tip of an end effector of a micromanipulation system, comprising a micromanipulator, a macro-micro combined imaging system, and a computer / user interface;

[0105] The micromanipulator includes two sets of three-axis Cartesian manipulators, which are used to control the end effector and the culture dish respectively. The three-axis Cartesian manipulators are symmetrically distributed on the left and right sides of the microscope. The three-axis Cartesian manipulators are used to control the end effector and the culture dish respectively. The two sets of three-axis Cartesian manipulators are defined as the left arm and the right arm respectively. Each axis of the micromanipulator (8MT173, Standa Ltd., Lithuania) has a single-step resolution of 1.25 microns and a working range of 25 mm, and can achieve a resolution of up to 5 nm under the subdivision control of a multi-axis motion controller (8SMC5, Standa Ltd., Lithuania).

[0106] The macro-micro combined imaging system includes a microscopic image unit and a macroscopic image unit. The microscopic image unit is collected by a 20x objective digital camera, and the macroscopic image unit is collected by a macro camera. The macroscopic and microscopic images will eventually be transmitted to the computer / user interface respectively. The microscopic image is collected by a digital camera (CS165MU, Thorlabs Inc., NJ, USA) equipped with a 20x objective (Plan Flour 20X / 0.50, Nikon, Japan) and transmitted to the computer at a speed of 30fps / s and a resolution of 1440x1080. The macroscopic field of view is provided by an additional macro camera (RealSense SR305, Intel, CA, USA. Only the RGB camera module is used), which can collect images with a resolution of 1920x1080 at 15fps, thereby providing the system with macroscopic information including the end effector, culture dish and ArUco Tag. The two ArUco Tags used in the system are both 45mm in size and are fixed to the end effector and microscope respectively through 3D printed connectors.

[0107] The system's programs, including image processing, manipulator control, and user interface, are developed in Python and run on a computer with a multi-core processor (Intel Core i7-8750H CPU @ 2.2 GHz). The above programs can be easily deployed on any laptop running Windows 10 operating system.

[0108] In the present application, a method for automatically initializing and positioning the tip of an end effector of a micromanipulation system is also provided, such as Figure 3 As shown, this step can be subdivided into three sub-steps from macro to micro: macro end effector tip positioning, micro end effector tip positioning. Specifically, it includes the following steps:

[0109] S1. Perform macro positioning of the end effector tip. The macro camera collects a macro image of the micro-operation space, moves the end effector to analyze the position of its tip in the image coordinate system, calculates the camera pose using the reference QR code ArUco Tag used for positioning, and realizes the three-dimensional positioning of the end effector tip through triangulation. The end effector is moved to an area close to the center of the focal plane based on visual servoing.

[0110] S2, positioning the tip of the micro end effector in a 4x2x2mm rectangular area, using the shadow of the end effector projected in the microscope image to locate the center axis and tip of the end effector, and moving the tip of the end effector to the focal plane;

[0111] S3, based on visual servoing, calibration data is collected, the calibration data is calibration data including microscopic image position information and micromanipulator encoder position information, the end effector tip and the culture dish are calibrated, and a left calibration matrix and a right calibration matrix reflecting the conversion relationship between the left and right micromanipulator coordinates and the image coordinates are obtained;

[0112] S4. Generate an intuitive human-computer interaction interface based on the mouse and keyboard according to the position information collected by S1-S3. The microscopic information in the interface is used for local precision micromanipulation, and the macroscopic information is used to help the operator grasp the global environment. During the operation, the system will detect the tip position based on visual feedback, and automatically perform recalibration when errors occur in the calibration-based control to improve the robustness and accuracy of the control.

[0113] Preferably, the specific steps of S1 are as follows:

[0114] S101, based on ArUco Tag, select the region of interest (ROI) in the macro camera image, and locate the two-dimensional position of the end effector tip in the image coordinate system through micromanipulator movement;

[0115] S102, estimating the three-dimensional position of the tip of the end effector by a triangulation method based on the image coordinates of the tip before and after the movement and the corresponding camera posture;

[0116] S103, analyzing the position difference between the center of the microscope focal plane and the tip of the end effector, and moving the tip of the end effector to the target area through visual servoing.

[0117] Preferably, the specific contents of selecting a region of interest (ROI) in the macro camera image and locating the two-dimensional position of the tip of the end effector in the image coordinate system in S101 are as follows:

[0118] Automatically select the region of interest (ROI) containing the end effector tip based on the fiducial markers;

[0119] Macro cameras can obtain comprehensive environmental information about micro-operation systems through a viewing angle and depth of field that is much larger than that of microscopes. However, the target of most concern in this application, the end effector, only occupies a small part of the screen. Therefore, in order to avoid interference from other objects in the background and improve the stability of target recognition, the system first automatically selects the region of interest (ROI) containing the tip of the end effector based on the reference marker.

[0120] Based on the pre-calibrated monocular camera, the intrinsic parameters of the macro camera are obtained. and distortion coefficient :

[0121] ;

[0122] like Figure 2 As shown, there are two ArUco Tags, namely Tag1 and Tag2, where Tag2 is bound to the end effector of the left arm and moves with the micromanipulator, while Tag1 is bound to the microscope and remains stationary during the micromanipulation process. ArUcoTag is a reference QR code for positioning, which is used to provide efficient real-time 3D position and attitude estimation.

[0123] The coordinate system of Tag1 and Tag2 is obtained through the calibrated camera and the ArUco Tag of known size. With the macro camera coordinate system The conversion relationship between them;

[0124] The expressions of the coordinate system of Tag1 and Tag2 and the macro camera coordinate system are:

[0125] ;

[0126] in, is the Tag1 or Tag2 coordinate system, and are the homogeneous coordinates of a point in the three-dimensional space in the macro camera and QR code coordinate systems, respectively. is the transformation matrix, which is Rotation Matrix and Translation Vector constitute;

[0127] like Figure 4 As shown, Figure 4 (a) is the system diagram before and after the tool moves. Figure 4 (b) is the selected ROI (region of interest). Figure 4 (c) shows the tool tip positioning before movement based on profile analysis. Figure 4 (d) shows the tool tip positioning after movement based on template matching.

[0128] Based on Tag1 bound to the microscope, according to the travel of the micromanipulator, in the Tag1 coordinate system Select the rectangular area that contains the motion range of the end effector tip. Figure 4 As shown in (a);

[0129] Calculate the coordinates of the projection points of the cuboid endpoints in the Tag1 coordinate system in the macro camera image;

[0130] The expression for calculating the coordinates of the projection points of the cuboid endpoints in the Tag1 coordinate system in the macro camera image is: ;

[0131] in, represents the scaling factor, represents the pixel coordinates of the projection point, Represents the point in the tag1 coordinate system, Represents the projection matrix from Tag1 to the macro camera;

[0132] The circumscribed rectangle of the convex hull of the projection point is taken as the ROI area;

[0133] The end effector is controlled to move horizontally for a certain distance and the image matrix before and after the movement is collected. as well as And calculate the image difference;

[0134] According to the influence of light on the end effector, define The positive part of and negative part They correspond to the end effector before and after the movement respectively;

[0135] right Binarization is performed to obtain a silhouette image of the end effector and determine the maximum contour of the image;

[0136] The expression for calculating image difference is: ;

[0137] The point at the lower right of the maximum contour of the image is taken as the position of the tip of the end effector before the movement;

[0138] exist The template is selected based on the tip position of the end effector, and the template matching method is used to Find the end effector tip position after the motion in .

[0139] The end effector is illuminated and has a higher brightness, so it can be considered that the positive and negative parts of correspond to the end effector before and after the movement, respectively. Figure 4 As shown in (c), the system will obtain a silhouette image of the end effector by binarization, find the maximum contour of the image, and use its lower right point as the position of the tip of the end effector before movement.

[0140] After obtaining the position of the tip before movement, the system will select a template at the end effector tip position in , and find the end effector tip position after movement in through the template matching method, so as to ensure the consistency of tip detection, such as Figure 4 In order to reduce the impact of image brightness changes and ensure the accuracy of visual tracking, the template matching method used in this work is based on the normalized correlation coefficient.

[0141] After obtaining the image coordinates of the end effector tip, this study took a method to treat the movement of the end effector as the equivalent process of the camera moving in the opposite direction at the same distance. With the help of this perspective transformation, this application can accurately calculate the three-dimensional position of the end effector tip in the Tag2 coordinate system through triangulation.

[0142] Triangulation is a method of determining a point in three-dimensional space based on its projections on different images. This application requires two sets of pixel coordinates of the target at different viewing angles and the corresponding camera projection matrix.

[0143] Preferably, the specific content of estimating the three-dimensional position of the tip of the end effector by the triangulation method in S102 is:

[0144] Obtain the image coordinates of the end effector tip and calculate the three-dimensional position of the end effector tip in the Tag2 coordinate system by triangulation;

[0145] Combining the camera intrinsic parameter K and the distortion parameter D, we can obtain the pixel coordinates of the end effector tip before and after the movement in the dedistorted image. ;

[0146] The camera projection matrix corresponding to the pixel coordinates is obtained by keeping tag2 relatively still with the tip of the end effector ;

[0147] Define the homogeneous three-dimensional coordinates of the tip in the tag2 coordinate system as ;

[0148] The three-dimensional point With 2D image points The conversion relationship is: ;

[0149] Where p is the pixel coordinate of the end effector tip before and after the movement in the dedistorted image, represents the projection matrix from Tag2 to the macro camera, They represent the rotation and projection matrices from Tag2 to the macro camera, respectively, and K is the camera intrinsic parameter;

[0150] according to and The vector parallel relationship is obtained and The relationship between

[0151] and The relationship is: ;

[0152] in, To constitute of The row vector, p, is the pixel coordinates before and after the end effector tip moves in the dedistorted image;

[0153] Based on the tip coordinates after movement and the corresponding camera projection matrix get and The relationship between

[0154] and The relationship is: ;

[0155] Will and The relationship between and In the relationship between x and y, SVD is applied to the equation x to solve the target 3D point q in the case of noise;

[0156] It can be observed that in the formula xy, the third line is actually a linear combination of the first two lines. Combining the formula x with y, we get: ;

[0157] In triangulation, the present application knows the matrix A, and can use the singular value decomposition (SVD) method to solve the optimal three-dimensional point q in the case of noise. Apply SVD to equation x.

[0158] The expression for solving the equation x by applying SVD is: ;

[0159] Analyze the expression and get The target three-dimensional point q, ;

[0160] in, Representation and Matrix The smallest non-zero singular value of The associated eigenvectors, according to the definition of SVD decomposition, is a matrix The last column in , As The optimal solution to the equation, which actually minimizes Length.

[0161] Calculate the normalized three-dimensional coordinates of the tip , and through the coordinate system conversion expression, the coordinates of the end effector tip relative to tag1 after movement are obtained ;

[0162] The coordinate system transformation expression is: ;

[0163] Through the above formula, the present application can realize real-time tracking of the tip of the end effector when the micromanipulator moves based on the positioning of the reference mark.

[0164] To improve the stability and accuracy of positioning, the system performs average filtering on the continuously collected 3D position data of the end effector tip. Specifically, the system averages the data obtained based on the latest 5 frames of images to obtain a smoothed result. .

[0165] After determining the three-dimensional position of the end effector tip, in order to enable it to move accurately to the target area, the application needs to determine the center of the focal plane such as Figure 4 The three-dimensional position of the point in (a) is shown in Figure 1. This can be achieved through a one-time manual end-effector tip positioning: after the end-effector tip positioning and tracking in the previous section, the end-effector tip is manually moved to the center of the microscope image and the focus is completed. The end-effector tip position obtained at this time is used as the end-effector tip position. Note that this process only needs to be performed once during the system setup phase, and the system will automatically record the value for subsequent experiments, because the relative position of the center of the focal plane and Tag1 will no longer change in subsequent experiments.

[0166] Preferably, the specific contents of analyzing the position difference between the center of the focal plane of the microscope and the tip of the end effector in S103 and moving the tip of the end effector to the target area by visual servoing are:

[0167] Manually move the tip of the end effector to the center of the microscope screen and complete the focus. The position of the end effector tip obtained at this time As ;

[0168] Based on the pre-recorded focal plane center position , the system will select as the target location;

[0169] in, , including horizontal offset, which facilitates the end-effector tip positioning process based on sweeping in the next stage;

[0170] Target distance difference based on real-time feedback The scheme combined with the pseudocode to implement visual servoing will first control the tip of the manipulator to move above the target area, and then move it down to the target area, thereby avoiding contact with the outer wall of the culture dish, where .

[0171] The pseudo code is:

[0172] ;

[0173] As shown in the pseudo code, the system will first control the tip of the manipulator to move above the target area, and then move it down to the target area, thereby avoiding contact with the outer wall of the culture dish. In addition, the movement speed of each axis of the micromanipulator decreases as the distance from the target decreases.

[0174] After the end effector tip positioning and servoing in the previous stage, the tip is roughly positioned and moved to a 4x2x2mm area close to the center of the microscope focal plane. Figure 5 As shown, Figure 5 (a) shows the initial end effector state and motion trajectory. Figure 5 (b) is a schematic diagram of the actuator sweeping the detector. Because it is preset, the tip will be in an initial plane higher than the focal plane and biased to the upper right, as shown in Figure 5 In the microscopic end-effector tip positioning stage, the system estimates the position of the end-effector's central axis and the end-effector's tip by scanning based on the microscope image, moves it from the initial state to the microscope field of view, and moves the tip to the focal plane through two rounds of autofocus.

[0175] Through the microscopic end-effector tip positioning in this section, the system will move the end-effector tip into the field of view of the microscope and achieve focusing.

[0176] Preferably, the specific content of performing microscopic end effector tip positioning in S2 is:

[0177] S201, collecting microscope images and completing the initialization of the actuator scanning detector;

[0178] S202, estimating the position of the tip of the end effector based on a scanning method, and moving it into the field of view of the microscope;

[0179] S203, moving the tip of the end effector to the focal plane through two rounds of automatic focusing.

[0180] Preferably, the specific contents of S201, collecting microscope images and completing the initialization of the actuator scanning detector are:

[0181] For the end effector above the focal plane, it will generate an enlarged and blurred shadow in the focal plane. The area of ​​the shadow relative to the field of view will increase with the distance between the end effector and the focal plane and the increase of the microscope magnification. In this section, we will analyze the above shadow during the motion of the end effector by constructing an actuator sweep detector based on frame difference images, so as to estimate the position of the end effector tip.

[0182] The actuator sweeps the detector to segment the microscope image into sub-areas, any of which can be used express, ;

[0183] During the initialization process, the detector collects 60 frame difference images, calculates the standard deviation of the average gray value of each sub-region, and sets the threshold according to the standard deviation;

[0184] The expression for calculating the standard deviation of the average gray value of each sub-region is: ;

[0185] In the formula, represents the standard deviation of the corresponding area, Indicates the trigger coefficient preset according to the imaging conditions;

[0186] During the detection process, if the average grayscale of the frame difference image of the region exceeds the corresponding threshold, the sub-region is marked as triggered.

[0187] Furthermore, after completing the initialization of the actuator sweep detector, the system will control the end effector to sweep along the direction. When the shadow of the end effector enters the screen, the corresponding sub-area will be triggered, such as Figure 5(b). When the three sub-areas at the bottom are triggered simultaneously, if the other six sub-areas are all triggered within the previous 0.5 seconds, the system will determine that the end effector has entered the screen. Similarly, when the end effector moves in the direction, the system will analyze whether the other six sub-areas are triggered within 0.5 seconds when the three sub-areas at the top are triggered simultaneously to determine whether the end effector has entered the screen.

[0188] The actuator sweep detector determines whether the end effector enters the field of view by analyzing the information of the complete frame difference image, which can effectively avoid the interference caused by local cell movement and is robust to the slow changes in illumination during the movement of the end effector.

[0189] Based on the actuator sweep detector, the system will control the end effector tip along the following Figure 5 In (a), the red track moves to the microscope field of view. Compared with the common traversal scanning starting from a corner of the workspace, the tip positioning method proposed in the present invention first locates the middle axis of the end effector and then retracts the end effector to locate the tip, which can reduce the time required for positioning and improve efficiency.

[0190] S202, the specific steps of estimating the position of the tip of the end effector based on the scanning method and moving it into the field of view of the microscope are:

[0191] S2021, along Sweep the end effector in a certain direction to bring it into the picture;

[0192] S2022, record the end effector position in the current micromanipulator coordinate system, and continue moving along the original sweeping direction Ensure that the shadow of the end effector leaves the field of view and moves along Direction movement distance Complete the end effector retraction;

[0193] S2023, along The axis sweeps the end effector in the reverse direction. If the end effector is detected during the motion, S2022 is repeated. If the end effector is not detected, it is judged that the end effector has been retracted;

[0194] S2024, based on the last two records The average value of the direction position, the system along Axis moves the middle axis of the end effector into the picture and controls the end effector to move along Direction of movement , move the tip of the end effector into the field of view of the microscope. In summary, compared with the traditional technology, the present invention proposes a method for automatically initializing and positioning the tip of an end effector for a micromanipulation system. The method combines macroscopic and microscopic visual information and uses computer vision to realize automatic initialization and positioning of the tip of the end effector of the micromanipulation system.

[0195] This application conducts experiments on macroscopic tip positioning accuracy and automatic initialization execution time.

[0196] like Figure 6 As shown, the three-dimensional positioning of the tool tip based on the macro camera can help the system identify the approximate position of the tool tip relative to the microscopic field of view, and its accuracy will directly affect the time and success rate of subsequent tip positioning in the microscopic field of view.

[0197] Therefore, this paper conducted an experiment on the three-dimensional positioning accuracy of the needle tip based on a macro camera. The experimental process is as follows: First, the operator manually moves the tool tip to the center of the screen and completes the focus. The tip position at this time is used as the origin, and the position of the robotic arm is recorded. After that, the system controls the tool to move to a random position higher than the culture dish (simulating the state after the tool is replaced in cell operation) and performs three-dimensional positioning of the needle tip based on a macro camera. After completing the three-dimensional positioning, the system controls the tool to move to the previously recorded origin position and reads the three-dimensional position of the tip estimated based on the reference QR code. At this time, the difference with the center position of the focus plane is the three-dimensional positioning error. The camera is collected when it is in a normal view.

[0198] like Figure 6 As shown, Figure 6 (a) is a schematic diagram of the macroscopic needle tip positioning accuracy experiment. Figure 6 (b)-(d) are schematic diagrams of macro positioning errors under different camera viewing angles. This application conducted experiments on the macro three-dimensional positioning accuracy under five different camera viewing angles to verify the system performance, namely, front view, left and right 30°, up and down 15° (indicated by different colors), and 30 groups of test experiments were conducted for each viewing angle. For each viewing angle, the distance between the camera and the tool tip remains basically the same, about 240mm. (The reason why the range of the up and down viewing angle is smaller than the left and right tilt viewing angle is that the culture dish may block the tool tip and affect its visual positioning when pitching).

[0199] Figure 6 (b)-(d) show the error distribution of five viewing angles in different directions.

[0200] It can be seen that the tool tip positioning method based on macro camera proposed in this paper can achieve high three-dimensional positioning accuracy under different camera viewing angles. Specifically, the error in the axial direction is less than 2mm, and the error in the axis-to-axis direction is less than 1mm. The five camera viewing angles can meet the accuracy requirements of the subsequent tool tip positioning task under the microscopic field of view. The reason for the large error in the axial direction is that this direction is close to the optical axis direction of the macro camera, and the depth information of the monocular camera is estimated based on image features, which usually has a relatively large error. In the micro-estimation positioning stage, the position of the tool in the axial direction can be quickly determined by the tool sweep detector proposed in this paper, so a slightly larger error in the axial direction is acceptable.

[0201] For the axis errors under different camera viewing angles ( Figure 6 In (b), the average error in the front view direction (viewing angle 1) is the smallest and has a small fluctuation, the data stability in the left 30° direction (viewing angle 2) is the highest, and the data error in the right 30° direction (viewing angle 3) is large and unstable. The main reason for this result is that the tag2 used for macro positioning in viewing angle 2 is close to the camera, and the larger imaging area helps to improve the accuracy and stability of the visual detection algorithm. On the contrary, the distance between viewing angle 3 and tag2 is far, resulting in a decrease in accuracy and stability.

[0202] pass Figure 6 It can be seen from (b)-(d) that among the five viewing angles, the overall effect of three-dimensional positioning of the tool tip based on viewing angle 1 is the best, with the smallest average error and higher stability (the maximum errors in the axial direction are 0.912mm, -0.049mm, -0.100mm, the average errors are 0.213mm, -0.087mm, -0.030mm, and the standard deviations are 0.376, 0.024, and 0.038, respectively). This result is partly due to the fact that the acquisition of the center position of the focal plane is based on viewing angle 1, which helps to reduce the influence of systematic errors in the positioning process and reduce the average error. In addition, the tip position of the tool can be clearly observed under viewing angle 1 without being easily blocked, and the distance between the camera and the two reference QR codes is also relatively appropriate. Based on the above conclusions, subsequent experiments are carried out based on viewing angle 1 to improve the accuracy and stability of macro positioning.

[0203] It should be noted that the three-dimensional positioning accuracy based on macro cameras is closely related to the image quality and the accuracy of the reference QR code. Higher positioning accuracy can be achieved by selecting higher-resolution cameras and printing higher-precision reference QR codes.

[0204] like Figure 7 As shown, Figure 7 (a) is the execution time required for each sub-step in the automatic initialization process. Figure 7 (b) is the execution time required for automatic and manual end-effector tip positioning. The automatic initialization scheme proposed in this paper can achieve a wide range of tool tip positioning and system calibration, thereby effectively improving the efficiency of micro-operation tasks and reducing the workload of operators. The time required for automatic initialization is an important indicator for measuring system performance. Therefore, this application has carried out execution time tests for its various sub-steps, including macro tool tip positioning, micro tool tip positioning, autofocus and autocalibration, and compared them with the time required for manual tool positioning by a skilled operator.

[0205] During the test phase, the tool tip was initially located at a random position above the center of the focal plane of the microscope. Specifically, the position difference between the tip and the center of the focal plane ranged from 5 to 20 mm in the along direction and from -10 to 10 mm in the along, and , directions. The system performed 30 automatic initializations and automatically recorded the time taken for each sub-step in the process.

[0206] The execution time of each sub-step obtained by the test is as follows Figure 7 As shown in (a), according to the execution order of the sub-steps, the average execution time of macro positioning, sweep-based positioning, autofocus, and automatic calibration are 22.89s, 16.60s, 10.59s, and 19.22s, respectively. The tool tip positioning step under the microscopic field of view is subdivided into two parts: sweep-based positioning and autofocus. The time statistics of automatic calibration are the calibration process of the left arm that controls the tool. The calibration process of the right arm is the same, and the execution time is almost the same. It can be observed that the execution time of macro positioning is relatively long because it involves a large range of tool movement, and the limited robot arm movement speed (maximum 2.5mm / s) makes it take a relatively long operation time. The time spent on sweep-based positioning has a relatively large fluctuation, with a standard deviation of 2.31. This is because the error in the macro tip positioning in the axial direction causes differences in the number of sweeps in the process of positioning the tool tip. According to statistics, the complete automatic initialization process including automatic tool tip positioning and left and right robot arm calibration processes takes an average of 88.52s.

[0207] Figure 7 (b) shows the comparison of the time required for automatic and manual tool tip positioning. The positioning of the manual tool tip was completed by a skilled operator, also based on 30 sets of data and statistical execution time. The test results show that the success rate of automatic and manual tool tip positioning is 100%, and the execution time of automatic tip positioning (average 50.08s) is shorter than that of manual tip positioning (average 65.70s).

[0208] In addition, the stability of automatic tip positioning time (standard deviation 2.63) is also better than manual tip positioning (standard deviation 7.90). This is mainly due to the fact that in the manual tip positioning task, the operator needs to repeatedly switch between the macroscopic field of view and the microscopic field of view of the microscope to locate the tool position, and the workload of this step is affected by the initial distance between the tool tip and the center of the microscope focal plane. A longer initial distance will significantly increase the operation time. In addition, in actual operation, the operator's proficiency will also significantly affect the time and success rate of manual tip positioning.

[0209] In summary, the automatic tool tip positioning scheme proposed in this paper can ensure the efficiency and stability of micro-manipulation tasks and reduce the operator's workload.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for automatically initializing and positioning the tip of an end effector of a micromanipulation system, characterized in that: The method is implemented by an automatic initialization positioning device for the tip of an end effector of a micromanipulation system, the device comprising a micromanipulator, a macro-micro combined imaging system, and a computer / user interface; The micromanipulator includes two groups of three-axis Cartesian manipulators, which are symmetrically distributed on the left and right sides of the microscope. The three-axis Cartesian manipulators are used to control the end effector and the culture dish respectively. The two groups of three-axis Cartesian manipulators are defined as the left arm and the right arm respectively. The macro-micro combined imaging system includes a microscopic image unit and a macroscopic image unit. The microscopic image unit is collected by a 20x objective lens digital camera, and the macroscopic image unit is collected by a macroscopic camera. The macroscopic and microscopic images will eventually be transmitted to a computer / user interface respectively. The method comprises the following steps: S1. Perform macro positioning of the end effector tip. The macro camera collects a macro image of the micro-operation space, moves the end effector to analyze the position of its tip in the image coordinate system, calculates the camera pose using the reference QR code ArUco Tag used for positioning, and realizes the three-dimensional positioning of the end effector tip through triangulation. The end effector is moved to an area close to the center of the focal plane based on visual servoing. S2, positioning the tip of the microscopic end effector in a 4×2×2 mm rectangular area, using the shadow of the end effector projected in the microscope image to locate the center axis and tip of the end effector, and moving the tip of the end effector to the focal plane; S3, based on visual servoing, collecting calibration data, wherein the calibration data is calibration data including microscopic image position information and micromanipulator encoder position information, calibrating the tip of the end effector and the culture dish, and obtaining a left calibration matrix and a right calibration matrix reflecting the conversion relationship between the left and right micromanipulator coordinates and the image coordinates; S4. Generate an intuitive human-computer interaction interface based on mouse and keyboard according to the position information collected by S1-S3. The microscopic information in the interface is used for local precision microscopic operation, and the macroscopic information is used to help the operator grasp the global environment.

2. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 1, characterized in that: The specific steps of S1 are as follows: S101, based on ArUco Tag, select the region of interest ROI in the macro camera image, and locate the two-dimensional position of the end effector tip in the image coordinate system through the movement of the micromanipulator; S102, estimating the three-dimensional position of the tip of the end effector by a triangulation method based on the image coordinates of the tip before and after the movement and the corresponding camera posture; S103, analyzing the position difference between the center of the microscope focal plane and the tip of the end effector, and moving the tip of the end effector to the target area through visual servoing.

3. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 2, characterized in that: The specific contents of selecting the region of interest ROI in the macro camera image and locating the two-dimensional position of the tip of the end effector in the image coordinate system in S101 are as follows: Automatically select the region of interest (ROI) containing the tip of the end effector based on the fiducial markers; Based on the pre-calibrated monocular camera, the intrinsic parameters of the macro camera are obtained. and distortion coefficient : ， ； There are two ArUco Tags, namely Tag1 and Tag2, where Tag2 is bound to the end effector of the left arm and moves with the micromanipulator, while Tag1 is bound to the microscope and remains stationary during the micromanipulation process; The coordinate system of Tag1 and Tag2 is obtained through the calibrated camera and the ArUco Tag of known size. With the macro camera coordinate system The conversion relationship between them; The expressions of the coordinate system of Tag1 and Tag2 and the macro camera coordinate system are: ； in, is the Tag1 or Tag2 coordinate system, and are the homogeneous coordinates of a point in the three-dimensional space in the macro camera and QR code coordinate systems, respectively. is the transformation matrix, which is Rotation Matrix and Translation Vector constitute; Based on Tag1 bound to the microscope, according to the travel of the micromanipulator, in the Tag1 coordinate system Next, select the rectangular area that contains the motion range of the end effector tip; Calculate the coordinates of the projection points of the cuboid endpoints in the Tag1 coordinate system in the macro camera image; The expression for calculating the coordinates of the projection points of the cuboid endpoints in the Tag1 coordinate system in the macro camera image is: ； in, represents the scaling factor, represents the pixel coordinates of the projection point, Represents the point in the tag1 coordinate system, Represents the projection matrix from Tag1 to the macro camera; The circumscribed rectangle of the convex hull of the projection point is taken as the ROI area; The end effector is controlled to move horizontally for a certain distance and the image matrix before and after the movement is collected. as well as And calculate the image difference to obtain the part with different grayscale values ​​before and after the movement; The expression for calculating image difference is: ； According to the influence of light on the end effector, define The positive part of and negative part They correspond to the end effector before and after the movement respectively; right Binarization is performed to obtain a silhouette image of the end effector and determine the maximum contour of the image; The point at the lower right of the maximum contour of the image is taken as the position of the tip of the end effector before the movement; exist The template is selected based on the tip position of the end effector, and the template matching method is used to Find the end effector tip position after the motion in .

4. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 2, characterized in that: The specific content of estimating the three-dimensional position of the tip of the end effector by the triangulation method in S102 is: Obtain the image coordinates of the end effector tip and calculate the three-dimensional position of the end effector tip in the Tag2 coordinate system by triangulation; Combining the camera intrinsic parameter K and the distortion parameter D, we can obtain the pixel coordinates of the end effector tip before and after the movement in the dedistorted image. ; The camera projection matrix corresponding to the pixel coordinates is obtained by keeping tag2 relatively still with the tip of the end effector ; Define the homogeneous three-dimensional coordinates of the tip in the tag2 coordinate system as ; The three-dimensional point With 2D image points The conversion relationship is: ,in are the pixel coordinates of the end effector tip before and after movement in the dedistorted image, represents the projection matrix from Tag2 to the macro camera, and Respectively represent the rotation and projection matrices from Tag2 to the macro camera, is the camera internal reference; according to and The vector parallel relationship is obtained and The relationship between and The relationship is: ； in, To constitute of row vector; Based on the tip coordinates after movement and the corresponding camera projection matrix get and The relationship between and The relationship is: ； Will and The relationship between and In the relationship between x and y, SVD is applied to the equation x to solve the target 3D point q in the case of noise; The expression for solving the equation x by applying SVD is: ； Analyze the expression and get The target three-dimensional point q, ; in, Representation and Matrix The smallest non-zero singular value of The associated eigenvectors, according to the definition of SVD decomposition, is a matrix The last column in ; Calculate the normalized three-dimensional coordinates of the tip , and through the coordinate system conversion expression, the coordinates of the end effector tip relative to tag1 after movement are obtained ; The coordinate system transformation expression is: ； Among them, the transformation matrix Describes the geometric transformation from the tag2 coordinate system to the macro camera coordinate system, the transformation matrix describes the geometric transformation from the macro camera coordinate system to the tag1 coordinate system, is the normalized three-dimensional coordinate of the tip, ; The coordinates obtained for the latest 5 frames of images The data is averaged to obtain a smoothed result .

5. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 2, characterized in that: In S103, the specific contents of analyzing the position difference between the center of the microscope focal plane and the tip of the end effector and moving the tip of the end effector to the target area through visual servoing are as follows: Manually move the tip of the end effector to the center of the microscope screen and complete the focus. The position of the end effector tip obtained at this time As ; Based on the pre-recorded focal plane center position , the system will select as the target location; in, , including horizontal offset, which facilitates the end-effector tip positioning process based on sweeping in the next stage; Target distance difference based on real-time feedback The scheme combined with the pseudocode to implement visual servoing will first control the tip of the manipulator to move above the target area, and then move it down to the target area, thereby avoiding contact with the outer wall of the culture dish, where .

6. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 1, characterized in that: The specific contents of micro end effector tip positioning in S2 are: S201, collecting microscope images and completing the initialization of the actuator scanning detector; S202, estimating the position of the tip of the end effector based on a scanning method, and moving it into the field of view of the microscope; S203, moving the tip of the end effector to the focal plane through two rounds of automatic focusing.

7. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 6, characterized in that: S201, collecting microscope images and completing the initialization of the actuator scanning detector, specifically includes: The actuator sweeps the detector to segment the microscope image into sub-areas, any of which can be used express, ; During the initialization process, the detector collects 60 frame difference images, calculates the standard deviation of the average gray value of each sub-region, and sets the threshold according to the standard deviation; The expression for calculating the standard deviation of the average gray value of each sub-region is: ； In the formula, represents the standard deviation of the corresponding area, Indicates the trigger coefficient preset according to the imaging conditions; During the detection process, if the average grayscale of the frame difference image of the region exceeds the corresponding threshold, the sub-region is marked as triggered.

8. The method for automatic initialization positioning of the tip of an end effector for a micromanipulation system according to claim 6, characterized in that: S202, the specific steps of estimating the position of the tip of the end effector based on the scanning method and moving it into the field of view of the microscope are: S2021, along Sweep the end effector in a certain direction to bring it into the picture; S2022, record the end effector position in the current micromanipulator coordinate system, and continue moving along the original sweeping direction Ensure that the shadow of the end effector leaves the field of view and moves along Direction movement distance Complete the end effector retraction; S2023, along The axis sweeps the end effector in the reverse direction. If the end effector is detected during the motion, S2022 is repeated. If the end effector is not detected, it is judged that the end effector has been retracted; S2024, based on the last two records The average value of the direction position, the system along Axis moves the middle axis of the end effector into the picture and controls the end effector to move along Direction of movement , move the end effector tip into the microscope field of view.

Images