A high-precision micromanipulation control method and system based on visual servoing

Abstract

The present invention discloses a high-precision micromanipulation control method and system based on visual servo, which firstly uses a high-frame rate industrial camera to asynchronously collect a grayscale image containing a puncture needle and an object to be operated; performs image enhancement processing on the grayscale image; performs threshold segmentation on the grayscale image to generate a threshold image, and performs a closed operation on the threshold image; then obtains the initial position of the needle tip of the puncture needle and the center position of the area to be operated, and plans the motion trajectory of the puncture needle; drives the puncture needle to move along the motion trajectory, obtains the needle tip position in real time, and when the connected domain analysis determines that the needle tip contacts the shell of the object to be operated, switches to an optimized positioning method, and accurately positions the needle tip position in real time; at the same time, a feedback control module calculates the trajectory error of the needle tip in real time, and adjusts the needle tip position until the puncture needle moves to the end position along the motion trajectory; the advantage is that the method has high positioning accuracy for the needle tip position, and accurately controls the puncture process, thereby ensuring the stability and accuracy of the operation.

Description

Technical Field

[0001] The present invention relates to the technical field of visual servo systems, and in particular to a high-precision micromanipulation control method and system based on visual servoing, which is suitable for automatic puncture operations in a microscopic image environment. Background Art

[0002] With the rapid development of science and technology, especially in the fields of medicine and biology, the demand for high-precision operation and control at the microscopic scale has become particularly important. Micromanipulation technology has therefore gradually become the focus of research and application. Micromanipulation is mainly used in experiments such as cell labeling, microinjection and cell manipulation, aiming to study the structure and function of microscopic objects. Since these operations involve high-precision control at a microscopic scale, traditional micromanipulation systems still have many limitations in positioning accuracy, motion control and real-time feedback. Existing micromanipulation systems often have errors when accurately locating the dynamic contact between the puncture needle and the target object, making the puncture process difficult to control. This deficiency is particularly evident during the injection operation, especially when the puncture needle contacts the biological sample, the contact process between the needle tip and the target area is complex and susceptible to jitter or external interference, which in turn affects the accuracy of the puncture operation. In addition, although the geometry of the needle tip is relatively simple, due to the slender characteristics of the puncture needle and the dynamic changes during operation, it is difficult for traditional image processing technology to accurately capture and analyze these details. This makes it difficult to improve the accuracy of puncture depth control and needle tip positioning. Therefore, technical improvements based on the characteristics of the slender structure of the needle tip will be the key to improving accuracy.

[0003] In this context, it is particularly important to optimize the design of the geometric shape of the puncture needle and the image features during its contact with the operating object. Traditional image processing methods are often plagued by image noise and continuity problems, resulting in low accuracy in image analysis and positioning. Therefore, the prior art urgently needs a positioning algorithm that can combine the geometric features of the needle tip with the image features of the contact process to enhance the ability of the micromanipulation system in needle tip positioning and puncture depth control. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a high-precision micromanipulation control method and system based on visual servoing, which combines a high-frame rate industrial camera with real-time image processing technology, utilizes the geometric features of the needle tip and the image features during the dynamic contact process, and achieves precise positioning of the puncture needle and precise control of the puncture process through a positioning method based on Blob analysis and optimization; and adjusts the motion trajectory of the puncture needle in real time to ensure high precision and stability in microinjection operations.

[0005] The technical solution adopted by the present invention to solve the above technical problems is: a high-precision micromanipulation control method based on visual servoing, comprising the following specific steps:

[0006] (1) Use a high frame rate industrial camera to asynchronously capture grayscale images of the puncture needle and the object to be operated in the microscopic field of view;

[0007] (2) Scale the grayscale image according to a predetermined ratio and perform image enhancement processing based on median filtering to improve the contrast and clarity of the image;

[0008] (3) Apply the Otsu algorithm to the first frame of grayscale image collected to determine the segmentation threshold T, and then use the fixed threshold T to perform binarization processing on the subsequent grayscale images collected to generate a threshold image containing the puncture needle and the object to be operated;

[0009] (4) Perform a closing operation on the threshold image to eliminate image noise caused by threshold segmentation and fill the holes in the operation area of ​​the puncture needle and the object to be operated to ensure that the image is continuous and complete;

[0010] (5) Using a positioning method based on Blob analysis, the initial position of the needle tip and the center position of the area to be operated are obtained from the first frame threshold image;

[0011] (6) Plan the motion trajectory of the puncture needle according to the initial position of the needle tip and the center position of the area to be operated;

[0012] (7) Drive the puncture needle to move along the planned motion trajectory, and obtain the needle tip position in real time through the positioning method based on Blob analysis. When the connected domain analysis determines that the needle tip contacts the shell of the object to be operated, switch to the optimized positioning method, that is, accurately locate the needle tip position in real time through row cumulative histogram and pixel scanning;

[0013] (8) The feedback control module generates an adjustment signal by calculating the trajectory error of the needle tip and adjusts the needle tip position in real time until the puncture needle moves to the target end position along the planned motion trajectory.

[0014] Furthermore, in the step (2), a bicubic interpolation smoothing interpolation method is used to scale the grayscale image and maintain the image aspect ratio, thereby reducing pixels while maintaining image quality.

[0015] Furthermore, in the step (2), the relationship of image enhancement processing is:

[0016] ,

[0017] in: g ( x,y ) is the scaled image at position ( x,y) at the gray value; median _ g ( x,y ) is the zoomed image after median filtering at position ( x,y ) at the gray value; α is the adjustment parameter, which mainly adjusts the scaling factor of the pixel grayscale difference before and after the median filter; round(·) is the rounding function; t ( x,y ) is the scaled image at position ( x,y ) is processed to obtain the pixel gray value.

[0018] Furthermore, in the step (5), the process of obtaining the initial position of the needle tip by the positioning method based on Blob analysis is as follows: a connectivity analysis is performed on the first frame threshold image after the closing operation to obtain three connected domains, namely the puncture needle, the operating object shell and the area to be operated. The geometric shape of the puncture needle connected domain is approximately a rectangle, and the needle tip position is located at the rightmost end of the rectangle. The timestamp of the first frame threshold image is set to 0, and the process of obtaining the initial position of the needle tip is as follows:

[0019] The needle tip is at the rightmost end of the puncture needle connected domain. First determine the column coordinate value of the needle tip:

[0020] ,

[0021] in: Indicates the needle tip column coordinate value located in the first frame threshold image with a timestamp of 0; P Needle (0) represents the connected domain of the puncture needle in the first frame threshold image with a timestamp of 0; i is the pixel number, whose value is the row coordinate value of the row multiplied by the number of columns, plus the column coordinate value of the column; ( r i ,c i ) is a pixel i The row and column coordinate values ​​of , both of which start from 0; max(·) is the maximum value function, which selects the maximum value from the given value set;

[0022] Then cut off a section of the area containing the needle tip from the connected domain of the puncture needle. The expression of this area is:

[0023] ,

[0024] in: d is the intercept width, P section (0) represents the area captured in the first frame of the threshold image with a timestamp of 0;

[0025] Then calculate the mean value of the row coordinates of all pixels in the above intercepted area as the row coordinate value of the needle tip:

[0026] ,

[0027] in: Indicates the row coordinate value of the needle tip located in the first frame threshold image with a timestamp of 0; symbol Indicates the number of pixels contained in the statistical area;

[0028] The process of obtaining the center position of the area to be operated by the positioning method based on Blob analysis is as follows: obtain the connected domain of the area to be operated through connected domain analysis, calculate the centroid position of the area to be operated to determine its center position. The relationship between the center position of the area to be operated is as follows:

[0029] ,

[0030] in: r target is the row coordinate value of the centroid of the area to be operated in the first frame threshold image; c target is the column coordinate value of the centroid of the area to be operated in the first frame threshold image; P target is the connected domain of the area to be operated.

[0031] Furthermore, in step (6), the planned motion trajectory of the puncture needle is divided into three stages: outward journey, dwelling segment and return journey. The complete motion trajectory of the puncture needle tip includes a row direction motion trajectory and a column direction motion trajectory, and the relationship between them is:

[0032] Movement trajectory in row direction:

[0033] ,

[0034] in: Indicates the timestamp is t The planned needle tip row coordinate value in the threshold image; for τ The instantaneous speed in the outbound direction at the time; for τ The instantaneous speed in the direction of the return trip; T r The time of movement in the outbound direction; T c The moving time in the outbound direction; T stay is the residence time; T c ＇ is the movement time in the return direction; Tr ＇ The time of movement in the return direction;

[0035] Column direction motion trajectory:

[0036] ,

[0037] in: Indicates the timestamp is t The planned needle tip column coordinate values ​​in the threshold image; for τ The instantaneous speed in the outbound direction at time; for τ The instantaneous velocity in the column direction of the return trip.

[0038] Furthermore, in the step (7), when the number of connected domains is determined to be 3 through the connected domain analysis, the connected domain includes the puncture needle, the shell of the object to be operated, and the area to be operated, that is, the needle tip does not contact the shell of the object to be operated, and the position of the needle tip is obtained in real time using a positioning method based on Blob analysis;

[0039] When the number of connected domains is determined to be 2 through connected domain analysis, the puncture needle and the shell of the object to be operated form a connected domain, that is, the puncture needle contacts the shell of the object to be operated but does not contact the area to be operated, and the needle tip position is obtained in real time using an optimized positioning method;

[0040] When the number of connected domains is determined to be 1 through connected domain analysis, the puncture needle, the shell of the object to be operated and the area to be operated constitute a connected domain, that is, the puncture needle contacts the area to be operated, and the row coordinate value of the needle tip is obtained using the optimized positioning method. The column coordinate value of the needle tip is consistent with the column coordinate value of the motion trajectory planned at that moment.

[0041] Furthermore, in step (7), the optimized positioning method is specifically:

[0042] Perform row cumulative pixel counting operation on the threshold image, that is, count the row coordinates of all pixels, record the number of times each row coordinate value appears in the image, and obtain the row cumulative histogram. The relationship between the row cumulative pixel count is:

[0043] ,

[0044] in: m is the index of the row; S ( t,m ) indicates that the timestamp is t In the threshold image m The cumulative sum of non-zero pixels in a row; I ( t,i ) indicates that the timestamp is t The pixel point in the threshold imagei The pixel value of is 0 or 1; w is the width of the threshold image, that is, the number of pixels per row;

[0045] In the row cumulative histogram, the first row coordinate value when the cumulative sum begins to increase rapidly and the last row coordinate value when the cumulative sum ends to decrease rapidly represent the minimum and maximum row coordinate values ​​of the puncture needle in the row direction, respectively; select all row coordinate values ​​corresponding to the high-order fluctuation segment in the row cumulative histogram, calculate the average value of these row coordinate values, and obtain the row coordinate value of the needle tip. The row coordinate relationship of the needle tip is:

[0046] ,

[0047] in: Indicates the timestamp is t The coordinate value of the needle tip located in the threshold image; max S ( t ) indicates that the timestamp is t The maximum value of the cumulative sum in all rows of the threshold image; O is the amplitude of the high-level fluctuation segment; h is the height of the threshold image, i.e., the number of pixels in each column; I(·) is the indicator function, which is 1 when the condition is true and 0 otherwise;

[0048] Take the row where the row coordinate of the needle tip is located as the scan row, scan pixel by pixel from left to right along the scan row in the threshold image, find the position where the first pixel value drops from 1 to 0, and determine the column coordinate value of this position as the column coordinate value of the needle tip. The relationship between the column coordinate values ​​of the needle tip is:

[0049] ,

[0050] in: Indicates the timestamp is t The coordinate value of the needle tip column located in the threshold image; c i+1 Pixel i +1; min(·) is the minimum function, which selects the minimum value from the given value set.

[0051] Furthermore, in the step (8), the calculation of the needle tip trajectory error includes the calculation of the row direction error and the column direction error, and the relationship between the row direction error and the column direction error is respectively:

[0052] ,

[0053] in: e r ( t ) indicates that the timestamp is tThe row direction error in the threshold image; e c ( t ) indicates that the timestamp is t The column-wise error in the thresholded image.

[0054] A high-precision micromanipulation control system based on visual servoing, comprising an image acquisition module, an image processing module, a threshold segmentation module, a closed operation processing module, a position information determination module, a motion trajectory planning module, a feedback control module and a micromanipulation execution module;

[0055] The image acquisition module uses a high frame rate industrial camera to asynchronously collect grayscale images in the microscopic field of view, which is used to capture the visual information of the puncture needle and the object to be operated in real time, and provide data for subsequent processing;

[0056] The image processing module is used to preprocess the grayscale image containing the puncture needle and the object to be operated, including image scaling and image enhancement, so as to improve the contrast and clarity of the image and ensure that the subsequent image analysis is more accurate;

[0057] The threshold segmentation module is used to determine the optimal segmentation threshold T for the first frame of grayscale image using the Otsu method, and directly apply the fixed threshold T to perform binarization processing on subsequent grayscale images to generate a threshold image containing the puncture needle and the object to be operated;

[0058] The closing operation processing module eliminates noise points and holes in the threshold image through closing operation processing, ensuring that the outlines of the puncture needle and the area to be operated are continuous and complete in the image, providing a cleaner image for subsequent positioning and analysis;

[0059] The position information determination module obtains the initial position of the needle tip of the puncture needle and the center position of the area to be operated by a positioning method based on Blob analysis for the first frame threshold image; and determines the positioning method of the needle tip in subsequent frame threshold images by connected domain analysis, and locates the needle tip position in real time;

[0060] The motion trajectory planning module plans the motion trajectory of the puncture needle according to the initial position of the needle tip and the center position of the area to be operated, so as to achieve accurate puncture operation;

[0061] The feedback control module is used to calculate the trajectory error of the puncture needle and generate a motion trajectory adjustment signal of the puncture needle and send it to the micromanipulation execution module;

[0062] The micromanipulation execution module is used to drive the puncture needle to move along the planned motion trajectory, and execute specific micromanipulation according to the adjustment signal of the feedback control module.

[0063] Compared with the prior art, the advantages of the present invention are:

[0064] (1) High positioning accuracy: The present invention combines the geometric shape of the needle tip and the image features during the contact process of the object to be operated, and adopts an optimized positioning method based on Blob analysis and row cumulative histogram. It can accurately obtain the position information of the needle tip in real time during the micromanipulation process. In particular, when the needle tip contacts the object to be operated, the positioning accuracy is significantly improved, reducing operational errors caused by needle tip position errors.

[0065] (2) Precise control of the puncture process: The present invention automatically adjusts the motion trajectory of the needle tip through real-time image analysis and feedback control. Compared with the traditional micromanipulation system, it avoids the problem of radial jitter of the puncture needle during the puncture process caused by human intervention and visual errors, ensuring that the puncture needle can enter the object to be operated at the optimal position, reducing mechanical damage to the cells or embryos to be operated.

[0066] (3) Real-time feedback and dynamic adjustment: The system captures real-time images through a high-frame-rate industrial camera and dynamically adjusts the needle tip position and trajectory in combination with a feedback control module. In particular, after the needle tip contacts the object, it can quickly switch to the optimized positioning method according to the changes in image features, and correct the needle tip's motion trajectory in real time to avoid needle tip deviation caused by external forces or system errors, thus ensuring the stability and accuracy of the operation.

[0067] (4) Efficient image processing: The present invention effectively reduces the number of pixels in the grayscale image through image scaling technology, thereby reducing the complexity of subsequent operations and improving processing efficiency. In addition, the Otsu algorithm is only executed once on the first frame of the grayscale image to obtain the segmentation threshold T, and the threshold is applied to the binarization processing of subsequent images, avoiding the tedious step of recalculating the threshold for each frame of the grayscale image, thereby greatly improving the operating efficiency and real-time performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of the micromanipulation control method of the present invention;

[0069] Figure 2 This is a rendering of the grayscale image after image enhancement processing according to an embodiment of the present invention;

[0070] In the figure: (a) is a grayscale image after image scaling; (b) is a grayscale image after image enhancement based on median filtering;

[0071] Figure 3 The threshold image is obtained by binarizing the grayscale image of the embodiment of the present invention using a fixed threshold T;

[0072] Figure 4is a threshold image after the closing operation in an embodiment of the present invention;

[0073] Figure 5 This is a positioning effect diagram of a positioning method based on Blob analysis in an embodiment of the present invention;

[0074] In the figure: (a) is a diagram showing the initial position of the needle tip and the center position of the yolk in the threshold image of the first frame; (b) is a diagram showing the position of the needle tip obtained in the threshold image of the current frame when the puncture needle is moving and the puncture needle has not yet touched the shell of the zebrafish embryo;

[0075] Figure 6 The motion trajectory of the puncture needle planned in the embodiment of the present invention;

[0076] In the figure: (a) is the motion trajectory of the puncture needle on its outward journey; (b) is the motion trajectory of the puncture needle on its return journey;

[0077] Figure 7 This is a positioning effect diagram of an optimized positioning method used in an embodiment of the present invention;

[0078] In the figure: (a) is a diagram showing the use of an optimized positioning method to position the needle tip when the puncture needle contacts the shell of the zebrafish embryo but does not contact the yolk; (b) is a diagram showing the use of an optimized positioning method to obtain the row coordinate value of the needle tip when the puncture needle contacts the yolk, and the column coordinate value of the needle tip is obtained based on the planned motion trajectory; (c) is the row cumulative histogram of (a); (d) is the row cumulative histogram of (b). DETAILED DESCRIPTION

[0079] The present invention is further described in detail below with reference to the accompanying drawings.

[0080] Embodiment 1: In this embodiment, a zebrafish embryo is used as the object to be operated, and the yolk in the embryo is used as the area to be operated. The zebrafish embryo is embedded in a semicircular groove of a transparent culture medium and fixed in a culture dish. The yolk of the zebrafish embryo is punctured. The high-precision micromanipulation control method based on visual servoing is as follows: Figure 1 As shown, the specific steps include:

[0081] (1) Use a high frame rate industrial camera to asynchronously capture grayscale images of the puncture needle and zebrafish embryo in the microscopic field of view;

[0082] (2) Use the bicubic interpolation smoothing method to scale the grayscale image to a predetermined ratio and maintain the image aspect ratio, reducing pixels while maintaining image quality; and perform image enhancement processing based on median filtering to improve the image contrast and clarity, such as Figure 2 As shown; the relationship for image enhancement processing is:

[0083] ,

[0084] in: g ( x,y ) is the scaled image at position ( x,y ) at the gray value; median _ g ( x,y ) is the zoomed image after median filtering at position ( x,y ) at the gray value; α is the adjustment parameter, which mainly adjusts the scaling factor of the pixel grayscale difference before and after the median filter; round(·) is the rounding function; t ( x,y ) is the scaled image at position ( x,y ) is processed to obtain the pixel gray value;

[0085] (3) Apply the Otsu algorithm to the first frame of grayscale image collected to determine the segmentation threshold T, and then use the fixed threshold T to perform binarization processing on the subsequent grayscale images collected to generate a threshold image containing the puncture needle and zebrafish embryo, as shown in Figure 3 As shown;

[0086] (4) Perform a closing operation on the threshold image to eliminate image noise caused by threshold segmentation and fill the holes in the puncture needle and the yolk region in the zebrafish embryo to ensure that the image is continuous and complete. Figure 4 As shown;

[0087] (5) Using the positioning method based on Blob analysis, the initial position of the needle tip and the center position of the yolk area are obtained from the first frame threshold image, as shown in Figure 5 As shown in (a), in this threshold image, the puncture needle has not yet touched the zebrafish embryo, and the needle tip is at the initial position, specifically:

[0088] (5-1) The process of obtaining the initial position of the needle tip using the positioning method based on Blob analysis is as follows: Connectivity analysis is performed on the first frame threshold image after the closing operation to obtain three connected domains, namely the puncture needle, the shell of the zebrafish embryo, and the yolk area. The geometric shape of the puncture needle connected domain is approximately rectangular, and the needle tip is located at the rightmost end of the rectangle. The timestamp of the first frame threshold image is set to 0, and the process of obtaining the initial position of the needle tip is as follows:

[0089] The needle tip is at the rightmost end of the connected domain of the puncture needle. First determine the column coordinate value of the needle tip:

[0090] ,

[0091] in: Indicates the needle tip column coordinate value located in the first frame threshold image with a timestamp of 0; PNeedle (0) represents the connected domain of the puncture needle in the first frame threshold image with a timestamp of 0; i is the pixel number, whose value is the row coordinate value of the row multiplied by the number of columns, plus the column coordinate value of the column; ( r i ,c i ) is a pixel i The row and column coordinate values ​​of , both of which start from 0; max(·) is the maximum value function, which selects the maximum value from the given value set;

[0092] Then cut off a section of the area containing the needle tip from the connected domain of the puncture needle. The expression of this area is:

[0093] ,

[0094] in: d is the intercept width, P section (0) represents the area captured in the first frame of the threshold image with a timestamp of 0;

[0095] Then calculate the mean value of the row coordinates of all pixels in the above intercepted area as the row coordinate value of the needle tip:

[0096] ,

[0097] in: Indicates the row coordinate value of the needle tip located in the first frame threshold image with a timestamp of 0; symbol Indicates the number of pixels contained in the statistical area;

[0098] (5-2) The process of obtaining the center position of the yolk region by the positioning method based on Blob analysis is as follows: obtain the connected domain of the yolk region through connected domain analysis, calculate the centroid position of the yolk region to determine its center position. The relationship between the center position of the yolk region is as follows:

[0099] ,

[0100] in: r target is the row coordinate value of the centroid of the yolk area in the first frame threshold image; c target is the column coordinate value of the centroid of the yolk area in the first frame threshold image; P target is the connected domain of the yolk region;

[0101] (6) The motion trajectory of the puncture needle is planned according to the initial position of the needle tip and the center position of the yolk area. It is specifically divided into three stages: the outward journey (i.e., the stage where the needle tip moves from the initial position to the center position of the yolk area), the dwelling segment (i.e., the stage where the needle tip stays at the center position of the yolk area for injection), and the return journey (i.e., the stage where the needle tip returns from the center position of the yolk area to the initial position). Figure 6 As shown, the complete motion trajectory of the puncture needle tip includes the row direction motion trajectory and the column direction motion trajectory, and the relationship between them is:

[0102] Movement trajectory in row direction:

[0103] ,

[0104] in: Indicates the timestamp is t The planned needle tip row coordinate value in the threshold image; for τ The instantaneous speed in the outbound direction at the time; for τ The instantaneous speed in the direction of the return trip; T r The time of movement in the outbound direction; T c The moving time in the outbound direction; T stay is the residence time; T c ＇ is the movement time in the return direction; T r ＇ The time of movement in the return direction;

[0105] Column direction motion trajectory:

[0106] ,

[0107] in: Indicates the timestamp is t The planned needle tip column coordinate values ​​in the threshold image; for τ The instantaneous speed in the outbound direction at time; for τ The instantaneous speed in the column direction of the return trip;

[0108] (7) Drive the puncture needle to move along the planned motion trajectory. When the number of connected domains is determined to be 3 through connected domain analysis, the connected domain includes the puncture needle, the shell of the zebrafish embryo and the yolk area, that is, the needle tip does not touch the shell of the zebrafish embryo. Use the positioning method based on Blob analysis to obtain the position of the needle tip in real time, such as Figure 5 (b) as shown;

[0109] When the number of connected domains is determined to be 2 through connected domain analysis, the puncture needle and the shell of the zebrafish embryo constitute a connected domain, that is, the puncture needle touches the shell of the zebrafish embryo but does not touch the yolk area, and the needle tip position is obtained in real time using an optimized positioning method (i.e., through row cumulative histogram and pixel scanning), as shown in Figure 7 As shown in (a) and (c);

[0110] When the number of connected domains is determined to be 1 through connected domain analysis, the puncture needle, the shell of the zebrafish embryo and the yolk area constitute a connected domain, that is, the puncture needle contacts the yolk area, and the row coordinate value of the needle tip is obtained using the optimized positioning method. The column coordinate value of the needle tip is consistent with the column coordinate value of the motion trajectory planned at that moment, such as Figure 7 (b) and (d); and the optimized positioning method is specifically:

[0111] Perform row cumulative pixel counting operation on the threshold image, that is, count the row coordinates of all pixels, record the number of times each row coordinate value appears in the image, and obtain the row cumulative histogram. The relationship between the row cumulative pixel count is:

[0112] ,

[0113] in: m is the index of the row; S ( t,m ) indicates that the timestamp is t In the threshold image m The cumulative sum of non-zero pixels in a row; I ( t,i ) indicates that the timestamp is t The pixel point in the threshold image i The pixel value of is 0 or 1; w is the width of the threshold image, that is, the number of pixels per row;

[0114] In the row cumulative histogram, the first row coordinate value when the cumulative sum begins to increase rapidly and the last row coordinate value when the cumulative sum ends to decrease rapidly represent the minimum and maximum row coordinate values ​​occupied by the puncture needle in the row direction, respectively; select all row coordinate values ​​corresponding to the high-order fluctuation segment in the row cumulative histogram, calculate the average value of these row coordinate values, and obtain the row coordinate value of the needle tip. The row coordinate relationship of the needle tip is:

[0115] ,

[0116] in: Indicates the timestamp is t The coordinate value of the needle tip located in the threshold image; max S ( t ) indicates that the timestamp ist The maximum value of the cumulative sum in all rows of the threshold image; O is the amplitude of the high-level fluctuation segment; h is the height of the threshold image, i.e., the number of pixels in each column; I(·) is the indicator function, which is 1 when the condition is true and 0 otherwise;

[0117] Take the row where the row coordinate of the needle tip is located as the scan row, scan pixel by pixel from left to right along the scan row in the threshold image, find the position where the first pixel value drops from 1 to 0, and determine the column coordinate value of this position as the column coordinate value of the needle tip. The relationship between the column coordinate values ​​of the needle tip is:

[0118] ,

[0119] in: Indicates the timestamp is t The coordinate value of the needle tip column located in the threshold image; c i+1 Pixel i +1 column coordinate value; min(·) is the minimum function, which selects the minimum value from the given value set;

[0120] (8) The feedback control module generates an adjustment signal by calculating the trajectory error of the needle tip and adjusts the needle tip position in real time until the puncture needle moves to the target end position along the planned motion trajectory; wherein the calculation of the needle tip trajectory error includes the calculation of the row direction error and the column direction error, and the relationship between the row direction error and the column direction error is:

[0121] ,

[0122] in: e r ( t ) indicates that the timestamp is t The row direction error in the threshold image; e c ( t ) indicates that the timestamp is t The column-wise error in the threshold image of ;

[0123] Based on the trajectory error calculated above, the feedback control module generates an adjustment signal, and the micromanipulation execution module dynamically adjusts the motion trajectory of the puncture needle. The feedback control module can adopt a variety of existing control strategies, such as proportional control, PID control, fuzzy control, etc., to achieve high-precision needle tip positioning and operation.

[0124] Embodiment 2: A system for implementing the above-mentioned high-precision micromanipulation control method based on visual servoing, comprising an image acquisition module, an image processing module, a threshold segmentation module, a closed operation processing module, a position information determination module, a motion trajectory planning module, a feedback control module and a micromanipulation execution module;

[0125] The image acquisition module uses a high frame rate industrial camera to asynchronously collect grayscale images in the microscopic field of view, which is used to capture the visual information of the puncture needle and the object to be operated in real time, providing data for subsequent processing;

[0126] The image processing module is used to pre-process the grayscale image containing the puncture needle and the object to be operated, including image scaling and image enhancement, to improve the contrast and clarity of the image and ensure that the subsequent image analysis is more accurate;

[0127] The threshold segmentation module is used to determine the optimal segmentation threshold T for the first frame of grayscale image using the Otsu method, and directly apply the fixed threshold T to perform binarization processing on subsequent grayscale images to generate a threshold image containing the puncture needle and the object to be operated; this can improve the calculation efficiency and ensure the consistency of image segmentation;

[0128] The closed operation processing module eliminates noise and holes in the threshold image through closed operation processing, ensuring that the outlines of the puncture needle and the area to be operated are continuous and complete in the image, providing a cleaner image for subsequent positioning and analysis;

[0129] The position information determination module obtains the initial position of the needle tip of the puncture needle and the center position of the area to be operated for the first frame threshold image through a positioning method based on Blob analysis; and determines the positioning method of the needle tip in the subsequent frame threshold image through connected domain analysis. When the number of connected domains is 3, the positioning method based on Blob analysis is used to locate the needle tip in real time; when the number of connected domains is 2 (that is, the needle tip contacts the shell of the zebrafish embryo but does not contact the yolk area), the optimized positioning method is used to locate the needle tip in real time; when the number of connected domains is 1 (that is, the needle tip contacts the yolk area), the row coordinate value of the needle tip is obtained through the optimized positioning method, and the column coordinate value is consistent with the column coordinate value of the planned motion trajectory;

[0130] The motion trajectory planning module plans the motion trajectory of the puncture needle according to the initial position of the needle tip and the center position of the area to be operated to achieve accurate puncture operation;

[0131] The feedback control module is used to calculate the trajectory error of the puncture needle and generate a motion trajectory adjustment signal of the puncture needle, which is sent to the micromanipulation execution module;

[0132] The micromanipulation execution module is used to drive the puncture needle to move along the planned motion trajectory and perform specific micromanipulation according to the adjustment signal of the feedback control module.

[0133] The protection scope of the present invention includes but is not limited to the above embodiments, and its protection scope is subject to the claims. Any replacement, deformation, and improvement of the technology that can be easily thought of by technicians in this field shall fall within the protection scope of the present invention.

Claims

1. A high-precision micromanipulation control method based on visual servoing, characterized in that The specific steps include: (1) Use a high frame rate industrial camera to asynchronously capture grayscale images of the puncture needle and the object to be operated in the microscopic field of view; (2) Scale the grayscale image according to a predetermined ratio and perform image enhancement processing based on median filtering to improve the contrast and clarity of the image; (3) Apply the Otsu algorithm to the first frame of grayscale image collected to determine the segmentation threshold T, and then use the fixed threshold T to perform binarization processing on the subsequent grayscale images collected to generate a threshold image containing the puncture needle and the object to be operated; (4) Perform a closing operation on the threshold image to eliminate image noise caused by threshold segmentation and fill the holes in the operation area of ​​the puncture needle and the object to be operated to ensure that the image is continuous and complete; (5) Using a positioning method based on Blob analysis, the initial position of the needle tip and the center position of the area to be operated are obtained from the first frame threshold image; (6) Plan the motion trajectory of the puncture needle according to the initial position of the needle tip and the center position of the area to be operated; (7) Drive the puncture needle to move along the planned motion trajectory, and obtain the needle tip position in real time through the positioning method based on Blob analysis. When the connected domain analysis determines that the needle tip contacts the shell of the object to be operated, switch to the optimized positioning method, that is, accurately locate the needle tip position in real time through row cumulative histogram and pixel scanning; specifically: When the number of connected domains is determined to be 3 through connected domain analysis, the connected domain includes the puncture needle, the shell of the object to be operated, and the area to be operated, that is, the needle tip does not touch the shell of the object to be operated, and the position of the needle tip is obtained in real time using a positioning method based on Blob analysis; When the number of connected domains is determined to be 2 through connected domain analysis, the puncture needle and the shell of the object to be operated form a connected domain, that is, the puncture needle contacts the shell of the object to be operated but does not contact the area to be operated, and the needle tip position is obtained in real time using an optimized positioning method; When the number of connected domains is determined to be 1 through connected domain analysis, the puncture needle, the shell of the object to be operated and the area to be operated constitute a connected domain, that is, the puncture needle contacts the area to be operated, and the row coordinate value of the needle tip is obtained using the optimized positioning method, and the column coordinate value of the needle tip is consistent with the column coordinate value of the motion trajectory planned at that moment; (8) The feedback control module generates an adjustment signal by calculating the trajectory error of the needle tip and adjusts the needle tip position in real time until the puncture needle moves to the target end position along the planned motion trajectory.

2. A high-precision micromanipulation control method based on visual servoing as claimed in claim 1, characterized in that: In the step (2), the grayscale image is scaled and the image aspect ratio is maintained by using a bicubic interpolation smoothing interpolation method, thereby reducing pixels while maintaining image quality.

3. A high-precision micromanipulation control method based on visual servoing as claimed in claim 1, characterized in that: In the step (2), the relationship of image enhancement processing is: ， in: g ( x,y ) is the scaled image at position ( x,y ) at the gray value; median _ g ( x,y ) is the zoomed image after median filtering at position ( x,y ) at the gray value; α is the adjustment parameter, which mainly adjusts the scaling factor of the pixel grayscale difference before and after the median filter; round(·) is the rounding function; t ( x,y ) is the scaled image at position ( x,y ) is processed to obtain the pixel gray value.

4. The high-precision micromanipulation control method based on visual servoing as claimed in claim 1, characterized in that: In the step (5), the process of obtaining the initial position of the needle tip by the positioning method based on Blob analysis is as follows: a connectivity analysis is performed on the first frame threshold image after the closing operation to obtain three connected domains, namely the puncture needle, the operating object shell and the area to be operated. The geometric shape of the puncture needle connected domain is approximately rectangular, and the needle tip position is located at the rightmost end of the rectangle. The timestamp of the first frame threshold image is set to 0, and the process of obtaining the initial position of the needle tip is as follows: The needle tip is at the rightmost end of the puncture needle connected domain. First determine the column coordinate value of the needle tip: ， in: Indicates the needle tip column coordinate value located in the first frame threshold image with a timestamp of 0; P Needle (0) represents the connected domain of the puncture needle in the first frame threshold image with a timestamp of 0; i is the pixel number, whose value is the row coordinate value of the row multiplied by the number of columns, plus the column coordinate value of the column; ( r i ,c i ) is a pixel i The row and column coordinate values ​​of , both of which start from 0; max(·) is the maximum value function, which selects the maximum value from the given value set; Then cut off a section of the area containing the needle tip from the connected domain of the puncture needle. The expression of this area is: ， in: d is the intercept width, P section (0) represents the area captured in the first frame of the threshold image with a timestamp of 0; Then calculate the mean value of the row coordinates of all pixels in the above intercepted area as the row coordinate value of the needle tip: ， in: Indicates the row coordinate value of the needle tip located in the first frame threshold image with a timestamp of 0; symbol Indicates the number of pixels contained in the statistical area; The process of obtaining the center position of the area to be operated by the positioning method based on Blob analysis is as follows: obtain the connected domain of the area to be operated through connected domain analysis, calculate the centroid position of the area to be operated to determine its center position. The relationship between the center position of the area to be operated is as follows: ， in: r target is the row coordinate value of the centroid of the area to be operated in the first frame threshold image; c target is the column coordinate value of the centroid of the area to be operated in the first frame threshold image; P target is the connected domain of the area to be operated.

5. The high-precision micromanipulation control method based on visual servoing as claimed in claim 1, characterized in that: In the step (6), the planned motion trajectory of the puncture needle is divided into three stages: outward journey, dwelling segment and return journey. The complete motion trajectory of the puncture needle tip includes the row direction motion trajectory and the column direction motion trajectory, and the relationship between them is: Movement trajectory in row direction: ， in: Indicates the timestamp is t The planned needle tip row coordinate value in the threshold image; for τ The instantaneous speed in the outbound direction at the time; for τ The instantaneous speed in the direction of the return trip; T r The time of movement in the outbound direction; T c The moving time in the outbound direction; T stay is the residence time; T c ＇ is the movement time in the return direction; T r ＇ The time of movement in the return direction; Column direction motion trajectory: ， in: Indicates the timestamp is t The coordinate values ​​of the planned needle tip column in the threshold image; for τ The instantaneous speed in the outbound direction at time; for τ The instantaneous velocity in the column direction of the return trip.

6. A high-precision micromanipulation control method based on visual servoing as claimed in claim 1, characterized in that: In the step (7), the optimized positioning method is specifically: Perform row cumulative pixel counting operation on the threshold image, that is, count the row coordinates of all pixels, record the number of times each row coordinate value appears in the image, and obtain the row cumulative histogram. The relationship between the row cumulative pixel count is: ， in: m is the index of the row; S ( t,m ) indicates that the timestamp is t In the threshold image m The cumulative sum of non-zero pixels in a row; I ( t,i ) indicates that the timestamp is t The pixel point in the threshold image i The pixel value of is 0 or 1; w is the width of the threshold image, that is, the number of pixels per row; In the row cumulative histogram, the first row coordinate value when the cumulative sum begins to increase rapidly and the last row coordinate value when the cumulative sum ends to decrease rapidly represent the minimum and maximum row coordinate values ​​of the puncture needle in the row direction, respectively; select all row coordinate values ​​corresponding to the high-order fluctuation segment in the row cumulative histogram, calculate the average value of these row coordinate values, and obtain the row coordinate value of the needle tip. The row coordinate relationship of the needle tip is: ， in: Indicates the timestamp is t The coordinate value of the needle tip located in the threshold image; max S ( t ) indicates that the timestamp is t The maximum value of the cumulative sum in all rows of the threshold image; O is the amplitude of the high-level fluctuation segment; h is the height of the threshold image, i.e., the number of pixels in each column; I(·) is the indicator function, which is 1 when the condition is true and 0 otherwise; Take the row where the row coordinate of the needle tip is located as the scan row, scan pixel by pixel from left to right along the scan row in the threshold image, find the position where the first pixel value drops from 1 to 0, and determine the column coordinate value of this position as the column coordinate value of the needle tip. The relationship between the column coordinate values ​​of the needle tip is: ， in: Indicates the timestamp is t The coordinate value of the needle tip column located in the threshold image; c i+1 Pixel i +1; min(·) is the minimum function, which selects the minimum value from the given value set.

7. The high-precision micromanipulation control method based on visual servoing as claimed in claim 1, characterized in that: In the step (8), the calculation of the needle tip trajectory error includes the calculation of the row direction error and the column direction error. The relationship between the row direction error and the column direction error is: ， in: e r ( t ) indicates that the timestamp is t The row direction error in the threshold image; e c ( t ) indicates that the timestamp is t The column-wise error in the thresholded image.

8. A high-precision micromanipulation control system based on visual servoing, characterized in that: It includes an image acquisition module, an image processing module, a threshold segmentation module, a closed operation processing module, a position information determination module, a motion trajectory planning module, a feedback control module and a micro-operation execution module; The image acquisition module uses a high frame rate industrial camera to asynchronously collect grayscale images in the microscopic field of view, which is used to capture the visual information of the puncture needle and the object to be operated in real time, and provide data for subsequent processing; The image processing module is used to preprocess the grayscale image containing the puncture needle and the object to be operated, including image scaling and image enhancement, so as to improve the contrast and clarity of the image and ensure that the subsequent image analysis is more accurate; The threshold segmentation module is used to determine the optimal segmentation threshold T for the first frame of grayscale image using the Otsu method, and directly apply the fixed threshold T to perform binarization processing on subsequent grayscale images to generate a threshold image containing the puncture needle and the object to be operated; The closing operation processing module eliminates noise points and holes in the threshold image through closing operation processing, ensuring that the outlines of the puncture needle and the area to be operated are continuous and complete in the image, providing a cleaner image for subsequent positioning and analysis; The position information determination module obtains the initial position of the needle tip of the puncture needle and the center position of the area to be operated by using a positioning method based on Blob analysis for the first frame threshold image; The method of locating the needle tip in the subsequent frame threshold image is determined by connected domain analysis, and the needle tip position is located in real time; The motion trajectory planning module plans the motion trajectory of the puncture needle according to the initial position of the needle tip and the center position of the area to be operated, so as to achieve accurate puncture operation; The feedback control module is used to calculate the trajectory error of the puncture needle and generate a motion trajectory adjustment signal of the puncture needle and send it to the micromanipulation execution module; The micromanipulation execution module is used to drive the puncture needle to move along the planned motion trajectory, and execute specific micromanipulation according to the adjustment signal of the feedback control module.

Images