Micro-motion positioning and control system based on microscopic visual servo

Abstract

The invention discloses a micro-motion positioning and control system based on microscopic visual servo, and belongs to the technical field of micro-operation positioning control. The problems that a traditional micro-operation system is large in image interference, inaccurate in recognition and positioning, poor in control stability and low in automation degree are solved. The method specifically comprises the following steps: firstly, completing image preprocessing through weighted average method graying, Fourier transform band-pass filtering, improved morphological top-hat algorithm reflection removal and optimized threshold segmentation; the micro copper wire and the transfer pipette are accurately positioned by combining a main view angle and an overlook view angle and adopting gray template matching, Canny edge detection, Tomasi corner detection and a BRISK feature point matching algorithm, and the problems of shielding and rotation are solved; finally, fuzzy PID, a backstepping method and other algorithms are fused, and micro-operation is achieved through cooperative control of the stepping motor and the piezoelectric ceramic double micro-motion platform. The system can efficiently and stably complete automatic picking of the micro copper wire, remarkably improves the micro-operation precision and the automation level, and is suitable for the related field of micro-nano operation.

Description

Technical Field

[0001] This invention belongs to the field of micro-operation positioning and control technology under microscopic vision, specifically relating to a micro-motion positioning and control system based on microscopic vision servo. Background Technology

[0002] As micro-operation tasks become increasingly complex and large-scale, the requirements for system precision and automation level are becoming more and more stringent. Relying solely on manual operation is no longer sufficient to meet the demand for high-precision and high-efficiency operations.

[0003] With the rapid development of image processing-based target detection and localization technology and microscopic visual servoing theory, the automation level of micro-operating systems has been greatly improved.

[0004] In the pick-and-release process of micromanipulation, the identification and positioning of the pipette tip against the manipulated object, such as a micro-copper wire, is a prerequisite for achieving contact pick-and-release. However, problems such as shadows caused by uneven illumination, image noise, target occlusion, defocusing, and rotation can significantly affect the target threshold segmentation under microscopic vision, further reducing the accuracy of target identification and positioning in micromanipulation.

[0005] Currently, existing methods for solving these problems mainly fall into two categories. The first is to employ traditional image processing techniques, which involve designing specific algorithms to enhance the contrast between the foreground and background, and using filtering techniques to remove shadows caused by uneven lighting, thereby correcting or mitigating the effects of uneven lighting. For example, traditional methods often use grayscale processing, Fourier transform filtering, or morphological operations to optimize image quality. These methods are widely applicable and relatively simple to compute, but they often require long processing times and lack robustness in complex environments.

[0006] The second approach utilizes deep learning technology, combined with current powerful hardware resources and large-scale datasets, to learn the illumination distribution patterns in images in order to identify and correct shadow areas. Deep learning methods are highly targeted, relatively simple to implement, and highly stable. Their recognition accuracy is highly dependent on the size and quality of the dataset. Compared with traditional methods, they may be superior in accuracy, but they require a large amount of labeled data and the training process is time-consuming and labor-intensive.

[0007] Although scholars at home and abroad have conducted a lot of research on the segmentation, recognition and control of micro-manipulated targets under microscopic vision, such as threshold segmentation based on Otsu algorithm, template matching localization and servo system with PID control, existing microscopic vision micro-manipulation algorithms still have problems such as low image processing accuracy, large positioning error and poor control stability when affected by factors such as shadows caused by uneven lighting, image noise and dynamic changes of the target.

[0008] Specifically, in micromanipulation environments, such as tasks involving picking up micro-copper wires and pipettes, the traditional Otsu algorithm is easily affected by the ratio of foreground to background pixels, leading to unstable thresholds; template matching struggles to handle occlusion and rotation issues from both front-view and top-view perspectives; and motion control of stepper motors and piezoelectric ceramic platforms often lacks effective disturbance compensation, making nanoscale precise positioning difficult. These problems not only affect the overall performance of the system but also limit the application and promotion of micromanipulation in fields such as biomedicine and microelectronic assembly.

[0009] Therefore, researching a high-precision and robust microscopic vision servo system is of great significance for realizing the automation of micro-operations. Summary of the Invention

[0010] The purpose of this invention is to solve the problems of low recognition and positioning accuracy and poor motion control stability of existing micro-operation positioning and control systems under the influence of factors such as image noise, uneven lighting, target occlusion, defocus, and rotation. Therefore, a micro-motion positioning and control method based on microscopic vision servoing is proposed.

[0011] The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a micro-motion positioning and control method based on microscopic vision servoing, characterized in that the method specifically includes the following steps:

[0012] Step 1: Acquire microscopic images of the micro-copper wires and pipettes using a front-view camera and a top-view camera. Preprocess the acquired color images, first using a weighted average method for grayscale conversion. The formula is as follows:

[0013] Gray = 0.299·f R (i,j)+0.578·f G (i,j)+0.114·f B (i,j)

[0014] Then, a bandpass filter is constructed using Fourier transform to remove mid-to-high frequency noise; next, reflection interference is removed based on improved morphological top-hat operations; finally, optimized Otsu thresholding is used for segmentation and inter-class variance analysis.

[0015] σ=ω0ω1(μ0-μ1) 2

[0016] Extract the region of interest to obtain the preprocessed image;

[0017] Step 2: Perform target recognition and localization on the preprocessed image, and calculate the matching coefficient using grayscale template matching from the main viewpoint.

[0018]

[0019] Identify the pipette and micro-copper wire regions; then use the Canny operator to extract edges; use Tomasi corner detection on the pipette and locate the midpoint of the nozzle; construct the minimum bounding rectangle for the micro-copper wire; calculate the location point of the pipette using grayscale template matching from a top-down view; use the BRISK algorithm on the micro-copper wire to calculate the Hamming distance to filter out mismatches, solve the problems of occlusion, defocus, and rotation, and obtain the location coordinates.

[0020] Step 3: Motion and servo control are performed based on the positioning coordinates. A stepper motor platform is used for coarse positioning with an accuracy of 1.25 μm / pulse; switching to a piezoelectric ceramic platform achieves fine positioning with a range of 20 μm. Interlocking with an MFC host computer supports open-loop and closed-loop control. Fuzzy PID control is used to adjust parameters for servo control. A controller is designed using a backstepping method. ESO is introduced to estimate disturbances. DOB is used to compensate for noise. The software, based on OpenCV and MFC, implements image processing and the interface, and calculates errors during the process.

[0021]

[0022] Achieve precise pickup and release of micro-copper wires.

[0023] The beneficial effects of this invention are:

[0024] 1. This invention improves image quality and ensures accurate identification and positioning by removing noise and reflections through preprocessing;

[0025] 2. Integrating multi-view matching algorithms to solve occlusion and rotation problems and improve robustness;

[0026] 3. By combining coarse and fine adjustment platforms with advanced control, nanometer-level positioning is achieved, improving the success rate. Experiments have verified an accuracy of ±2.5μm, a time of <10s, and a success rate of over 85%. Attached Figure Description

[0027] Figure 1 This is a flowchart of the overall system operation of the present invention;

[0028] Figure 2 This is a flowchart of target recognition and localization. Detailed Implementation

[0029] Specific Implementation Method 1: This implementation method describes a micro-motion positioning and control system based on microscopic vision servoing. The system specifically includes an image acquisition module, an image preprocessing module, a target recognition and positioning module, a motion control module, and a servo control module. The system uses micro-copper wires and pipettes as operating objects. It optimizes the environment and preprocessed images, uses template matching and feature point matching to identify objects and tools, designs a motion control process that integrates stepper motors and piezoelectric ceramics, and introduces fuzzy PID and extended state observers to achieve accurate picking. The effectiveness and control accuracy of the system are verified through experiments.

[0030] Step 1: Image Acquisition. Microscopic images of the micro-copper wires and pipettes are acquired using a front-view camera and a top-view camera, and then transmitted to a host computer for processing. The front-view camera is placed under the microscope to capture the positional relationship between the pipette tip and the micro-copper wires; the top-view camera is placed above to observe the overall scene and avoid blind spots in the front-view perspective. The acquired images have a resolution of 1280×1024 pixels, are in RGB color format, and are acquired at a rate of 30 frames per second to ensure real-time performance. The hardware used is the Thorlabs DCC1545M microscope camera, which features high resolution and low noise. It connects to the host computer via a USB interface. The host computer uses the MFC framework to call the SDK function library to establish links, set parameters, and acquire images.

[0031] During the acquisition process, considering the effects of uneven lighting, shadows, and noise in the microscopic environment, this step optimized the light source in a laboratory setting, using a ring-shaped LED supplemental light to evenly illuminate the operating area and reduce glare interference. Simultaneously, camera parameter settings included automatic exposure and gain control to adapt to the dynamic changes of micron-sized targets. The acquired images were directly input into the preprocessing module to avoid data loss.

[0032] Step 2, Image Preprocessing: The acquired color image undergoes grayscale conversion, Fourier bandpass filtering, improved morphological top-hat algorithm for reflection removal, and optimized Otsu threshold segmentation to achieve noise removal, reflection suppression, and region of interest extraction, thereby improving the accuracy and efficiency of subsequent processing. First, weighted average grayscale conversion is performed, using the following formula:

[0033] Gray = 0.299·f R (i,j)+0.578·f G (i,j)+0.114·f B (i,j)

[0034] Where R(i,j), G(i,j), and B(i,j) are the red, green, and blue components of pixel (i,j), respectively. This formula assigns weights based on the human eye's sensitivity to color, preserving more detailed information and reducing computational load.

[0035] Secondly, a bandpass filter is constructed using Fourier transform to remove mid-to-high frequency noise. The grayscale image is converted to the frequency domain using Fourier transform, and a bandpass filter H(u,v) is designed to retain low-frequency information (background and main structure) while removing high-frequency noise (random interference). The formula is as follows:

[0036] H(u,v)=1

[0037] If D0 ≤ D(u,v) ≤ D1, otherwise it is 0, where D(u,v) is the distance from the frequency point to the center of the spectrum, and D0 and D1 are the inner and outer radii, respectively. The image is recovered by inverse Fourier transform, and the filtered signal is smooth, reducing the impact of noise on edge detection.

[0038] Next, an improved morphological top-hat algorithm is used to remove reflective interference from the target object. Top-hat operation.

[0039] top-hat(I) = I-open(I)

[0040] Here, `open(I)` is a morphological opening operation (erosion followed by dilation), using a circular structuring element (radius 5 pixels) to eliminate small bright spots and reflective areas. The improvement lies in the adaptive structuring element size, adjusting the radius based on the image's grayscale variance to enhance robustness to reflections with varying brightness. Experiments show that this algorithm can reduce the grayscale value of reflective areas by more than 20%, avoiding misjudgments in threshold segmentation.

[0041] Finally, threshold segmentation is performed using the optimized maximum inter-class variance (Otsu) method, with the inter-class variance function being:

[0042] σ=ω0ω1(μ0-μ1) 2

[0043] Where ω0 and ω1 represent the background and foreground probabilities, respectively, and μ0 and μ1 represent the mean gray levels of the background and foreground, respectively. Optimization reduces the impact of uneven foreground-background ratios by weighting several terms to enhance the mean difference, and the threshold corresponding to the maximum σ is obtained.

[0044] T = argmaxσ(T)

[0045] in

[0046] Foreground Probability

[0047] Background ω1=1-ω0

[0048] Foreground Mean

[0049] After binarizing the image, the ROI (Region of Interest) is extracted, such as the rectangle containing the micro-copper wire and the pipette, reducing the computational load by about 50%.

[0050] The image quality is significantly improved after preprocessing, with the signal-to-noise ratio increased by more than 15dB, providing a clear foundation for subsequent recognition.

[0051] Step 3: Target Recognition and Localization: From the main viewpoint, grayscale-based template matching is used to identify the pipette and micro-copper wire regions, with a matching coefficient...

[0052]

[0053] Where T' and I' are the normalized template and the image, respectively. This formula calculates the normalized correlation coefficient, exhibiting strong robustness and insensitivity to changes in illumination. The template is pre-trimmed from a standard image, and a threshold of 0.8 is set after matching to avoid false matches.

[0054] Then, Canny edge detection is applied to extract the contour, and the gradient magnitude is calculated.

[0055]

[0056] direction

[0057]

[0058] Continuous edges were obtained using non-maximum suppression and dual-threshold hysteresis (low threshold 0.4, high threshold 0.8). Tomasi corner detection was applied to the pipette, and the response function was...

[0059] R = min(λ1, λ2)

[0060] Where λ1 and λ2 are the eigenvalues ​​of the covariance matrix, the midpoint of the two corner points of the nozzle is taken as the positioning point, and the accuracy reaches the sub-pixel level. The minimum bounding rectangle is constructed for the micro copper wire, and the contour points are searched point by point downward from the midpoint to achieve a positioning error of <1μm.

[0061] From a top-down perspective, grayscale template matching was used for pipettes, and the positioning point was calculated based on the relative position of the upper left corner of the template and the nozzle. For the micro-copper wires, the BRISK feature point algorithm was used, with descriptors generated as binary strings through brightness comparison. A BF matcher was used to calculate the Hamming distance, combined with a FLANN approximate nearest neighbor search. Mismatches were filtered based on spatial scale similarity, with a threshold set at 0.7 times the average distance, addressing issues of occlusion, out-of-focus, and rotation. In the experiment, the matching success rate reached over 95%, with a time of <0.5s.

[0062] The positioning coordinates are output to the control module through pixel-to-physical distance conversion (calibration factor 1 pixel = 0.5μm).

[0063] Step 4. Motion control: Integrate the stepper motor micro-stage and the piezoelectric ceramic micro-stage to achieve large-range coarse adjustment and nano-scale fine adjustment. The stepper motor stage (Shanghai Zhengxin Optics TM06-300) is used for large-range displacement. The accuracy of a single pulse is 1.25μm, and the total range is 300mm.

[0064] The speed v = pulse frequency f * step distance s

[0065] Supports open-loop or closed-loop mode, displacement distance

[0066] d = ∫vdt

[0067] The feedback error e = target - current. The piezoelectric ceramic stage (Thorlabs MAX311D) is used for small-range precise displacement, with a range of 20μm and voltage control

[0068] u = k * Δx, where k is the gain.

[0069] The two are integrated through the MFC host computer and controlled联动 by calling the ActiveX control. Starting from the coarse adjustment, the stepper stage moves to the micron-level position, and the displacement model

[0070]

[0071] Avoid acceleration jitter. Switch to the piezoelectric stage for fine adjustment, closed-loop control

[0072]

[0073] Contact detection is based on

[0074] The force feedback F = ma

[0075] Threshold F th = 0.1μ.

[0076] Ensure that the target is not damaged.

[0077] Step 5. Servo control: Use the fuzzy PID algorithm to adjust the control parameters, input the error e and the error change rate ec, and output through the fuzzy rules

[0078] ΔK p = M(e,ec)

[0079] Combine the backstepping method to design a non-linear controller, the virtual control law

[0080]

[0081] where z is the tracking error, c>0; Introduce the extended state observer ESO to estimate the uncertainty, use the disturbance observer DOB to compensate for the noise, and the Q filter

[0082]

[0083] The system software implements image processing based on OpenCV, calling cv2.bilateralFilter for bilateral filtering, with weights...

[0084] ω(x,y)=ω d (x,y)*ω s The (x,y) MFC framework is used to construct the interface, including image display, serial communication, sending SDK codes, and establishing connections; the acquisition process involves preprocessing the acquired image and inputting it into the recognition module, as well as addressing positioning errors.

[0085]

[0086] The control module drives the stepper platform for coarse adjustment and switches to the piezoelectric platform for fine adjustment, achieving a pickup success rate of over 85%, an accuracy of ±2.5μm, and a pickup time of <10s.

[0087] The Robert operator is used to optimize edge detection.

[0088]

[0089] gradient magnitude

[0090] Improved threshold in Otsu

[0091] T = argmaxσ(T)

[0092] The morphological opening operation erosion(dilation(I)) diffuses the edge, the bitwise AND operation I_and = I&mask, and the bitwise OR operation I_or = I|C, to obtain the final segmented region.

[0093] This implementation method automates micro-manipulation under microscopic vision through the above steps, and experiments have verified its effectiveness in micro-copper wire picking tasks.

[0094] Specific implementation method two: This implementation method differs from the first one in that the acquisition module adds autofocus, uses phase difference detection, adjusts the lens to the optimal focal plane, and improves image sharpness by 15%.

[0095] The other steps are the same as in Method 1.

[0096] Specific implementation method three: This implementation method differs from one or two in that the preprocessing adds median filtering to assist Fourier transform, with a window size of 3×3, which improves the noise suppression rate by 8%.

[0097] The other steps are the same as in method one or two.

[0098] Specific implementation method four: This implementation method differs from one to three in that the identification module adds RANSAC to remove false matches, and the BRISK matching accuracy reaches 96%.

[0099] The other steps are the same as those in methods one through three.

[0100] Specific implementation method five: This implementation method differs from one to four in that vibration compensation is added to the control module, ESOβ1=100, β2=300, β3=1000, and the disturbance estimation error is <0.2μm.

[0101] The other steps are the same as those in methods one through four.

[0102] Specific implementation method six: This implementation method differs from one to five in that adaptive fuzzy rules are added to the servo, and the membership function is optimized based on a genetic algorithm, which improves the response speed by 20%.

[0103] The other steps are the same as those in steps one through five.

[0104] Example

[0105] This embodiment presents a micro-motion positioning and control system based on microscopic vision servoing, operating using the method described in Specific Implementation Method 1 above. Taking micro-copper wire pickup as an example, the acquired image resolution is 1280×1024. After grayscale conversion, noise is removed by filtering, and after reflection suppression, ROI is extracted by threshold segmentation. The main view matching R=0.85, Canny edge detection, and Tomasi nozzle positioning midpoint error are 0.3 pixels. The top-view BRISK matching has an average Hamming distance of 25, with a success rate of 98% after filtering. The stepper platform moves 100μm, piezoelectric fine-tuning is 5μm, fuzzy PID adjustment Kp=1.2, ESO estimated disturbance is <0.1μm, DOB noise compensation is used, and the pickup time is 8s with an accuracy of ±2μm.

[0106] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A micro-motion positioning and control system based on microscopic vision servoing, characterized in that, The system includes an acquisition module for capturing micro-manipulation images using a front-view camera and a top-view camera; an image preprocessing module based on weighted average grayscale conversion and Fourier bandpass filtering; a target recognition and localization module using grayscale template matching and BRISK feature point matching; a motion control module integrating a stepper motor micro-motion platform and a piezoelectric ceramic micro-motion platform; and a servo control module that achieves precise picking through fuzzy PID and an extended state observer. The system uses micro-copper wires and pipettes as the manipulation objects, optimizes the image environment through preprocessing, identifies and locates the target objects, and designs a motion control process to achieve automatic operation of the micro-copper wires.

2. The micro-motion positioning and control system based on microscopic vision servoing according to claim 1, characterized in that, The specific process of the image preprocessing module is as follows: The acquired color image is converted to grayscale using a weighted average method, with the following formula: Gray=0.299·f R (i,j)+0.578·f G (i,j)+0.114·f B (i,j) Where f R f G f B These are the red, green, and blue components of pixel (i,j); then, a bandpass filter is constructed using Fourier transform to remove mid-to-high frequency noise. The improved morphological top-hat algorithm is used to remove the reflective interference of the target object; finally, the optimized Otsu's method is used for threshold segmentation and the region of interest is extracted to reduce the amount of computation.

3. A micro-motion positioning and control system based on microscopic vision servoing according to claim 2, characterized in that, The specific process of the target recognition and localization module from the main viewpoint is as follows: Step 1: Use a grayscale-based template matching algorithm to identify the pipette and micro-copper wire regions; Step 2: Apply the Canny edge detection algorithm to extract the edge contours of the pipette and the micro-copper wire; Among them, G u (i,j) is the gradient image of the horizontal direction of the image, G v (i,j) is the gradient image of the vertical direction of the image, G uv (i,j) is the gradient, S u S v For Sobel operators; Step 3: Use the Tomasi corner detection algorithm to extract the two ends of the pipette tip and calculate the midpoint of the two ends as the positioning point of the pipette tip; Step 4: Construct the minimum bounding rectangle for the micro copper wire, extract the center point of the rectangle, and search for pixels on the contour point by point along the vertical downward direction to determine the positioning point of the micro copper wire.

4. A micro-motion positioning and control system based on microscopic vision servoing according to claim 3, characterized in that, The specific process of the target recognition and localization module from a top-down perspective is as follows: Step 1: Identify the pipette using a grayscale-based template matching algorithm, and calculate the positioning point of the pipette nozzle based on the relative positional relationship between the upper left corner point in the template image and the nozzle. Step 2: Use the BRISK feature point detection and description algorithm to extract feature points from the micro copper wires, count the gradient of all point pairs and divide them into short-distance point pairs P and set S and long-distance point pairs P and set representation according to the threshold. Step 3: Use the BF matcher or FLANN matcher to perform preliminary matching of feature points; Step 4: Filter matching points based on spatial feature similarity to eliminate mismatched points, solve occlusion, out-of-focus and rotation problems, and improve matching efficiency and accuracy.

5. A micro-motion positioning and control system based on microscopic vision servoing according to claim 4, characterized in that, The motion control module includes a stepper motor micro-motion platform for large-range displacement, with a single pulse accuracy of 1.25μm and a total range of 300mm; and a piezoelectric ceramic micro-motion platform for small-range precise displacement, with a range of 20μm; the two are linked and controlled by an MFC host computer and ActiveX controls to achieve speed and distance adjustment in open-loop or closed-loop modes.

6. A micro-motion positioning and control system based on microscopic vision servoing according to claim 5, characterized in that, The servo control module uses a fuzzy PID algorithm to adjust control parameters and combines it with a backstepping method to design a nonlinear controller. An extended state observer (ESO) is introduced to estimate system uncertainties and external disturbances. A disturbance observer (DOB) is used to compensate for high-frequency noise and jitter, thereby improving the system's robustness and stability.

7. A micro-motion positioning and control system based on microscopic vision servoing according to claim 6, characterized in that, The system implements image processing and target recognition modules using the OpenCV function library; the MFC application framework constructs the interactive interface, including image display, port communication, and control command sending; hardware and software integration ensures real-time feedback and automatic operation.

8. A micro-motion positioning and control system based on microscopic vision servoing according to claim 7, characterized in that, The pickup process of the system is as follows: after preprocessing, the image input recognition module positions the pipette and the micro copper wire; the control module drives the stepping platform to coarsely adjust the position to the micrometer level; the piezoelectric platform is switched to finely adjust to the nanometer level contact; the pickup success rate and positioning accuracy are verified through experiments to improve the automation level of micro-operations.

Images