A focusing method and system for a micro actuator of a micromanipulation system

Abstract

The present invention discloses a defocus tracking method and system for a micro actuator of a micromanipulation system. Images are acquired by an optical microscope in the present invention, and the micro actuator is located through machine vision; a manipulator driving the micro actuator is equipped with a nanoscale grating sensor, which is used to detect the displacement of the micro actuator in a plane parallel to the focal plane; by combining the image feedback and the feedback information of the grating sensor, the planar displacement of the micro actuator caused by defocus is calculated, and based on this planar displacement information, the movement speed of the microscope focusing module is changed to compensate for the defocus caused during the movement process. The present invention can be applied to a micromanipulation platform built based on an optical microscope and has broad application prospects in the fields of cell micromanipulation, microinjection, and other related life sciences.

Description

Technical Field

[0001] The present invention relates to the field of automated micromanipulation, and particularly to a focusing method and system for a micro actuator of a micromanipulation system. Background Art

[0002] In recent years, micromanipulation techniques, such as cell micromanipulation and microinjection, have been widely applied in research and practice related to the field of life sciences. However, the training cycle for relevant personnel engaged in microinjection is long, and manual operation has limitations such as low precision, low controllability, low stability, and poor repeatability, which restricts the efficiency of relevant research and practice. To improve the precision, controllability, stability, and repeatability of micromanipulation, and to reduce the dependence on professional personnel, automated micromanipulation has been widely studied. However, when the micro actuator moves in a direction perpendicular to the focal plane, the micro actuator will move out of the focal plane, and the image in the imaging plane will become blurred, which may lead to failure of machine vision recognition and thus failure of the automated microoperation process. Therefore, it is crucial to achieve focusing of the micro actuator during the movement process. Existing microscope autofocus techniques cannot be used for focusing and are only applicable to focusing on static objects, that is, when the micro actuator is stationary in a certain plane, the position of the microscope focal plane is adjusted through image feedback, and based on the clarity calculation method, it is determined that the micro actuator is in the focal plane. When the micro actuator moves in a direction perpendicular to the focal plane, the micro actuator first performs a small step displacement and then focuses, repeating the above process until the micro actuator moves to the target position. The process is not continuous, which limits the efficiency of micromanipulation.

[0003] During the microoperation process, affected by various liquids in the culture dish (such as mineral oil and culture medium), the optical path of the microscope focal plane is compressed. Therefore, when the microscope focusing axis moves a certain distance, the moving distance of the focal plane is different from the moving distance of the focusing axis. Therefore, even if the defocus distance of the micro actuator is clear, after the microscope focusing axis moves backward by the defocus distance, the micro actuator is still in the defocus state. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention proposes a focusing method and system for a micro actuator of a micromanipulation system. The device of the present invention is arranged such that a digital camera acquires the image of the micro actuator under an optical microscope, and the displacement information of the micro actuator is obtained through machine vision. The displacement information D of the micro actuator in the two-dimensional plane obtained based on machine vision includes two parts. One part is the displacement D 1 introduced by the defocus of the micro actuator, and the other part is the displacement D 2 introduced by the movement of the micro actuator itself in the two-dimensional plane. Through the grating sensor built in the manipulator, the movement parameter D of the micro actuator in the two-dimensional plane can be obtained 2Combining the feedback information of machine vision and the grating sensor, the displacement information D introduced by defocus of the micro actuator can be obtained. 1 Based on D 1 , change the movement speed of the microscope focusing axis to make the microscope focal plane continuously approach the plane where the micro actuator is located.

[0005] The present invention is realized by the following technical solutions:

[0006] The present invention provides a focusing method for a micro actuator in a microscopic operating system. Its essence is to perform defocus compensation based on the feedback information obtained from defocus detection during the movement of the micro actuator, and finally achieve focusing.

[0007] Defocus detection: The digital camera obtains the image of the micro actuator under the microscopic operating system, and the displacement information of the micro actuator is obtained through machine vision. The displacement information of the micro actuator in the two-dimensional plane includes two parts. One part is the displacement introduced by the defocus of the micro actuator, and the other part is the displacement introduced by the movement of the micro actuator itself in the two-dimensional plane. Through the grating sensor built in the manipulator, the movement parameters of the micro actuator in the two-dimensional plane can be obtained. Combining the feedback information of machine vision and the grating sensor, the displacement information introduced by defocus of the micro actuator can be obtained. Since the defocus distance in the Z-axis direction is proportional to the displacement information introduced by defocus, based on the displacement information introduced by defocus, the defocus direction and distance can be calculated.

[0008] Defocus compensation: Change the movement speed of the microscope focusing axis to reduce the distance between the micro actuator and the focal plane; repeat the defocus detection and defocus compensation, so that the distance between the micro actuator and the focal plane approaches 0, and the moving speed of the focal plane (i.e., the movement speed of the microscope focusing axis, the two are equal) gradually approaches the movement speed of the micro actuator on the imaging optical path, so as to ensure that the micro actuator is always in the focal plane and achieve the focusing process.

[0009] A focusing method for a micro actuator in a microscopic operating system is as follows:

[0010] The initial state of the micro actuator is a stationary state and is located in the focal plane.

[0011] Further, based on the image information fed back by the digital camera, detect the position information d of the end of the micro actuator at the initial time. 0 ;

[0012] Further, after the micro actuator starts to move, based on the image information fed back by the digital camera, use the template matching algorithm to detect the position information d of the end of the micro actuator at time t. t ;

[0013] Further, obtain the planar displacement information d of the manipulator at time t through the grating sensor.m ;

[0014] Further, calculate the plane displacement information introduced by defocus, Δd = d t - d m - d 0 ;

[0015] Further, based on the error information Δd, calculate the change in the movement speed of the microscope focusing axis through a PID controller. The change in speed output after passing through the PID controller includes both the change in speed magnitude and the change in speed direction;

[0016] Further, the microscope focusing axis changes its movement speed, and the change in movement speed is the output of the PID controller.

[0017] On the other hand, the present invention provides a focus tracking system for a micro - actuator of a microscopic operating system. The system includes a defocus detection system, a defocus analysis system, and a defocus compensation system.

[0018] The defocus detection system obtains the optical microscope image by a CCD camera and obtains the translational information of the manipulator by a grating sensor.

[0019] The defocus analysis system outputs the magnitude and direction of the change in the speed of the microscope focusing axis based on the feedback information of the CCD camera and the grating sensor.

[0020] The defocus compensation system changes the movement speed of the microscope focusing axis according to the output of the defocus analysis system to complete a focusing control, so that the microscope focal plane continuously approaches the plane where the micro - actuator is located, and the moving speed of the focal plane (the movement speed of the microscope focusing axis) is consistent with the movement speed of the micro - actuator on the imaging optical path, thereby ensuring that the micro - actuator is always within the focal plane and realizing focus tracking.

[0021] The beneficial effects of the present invention are as follows: A focus tracking method and system for a micro - actuator of a microscopic operating system designed by the present invention, for defocus detection, determines whether the micro - actuator is defocused by detecting the plane displacement introduced by the defocus of the micro - actuator. Compared with the clarity discrimination method, the defocus distance and defocus direction can be intuitively obtained; the defocus detection is based on the feedback information of the CCD camera and at the same time integrates the feedback information of the manipulator grating sensor, ensuring that defocus detection can be realized under any movement of the micro - actuator; for defocus compensation, the movement speed depends on the microscope. Since the method of the present invention can achieve automatic focus tracking, the focusing speed is significantly faster than the existing manual focusing. The present invention provides a focusing method for a moving micro - actuator under an optical microscope with a simple structure, convenient debugging, and can be realized based on existing commercial devices, which can be applied to a microscopic operation platform built based on an optical microscope and has a wide application prospect in the fields of cell micro - operation, micro - injection, and other related life sciences. Description of the Drawings

[0022] Figure 1 Flow chart of the focusing method for the micro actuator of the microscopic operation system according to the present invention;

[0023] Figure 2 Schematic diagram of the focusing system for the micro actuator of the microscopic operation system according to the present invention;

[0024] Figure 3 Schematic diagram of the device for implementing the method of the present invention;

[0025] Figure 4 Schematic diagram of the defocusing distance of the injection micro needle and the planar displacement caused by defocusing;

[0026] Figure 5 Relationship between the defocusing distance of the injection needle and the planar displacement caused by defocusing;

[0027] Figure 6 Focusing error during the movement of the injection needle measured by the method of the present invention. Detailed implementation manners

[0028] The following makes a detailed description of the embodiments of the present invention. The embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. However, the protection scope of the present invention is not limited to the following embodiments.

[0029] As Figure 1 shown, the focusing method for the moving micro actuator under an optical microscope includes the following steps: The CCD camera acquires the microscope imaging, transmits the image to the computer to extract the planar displacement information of the micro actuator, the grating sensor acquires the planar displacement information of the manipulator (the movement information of the micro actuator in the two-dimensional plane), transmits it to the computer and fuses it with the feedback information of the CCD camera to extract the defocusing information, the computer outputs the change amount of the movement speed of the microscope focusing axis according to the defocusing information and transmits it to the microscope movement controller, and the microscope focusing axis changes the movement speed.

[0030] In the process of the CCD camera sensor acquiring the microscope imaging, the optical microscope is responsible for imaging and the CCD camera takes pictures.

[0031] In the process of the computer extracting the displacement information of the micro actuator, the computer processes the image taken by the CCD camera through the template matching algorithm to extract the displacement information of the micro actuator.

[0032] In the process of the grating sensor acquiring the planar displacement information of the manipulator, the built-in grating sensor of the manipulator acquires the displacement information of the manipulator.

[0033] The computer fuses the feedback information with the CCD camera sensor to extract defocus information. The displacement information of the micro actuator in the two-dimensional plane obtained based on the image feedback includes two parts. One part is the displacement introduced by the defocus of the micro actuator, and the other part is the displacement introduced by the movement of the micro actuator itself in the two-dimensional plane. The displacement introduced by the movement of the micro actuator itself in the two-dimensional plane can be obtained through the grating sensor built in the manipulator. By combining the image feedback and the feedback information of the grating sensor, the plane displacement information introduced by the defocus of the micro actuator can be obtained.

[0034] The computer outputs the change amount of the movement speed of the microscope focusing axis based on the defocus information, and uses a PID controller to output the change amount of the speed according to the defocus distance.

[0035] By changing the movement speed of the microscope focusing axis, the distance between the focal plane and the plane where the micro actuator is located is reduced.

[0036] Repeat the above process in a loop, so that the focal plane gradually moves to the plane where the micro actuator is located, and the movement speed of the microscope focusing axis is consistent with the movement speed of the micro actuator on the imaging optical path, realizing the focus tracking of the moving micro actuator under the optical microscope.

[0037] As Figure 2 shown, a focus tracking system for the micro actuator of the microscopic operation system includes a defocus detection system, a defocus analysis system, and a defocus compensation system; each of the systems forms a feedback control loop.

[0038] The defocus detection system obtains the microscope imaging through a CCD camera, extracts the displacement information of the micro actuator through machine vision, and obtains the manipulator displacement information through a grating sensor.

[0039] The defocus analysis system calculates the defocus information of the micro actuator based on the displacement information output by the defocus detection system, and uses a PID controller to output the magnitude and direction of the change in the movement speed of the microscope focusing axis based on the defocus information.

[0040] The defocus compensation system changes the movement speed of the microscope focusing axis based on the magnitude and direction of the speed change output by the defocus analysis system, so that the microscope focal plane continuously approaches the plane where the micro actuator is located. The refresh rate of the movement speed of the microscope focusing axis is the same as the sampling rate of the CCD camera sensor.

[0041] As Figure 3 shown, it is an embodiment of the method for focus tracking of a moving micro actuator under an optical microscope. Among them, the optical microscope is an electric inverted optical microscope, the manipulator is a three-axis right-angle coordinate manipulator with a built-in grating sensor. In this example, the inverted microscope is equipped with a 40x objective lens, and the micro actuator is an injection micro needle for cell microinjection.

[0042] The initial state of the injection microneedle is a stationary state and it is located within the focal plane. Based on the image information fed back by the CCD, the position information d of the tip of the injection microneedle at the initial time is detected. 0 ;

[0043] After the injection microneedle starts to move, defocus detection begins: Based on the image information fed back by the CCD camera, the position information d of the tip of the micro actuator is detected. t , and the detection algorithm is the template matching algorithm; the planar displacement information d of the manipulator is obtained through the grating sensor. m ;

[0044] After completing the defocus detection, defocus analysis begins: Calculate the planar displacement information introduced due to the defocus of the injection microneedle, Δd = d t -d m -d 0 ; Based on the error information Δd, calculate the change in the movement speed of the microscope focusing axis through the PID controller. The change in speed output after passing through the PID controller includes the magnitude and direction of the speed change.

[0045] After completing the defocus analysis, perform defocus compensation. The microscope focusing axis changes its movement speed, and the amount of change in movement speed is the output of the PID controller.

[0046] During the movement of the injection microneedle, repeat the processes of defocus detection, defocus analysis, and defocus compensation to achieve the tracking focus of the moving injection microneedle.

[0047] As Figure 4 、 Figure 5 shown, it is the relationship between the defocus distance of the injection microneedle in the embodiment and the planar displacement caused by defocus, and the magnification is 40X. Figure 5 It shows that when the defocus distance is within a certain range (±25 pixels, approximately ±11 microns), there is a linear relationship between the defocus distance and the planar displacement caused by defocus. In this example, the sampling frequency of the digital camera is 25Hz. Therefore, when the displacement speed of the injection needle along the Z-axis direction is less than 250 microns per second, the linear displacement caused by the plane can accurately represent the defocus distance.

[0048] Figure 6 The figure shown is the comparison diagram of the implementation effects, which are the defocus compensation effects of the injection microneedle moving only along the Z-axis direction, the injection microneedle moving along the Z-axis and X-axis directions simultaneously, and the injection microneedle moving along the Z-axis and Y-axis directions simultaneously. It can be seen from the figure that the maximum focusing error is less than 3 pixel points (approximately 1.3 microns), meeting the usage requirements.

[0049] The specific embodiments of the present invention have been described above with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that various changes and substitutions can be made to the specific embodiments of the present invention without departing from the spirit and scope of the present invention. These changes and substitutions all fall within the scope defined by the claims of the present invention.

Claims

1. A focusing method for a micro actuator in a micro operation system, characterized in that, during the movement of the micro actuator, based on the feedback information obtained by defocus detection, defocus compensation is performed, and finally focusing is achieved; the specific steps are as follows: Step 1, defocus detection: Combine the micro actuator position information obtained by machine vision and the micro actuator position information feedback by the grating sensor, calculate the planar displacement of the micro actuator caused by defocus, and based on this planar displacement information, obtain the direction and distance of the micro actuator defocus; Step 2, defocus compensation: Based on the direction and distance of the micro actuator defocus obtained in Step 1, change the movement speed of the microscope focusing axis, move the focal plane to the plane where the micro actuator is located, thereby completing defocus compensation; and make the movement speed of the microscope focusing axis consistent with the movement speed of the micro actuator on the imaging optical path, so as to ensure that the micro actuator is always in the focal plane and achieve focusing.

2. A focusing method for a micro actuator in a micro operation system according to claim 1, characterized in that, The grating sensor is used to obtain the planar displacement information of the micro actuator, so as to ensure that even when the micro actuator moves along both the plane and the Z-axis directions simultaneously, defocus detection can still be performed.

3. A focusing method for a micro actuator in a micro operation system according to claim 1, characterized in that, By changing the movement speed of the microscope focusing axis, it is ensured that the micro actuator is always in the focal plane, and the micro actuator during the movement process is clearly visible.

4. A focusing method for a micro actuator in a micro operation system according to claim 1, characterized in that, Based on the planar displacement caused by defocus, through a PID controller, calculate the change amount and change magnitude of the movement speed of the microscope focusing axis, thereby reducing the distance and movement speed difference between the microscope focal plane and the plane where the micro actuator is located, and finally making the microscope focal plane coincide with the plane where the micro actuator is located.

5. A focusing system for a micro actuator in a micro operation system, characterized in that, The system includes a defocus detection system, a defocus analysis system, and a defocus compensation system; the defocus detection system collects microscope imaging information through a CCD camera and collects the in-plane displacement information of the manipulator through a grating sensor; the defocus analysis system, combining the image information collected by the CCD camera and the displacement information collected by the grating sensor, outputs the change magnitude and change direction of the speed of the microscope focusing axis; the defocus compensation system, based on the change magnitude and change direction of the speed output by the defocus analysis system, changes the movement speed of the microscope focusing axis, so that the microscope focal plane continuously approaches the plane where the micro actuator is located.

Images