A probe and sample precise control method and control system based on machine vision
Through machine vision-based methods, the distance between the probe and the sample is measured in real time and regulated, the problem of difficult position control between probe and biological samples in the prior art is solved, and high-precision biological sample measurement and detection is achieved, avoiding sample damage , improves measurement efficiency and detection accuracy.
Patent Information
- Application Number
- CN202210650852.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-10
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-06-10
AI Technical Summary
The lack of effective methods in the prior art to precisely control the relative position of the probe and biological samples, limiting the application of near-field scanning imaging technology in biological samples research.
Using a machine vision-based method, the distance between the probe and the sample is measured in real time through parallel light sources, CCD cameras and three-dimensional displacement tables, and the distance between the probe and the sample is measured by feedback to the displacement table through the computer data processing system to control it, so as to achieve accurate control of the distance between the probe and the sample. .
High-precision measurement and control of biological samples are achieved, sample damage is avoided, measurement efficiency and detection speed are improved, and the objective accuracy of detection results is ensured.
Smart Images

Figure CN114926451B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine vision measurement, and more particularly to a method and a control system for accurately controlling a probe and a sample based on machine vision. Background Art
[0002] In recent years, terahertz imaging technology has great application prospects in the biomedical field due to its good biosafety and label-free detection of biological molecules and tissues.
[0003] Compared with traditional imaging technology for studying biological tissue levels, today's near-field scanning imaging technology can break through the diffraction limit and obtain high resolution at the sub-wavelength scale. In the existing technology, the probe is used at a distance of 3-10μm from the sample surface to detect, collect and detect the sample surface signal corresponding to the needle tip.
[0004] However, due to the complexity of biological samples and the lack of effective and precise means to control the relative position of the probe and the sample, the existing technology has focused more on metamaterials and semiconductor materials with flat appearance and stable properties, and less on biological samples. There is no good method to precisely control the probe to promote near-field scanning imaging technology for imaging and detection of biological samples.
[0005] Therefore, how to provide a method and control system for precise control of probes and samples based on machine vision is an urgent problem to be solved by those skilled in the art. Summary of the invention
[0006] In view of this, the present invention provides a method and control system for precise control of probes and samples based on machine vision. The method uses a machine vision method to quickly measure the probe-sample distance of a terahertz near-field system and feed it back to a translation stage to adjust the needle tip-sample distance, thereby achieving real-time and rapid detection of the distance from the scanning probe to the sample, and feeding it back to the three-dimensional translation stage to achieve distance control.
[0007] In order to achieve the above object, the present invention provides the following technical solutions:
[0008] In one aspect, the present invention provides a method for precise control of a probe and a sample based on machine vision, comprising the following steps:
[0009] S100: Setting the position of the parallel light source and the irradiation position of the laser;
[0010] S200: obtaining a grayscale image of the probe and the sample;
[0011] S300: converting the grayscale image of the probe and the sample into a binary image of the probe and the sample;
[0012] S400: Acquire a target edge point from the probe to the sample in a binary image of the probe and the sample to obtain a contour from the probe to the sample;
[0013] S500: traverse the point set of the probe and the sample in the binary image of the probe and the sample, and obtain the distance from the needle tip to the sample in the binary image of the probe and the sample;
[0014] S600: based on the calibration coefficient, convert the distance from the needle tip to the sample in the binary image of the probe and the sample into an actual distance value and display it;
[0015] S700: Change the distance between the needle tip and the sample according to the actual distance value to control the probe and the sample.
[0016] Preferably, the method further includes timing while obtaining the grayscale images of the probe and the sample, and when the actual distance value is obtained, the timing ends and the time taken to measure the distance is displayed.
[0017] Preferably, the S300 specifically includes:
[0018] S310: obtaining a histogram of the grayscale image of the probe and the sample and normalizing it to obtain a normalized histogram;
[0019] S320: Segment the threshold of the normalized histogram based on the large-law binarization to obtain a binary image of the probe and the sample.
[0020] Preferably, it is characterized in that the S320 specifically includes:
[0021] Calculate the background pixel ratio ω1:
[0022]
[0023] Where N1 is the number of background pixels and N is the total number of pixels;
[0024] Foreground pixel ratio ω2:
[0025]
[0026] Where N2 is the number of foreground pixels;
[0027] Calculate the background average gray value μ1:
[0028]
[0029]
[0030] Where P i is the probability that the number of pixels with gray level i accounts for the total number of pixels N,
[0031] Foreground average gray value μ2:
[0032]
[0033] Where, M is a fixed coefficient, M = 256;
[0034] Calculate the grayscale cumulative value μ in the 0-M interval:
[0035] μ=u1*ω1+u2*ω2
[0036] Calculate the within-group variance g:
[0037] g=ω1*(μ-μ1) 2 +ω2*(μ-μ2) 2
[0038] Finally it simplifies to:
[0039] g=ω1*ω2*(μ1-μ2) 2 .
[0040] The simplification is to illustrate that the between-class variance is only related to the proportion of foreground and background and the average grayscale value, which is convenient for calculation.
[0041] When the inter-class variance is the largest, the gray level i corresponding to the maximum value g is used as the global threshold of the image, so that the image is divided into a binary image according to the threshold.
[0042] Preferably, the S400 specifically includes:
[0043] S410: using the Canny operator as a convolution kernel to convolve the binary image of the probe and the sample, calculating the gradient, and obtaining the target edge point from the probe to the sample;
[0044] S420: The coarse edge between the probe and the sample in the image is converted into a fine edge by a non-maximum suppression method, and the edge point from the needle tip to the sample is suppressed and deleted in the refinement. Along the gradient direction, each point of the coarse edge has a corresponding gradient, and the point corresponding to the maximum gradient is retained, and the remaining points are deleted, thereby converting the coarse edge point into a fine edge point.
[0045] S430: Use double thresholds to connect strong and weak edges to obtain complete probe-to-sample contour information.
[0046] Preferably, the S500 specifically includes:
[0047] S510: traverse the point set from the probe to the pixel coordinates of the sample contour, and retain the distance coordinates;
[0048] S520: Calculate the distance from the tip to the sample in the binary image of the probe and the sample according to the two coordinates.
[0049] Preferably, the S600:
[0050] S610: Acquire a calibration board checkerboard image;
[0051] S620: Detecting sub-pixel coordinates of corner points in the checkerboard image by using a corner point detection method;
[0052] S630: Calculate the actual distance of the sub-pixel coordinates and the coordinate pixel distance ratio;
[0053] S640: Calculate the actual distance from the probe to the sample according to the actual distance and the coordinate pixel distance ratio, convert the distance from the needle tip to the sample in the binary image into the actual distance value and display it.
[0054] Preferably, the S700 specifically includes:
[0055] S710: when the scanning probe is working, the actual distance value displayed is determined, and when the distance between the needle tip and the sample is less than the set threshold, the distance between the needle tip and the sample is controlled to be farther away;
[0056] S720: When the distance between the needle tip and the sample is greater than a set threshold, the distance between the needle tip and the sample is controlled to be closer.
[0057] On the other hand, the present invention also provides a system for a method for precise control of a probe and a sample based on machine vision, comprising:
[0058] A CCD camera (1) is used to obtain a grayscale image of the probe and the sample;
[0059] A double telecentric lens (2) connected to the CCD camera (1);
[0060] Probe and fixture 3;
[0061] A biological sample 4, wherein the probe and the fixing device 3 are arranged opposite to the biological sample 4;
[0062] A three-dimensional moving stage control device (5) is connected to the probe and the fixing device (3), and the biological sample (4) is placed on a displacement stage (10) controlled by the three-dimensional moving stage control device (5);
[0063] A computer data processing system (6) connected to the CCD camera (1) and the three-dimensional moving stage control device (5);
[0064] A parallel light source device (7), used for providing light source to the CCD camera (1);
[0065] A light source controller (8) connected to the parallel light source device (7);
[0066] A laser (9) is used to generate laser light and irradiate the laser light onto the biological sample (4).
[0067] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method and control system for precise control of a probe and a sample based on machine vision, which has the following beneficial effects:
[0068] 1. The present invention can achieve good measurement accuracy for biological samples with smooth or complex surfaces, and the nano-displacement stage can well feedback and control the changes in tiny distances;
[0069] 2. The present invention can realize the recovery of samples. Because the terahertz wave has low energy, the non-contact detection process of machine vision will not destroy the molecular structure of biological samples. After distance detection, the samples can be recovered and repeated experiments can be performed, thus avoiding the waste and loss of sample resources.
[0070] 3. The machine vision system provided by the present invention is relatively simple to build, easy to observe with visualization, fast detection speed, etc., which greatly improves the measurement efficiency;
[0071] 4. The present invention realizes real-time and rapid detection of the scanning probe-sample distance, and feeds back to the three-dimensional translation stage to realize distance regulation, so that the detection result is objective and accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0073] Figure 1 A schematic diagram of the flow chart of the method for precise control of probes and samples based on a machine vision system provided by the present invention;
[0074] Figure 2 A schematic diagram of the structure of a probe and sample precision control system based on a machine vision system provided in Example 1 of the present invention;
[0075] Figure 3 The needle tip-sample pixel distance provided in Example 2 of this invention is converted into the actual distance and displayed on the computer data processing display diagram. DETAILED DESCRIPTION
[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0077] Example 1
[0078] See attached Figure 1 As shown, Embodiment 1 of the present invention discloses a method for precise control of a probe and a sample based on a machine vision system, comprising:
[0079] S100: Parallel light source irradiates the CCD camera equipped with a double telecentric lens, and the laser irradiates the fixed probe tip and the sample surface placed on the three-dimensional translation stage;
[0080] S200: When the camera acquires the probe-sample image, the timing starts;
[0081] S300: converting the grayscale image of the probe-sample captured by the CCD camera into a binary image;
[0082] S400: obtaining edge points of the target according to an edge detection algorithm, and finding the contour of the probe sample by a target detection method;
[0083] S500: traverse all point sets in the two parts of the image to obtain the shortest distance between the needle tip and the sample in the image;
[0084] S600: Based on the calibration coefficient between the image size and the actual size of the machine vision system, the image minimum distance is converted into the actual minimum distance value and displayed. At the same time, the timing ends and the elapsed time is displayed;
[0085] S700: The three-dimensional translation stage is adjusted according to the distance value to change the distance between the needle tip and the sample, and the probe and the sample are controlled.
[0086] In a specific embodiment, in step S100:
[0087] Parallel light source can better observe the outline of probe-sample;
[0088] The CCD camera is installed with a double telecentric lens, and the image obtained will hardly be distorted.
[0089] In a specific embodiment, the specific method of step S300 is:
[0090] Get the histogram of the image and normalize it;
[0091] Based on the big law binarization, the image threshold is segmented, and the target to be detected in the image is divided into two parts: background and target according to the threshold. The optimal threshold is obtained to distinguish the two types of pixels, where M = 256 and S is the total number of pixels. The specific process is:
[0092] Calculate the background pixel ratio ω1 :
[0093]
[0094] Foreground pixel ratio ω2 :
[0095]
[0096] Calculate the background average gray value μ1:
[0097]
[0098] Foreground average gray value μ2:
[0099]
[0100] Calculate the grayscale cumulative value μ in the 0-M interval:
[0101] μ=u1*ω1+u2*ω2
[0102] Calculate the within-group variance g:
[0103] g=ω1*(μ-μ1) 2 +ω2*(μ-μ2) 2
[0104] Finally it simplifies to:
[0105] g=ω1*ω2*(μ1-μ2) 2
[0106] In a specific embodiment, the specific method of step S400 is:
[0107] Use the Canny operator as the convolution kernel to convolve the image and calculate the gradient;
[0108] Non-maximum suppression thins the tip-sample edge;
[0109] Double thresholds are used to connect strong and weak edges to obtain complete edge information.
[0110] In a specific embodiment, the specific method of step S500 is:
[0111] Since the determined contour and interior are composed of a finite number of pixel points, the coordinates of the probe-sample contour pixels are traversed and the coordinates with the shortest distance are retained;
[0112] Compute the Euclidean distance P between two coordinates.
[0113] In a specific embodiment, the specific method of step S600 is:
[0114] Use the system to capture a calibration checkerboard image;
[0115] Corner detection chessboard corner sub-pixel coordinates;
[0116] Calculate the actual distance of the sub-pixel coordinates and the coordinate pixel distance ratio K;
[0117] The actual distance between the probe and the sample is L=PK.
[0118] The connection line between the two coordinates in step S500 is visually drawn to display the actual distance between the probe and the sample. At the same time, the timing ends and the elapsed time is displayed.
[0119] In a specific embodiment, the specific method of step S800 is:
[0120] According to the displayed distance, when the scanning probe is working, when the distance between the needle tip and the sample is less than 3μm, the Z-axis spacing of the translation stage is controlled to be far away to avoid damaging the needle tip and destroying the morphological characteristics of the sample;
[0121] According to the displayed distance, when the scanning probe is working and the tip-sample distance is greater than 3 μm, the spacing of the Z axis of the translation stage is controlled to be closer to avoid failure to obtain high-resolution images.
[0122] On the other hand, see Appendix Figure 2 As shown, Embodiment 1 of the present invention further provides a probe and sample precision control system based on machine vision, comprising:
[0123] CCD camera 1, used to obtain grayscale images of the probe and sample;
[0124] A bi-telecentric lens 2 connected to the CCD camera 1;
[0125] Probe and fixture 3;
[0126] A biological sample 4, wherein the probe and the fixing device 3 are arranged opposite to the biological sample 4;
[0127] A three-dimensional moving stage control device 5 is connected to the probe and the fixing device 3 and places the biological sample 4 on a translation stage 10 controlled by the three-dimensional moving stage control device 5;
[0128] A computer data processing system 6 connected to the CCD camera 1 and the three-dimensional moving stage control device 5;
[0129] A parallel light source device 7, used to provide light source for the CCD camera 1;
[0130] A light source controller 8 connected to the parallel light source device 7;
[0131] The laser 9 is used to generate laser light and irradiate the biological sample 4 .
[0132] Specifically, the dual telecentric lens in the system device provided by the present invention will not produce image distortion during the shooting process, the parallel light source is conducive to the extraction of the target boundary part, the laser irradiates the needle tip to generate a detection laser, the biological sample is in a horizontal position of the translation stage, and the three-dimensional translation stage control device can accurately control the movement.
[0133] Example 2
[0134] This embodiment 2 takes a lung cell biological sample with a complex and rough surface as an example to explain the embodiment 1 of the present invention in detail:
[0135] (1) Visually measure the tip-sample distance quickly
[0136] 1. Run the terahertz near-field scanning system and control the three-dimensional translation stage to move the biological sample on the platform toward the end of the probe (photoconductive probe).
[0137] 2. The system camera captures images and sends real-time images to the computer data processing system.
[0138] 3. Image processing stage: convert to grayscale image, binarize by large law, use Canny operator for edge detection, non-maximum suppression, double threshold hysteresis to obtain accurate edge image of photoconductive probe tip-biological sample. Among them, Canny edge detection algorithm is not easily disturbed by noise.
[0139] 4. Perform edge detection and double threshold hysteresis, pre-set T1 and T2 thresholds (T1>T2), where pixel gradient greater than T1 is a strong edge, pixel gradient less than T2 is not an edge point, and the two are called weak edge points. A strong edge must be an edge point. When the pixels near the weak edge have a strong edge, the weak edge is converted to a strong edge through double threshold hysteresis. In this way, all edge point information of the photoconductive probe tip-biological sample is obtained.
[0140] 5. See Appendix Figure 3 As shown, all point set distances in the two contours of the probe and the sample are traversed, the shortest pixel distance is retained, and based on the relationship between the calibration plate image size and the actual distance, the needle tip-sample pixel distance is converted into the actual distance and displayed on the computer data processing interface.
[0141] (2) Data processing system feedback adjustment spacing
[0142] 1. Preset safety distance a=3μm, approach distance b=10μm.
[0143] 2. When the terahertz imaging photoconductive probe scans the biological sample to obtain special information, the method of improving machine vision can measure the actual distance L in real time.
[0144] 3. Compare the distance L with the values of a and b through a computer processing system.
[0145] If L < a, it poses a potential threat to the safety of the probe tip during the scanning process. The system parameters are changed through the three-dimensional displacement stage control device to ensure the separation of the tip-sample.
[0146] If L > b, it causes unclear imaging when the probe tip obtains biological sample information during the scanning process, and the target value cannot be accurately obtained. The system parameters are changed through the three-dimensional displacement stage control device to ensure the approach of the tip-sample.
[0147] If a ≤ L ≤ b, the system can achieve the purpose of scanning and imaging to obtain sample information in a stable and safe environment.
[0148] The present invention has been described and illustrated in combination with the best embodiments, but the present invention is not limited to the disclosed embodiments, and various modifications and equivalent combinations can be made in terms of form and details.
[0149] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.
[0150] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for precise control of probe and sample based on machine vision, characterized in that: The following steps are involved: S100: Setting the position of the parallel light source and the irradiation position of the laser; S200: obtaining a grayscale image of the probe and the sample; S300: converting the grayscale image of the probe and the sample into a binary image of the probe and the sample; S400: Acquire a target edge point from the probe to the sample in a binary image of the probe and the sample to obtain a contour from the probe to the sample; S5 00: traverse the point set of the probe and the sample in the binary image of the probe and the sample, and obtain the distance from the needle tip to the sample in the binary image of the probe and the sample; S600: based on the calibration coefficient, convert the distance from the needle tip to the sample in the binary image of the probe and the sample into an actual distance value and display it; S700: Change the distance between the needle tip and the sample according to the actual distance value, and control the probe and the sample; Wherein, the S300 specifically includes: S310: obtaining a histogram of the grayscale image of the probe and the sample and normalizing it to obtain a normalized histogram; S320: segmenting the threshold of the normalized histogram based on the large-law binarization to obtain a binary image of the probe and the sample; The S320 specifically includes: Calculate the background pixel ratio ω1: Where N1 is the number of background pixels and N is the total number of pixels; Foreground pixel ratio ω2: Where N2 is the number of foreground pixels; Calculate the background average gray value μ1: Where P i is the probability that the number of pixels with gray level i accounts for the total number of pixels N, Foreground average gray value μ2: Where, M is a fixed coefficient, M = 256; Calculate the grayscale cumulative value μ in the 0-M interval: μ=u1*ω1+u2*ω2 Calculate the between-class variance g: g=ω1*(μ-μ1) 2 +ω2*(μ-μ2) 2 Finally it simplifies to: g=ω1*ω2*(μ1-μ2) 2 ; When the inter-class variance is the largest, the gray level i corresponding to the maximum value g is used as the global threshold of the image, so that the image is divided into a binary image according to the threshold.
2. According to claim 1, a method for precise control of probe and sample based on machine vision is characterized in that: The method also includes timing while obtaining the grayscale image of the probe and the sample, and when the actual distance value is obtained, the timing ends and the time taken to measure the distance is displayed.
3. The method for precise control of a probe and a sample based on machine vision according to claim 2, characterized in that: The S400 specifically includes: S410: using the Canny operator as a convolution kernel to convolve the binary image of the probe and the sample, calculating the gradient, and obtaining the target edge point from the probe to the sample; S420: converting the coarse edge between the probe and the sample in the image into a thin edge by a non-maximum suppression method; S430: Use double thresholds to connect strong and weak edges to obtain complete probe-to-sample contour information.
4. The method for precise control of a probe and a sample based on machine vision according to claim 2, characterized in that: The S500 specifically includes: S510: traverse the point set from the probe to the pixel coordinates of the sample contour, and retain the distance coordinates; S520: Calculate the distance from the tip to the sample in the binary image of the probe and the sample according to the two coordinates.
5. The method for precise control of a probe and a sample based on machine vision according to claim 1, characterized in that: The S600: S610: Acquire a calibration board checkerboard image; S620: Detecting sub-pixel coordinates of corner points in the checkerboard image by using a corner point detection method; S630: Calculate the actual distance of the sub-pixel coordinates and the coordinate pixel distance ratio; S640: Calculate the actual distance from the probe to the sample according to the actual distance and the coordinate pixel distance ratio, convert the distance from the needle tip to the sample in the binary image into the actual distance value and display it.
6. The method for precise control of a probe and a sample based on machine vision according to claim 5, characterized in that: The S700 specifically includes: S710: when the scanning probe is working, the actual distance value displayed is determined, and when the distance between the needle tip and the sample is less than the set threshold, the distance between the needle tip and the sample is controlled to be farther away; S720: When the distance between the needle tip and the sample is greater than a set threshold, the distance between the needle tip and the sample is controlled to be closer.
7. A system for a method for precise control of a probe and a sample based on machine vision according to any one of claims 1 to 6, characterized in that: include: A CCD camera (1) is used to obtain a grayscale image of the probe and the sample; A double telecentric lens (2) for correcting the distortion produced by lens shooting and connected to the CCD camera (1); Probe and fixture 3; A biological sample 4, wherein the probe and the fixing device 3 are arranged opposite to the biological sample 4; A three-dimensional moving stage control device (5) is connected to the probe and the fixing device (3), and the biological sample (4) is placed on a displacement stage (10) controlled by the three-dimensional moving stage control device (5); A computer data processing system (6) connected to the CCD camera (1) and the three-dimensional moving stage control device (5); A parallel light source device (7), used for providing light source to the CCD camera (1); A light source controller (8) connected to the parallel light source device (7); A laser (9) is used to generate laser light and irradiate the laser light onto the biological sample (4).
