Microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision

Abstract

The invention discloses a microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision. A high-precision position correction module generates a new region of interest (ROI) mapping dictionary which is completely matched with the actual position of a current chip; the dynamic droplet recognition module is used for background frame acquisition, dynamic differential detection, region-of-interest accurate positioning, droplet image enhancement and recognition, and droplet recognition; and the intelligent communication and judgment module judges each area in sequence, only when the liquid drop proportions of all the specified target areas exceed a threshold value, the algorithm returns a success signal to the upper computer, otherwise, the algorithm returns a failure signal, and then the current pipetting step is repeatedly executed until the current pipetting step is successful or the maximum retry frequency is reached. The problem of identification drift caused by chip pose (translation and rotation) change is solved, and high-reliability automatic judgment of the pipetting success rate without manual intervention is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of droplet detection, and particularly relates to a microfluidic chip droplet motion detection system, a detection method and a detection device based on machine vision. BACKGROUND

[0002] Microfluidic chip technology, also known as "chip laboratory", is a technology for manipulating microfluids (usually in the order of nanoliters to picoliters) in a micron-scale channel network. It has broad application prospects in the fields of biochemical analysis, medical diagnosis, drug screening, etc. In the automatic operation of microfluidic chips, it is often necessary to drive droplets in the chip to move, fuse, split, etc. in the predetermined chambers or channels through electrical, acoustic or mechanical means.

[0003] However, the current automatic process in this field faces a key technical bottleneck: how to non-contact, high-precision, real-time judge whether the droplet is successfully driven to the target position. The common solutions and their drawbacks are as follows: 1. Detection method based on electrical signal: detect whether the droplet passes or reaches by detecting the capacitance or resistance change of the channel or chamber. This method requires the chip to integrate complex electrode structures, increasing the preparation cost and complexity of the chip, and is easily affected by factors such as solution ion strength, environmental interference, etc., with poor reliability.

[0004] 2. Determination method based on single image recognition: take a picture after operation, and find the droplet through simple image processing (such as threshold segmentation). This method is extremely dependent on the initial, fixed shooting position. Once the chip is slightly translated or rotated during operation due to mechanical vibration, thermal expansion and contraction, etc., the pre-set detection region (ROI) will be misaligned with the actual chip structure, resulting in recognition failure or misjudgment. The operation precision of microfluidic chips is usually in microns, so any slight pose change is unacceptable.

[0005] 3. Method relying on manual interpretation: completely observed and judged by the operator under the microscope whether the operation is successful. This method is extremely inefficient and highly subjective, and cannot realize high-throughput, automated experimental process, which has become the main obstacle to its large-scale application.

[0006] Therefore, there is an urgent need in the art for a fully automatic droplet tracking and determination technology that can adapt to chip position changes, has strong robustness, high precision and does not require modification of the chip structure, to promote the real landing of the microfluidic chip automation platform. SUMMARY

[0007] In view of the deficiencies of the prior art, a microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision and a computer medium are provided. The purpose is to solve the problem of recognition drift caused by changes in chip pose (translation and rotation) and to realize automatic judgment of pipetting success rate without manual intervention and with high reliability.

[0008] To achieve the foregoing purposes, the technical solutions adopted by the present application include: a microfluidic chip droplet motion detection system based on machine vision, comprising a vision imaging module, an image processing module, an upper computer control module and an upper computer, the vision program module is preferably an industrial camera, and further comprising: A high-precision position correction module is used to locate the marker points of the collected chip image through the image processing module, then the current position is calculated according to the center coordinates of the marker point positioning, then the translation and rotation are performed, and then the ROI mapping is updated to generate a new ROI mapping dictionary that completely matches the actual position of the current chip; A dynamic droplet recognition module is used to perform background frame acquisition, dynamic difference detection, accurate positioning of the region of interest, droplet image enhancement and recognition after completing the chip image position correction, and extract the connected white region, which is recognized as a droplet; An intelligent communication and judgment module is used to parse a list of one or more target numbers, find the corresponding image region from the ROI mapping dictionary according to the target number, and then judge each region in turn. Only when the droplet proportion of all specified target regions exceeds the threshold value, the algorithm returns a success signal to the upper computer, otherwise a failure signal is returned, and then the current pipetting step is repeatedly executed until success or the maximum number of retries is reached. The intelligent communication and judgment module and the upper computer control module are in closed-loop communication.

[0009] Further, the high-precision position correction module specifically includes: Image acquisition: after moving the chip to the center of the field of view of the vision imaging module (such as an industrial camera), a clear image containing specific marker points (such as circles, crosses) on the chip is taken by an industrial camera with an exposure time of 10000 microseconds; Marker point positioning: the possible area of the marker point is quickly and roughly positioned by a circle detection algorithm, at least two marker points (such as the upper left marker and the upper right marker) on the chip image are accurately positioned, and then a pre-prepared marker point template image is used to perform template matching in each candidate area. The final sub-pixel level center point coordinates of the two marker points are determined by calculating the similarity score. The template image calculates the similarity by the normalized square difference matching method, and the formula is: Wherein, T is a template image, I is an image to be searched, (x, y) is a search position, the value of R(x, y) is closer to 0, indicating a higher matching degree, by finding the minimum point of R(x, y), the center coordinates Ptl(xtl, ytl) and Ptr(xtr, ytr) of the marker points can be accurately positioned at a sub-pixel level; Pose parameter calculation, according to the center coordinates of the two marker points, the current pose of the computer chip is calculated, including: Translation (dx, dy), assuming that the theoretical design coordinates of the upper left marker point are Ptl0(xtl0, ytl0), the translation offset is calculated as: Rotation angle θ calculation, the vector V=Ptr-Ptl formed by the two marker points, and the included angle between the theoretical horizontal vector V0=(1, 0) is the rotation angle, which is solved by vector cross product and dot product: ROI mapping update, the (dx, dy, θ) transformation matrix obtained is subjected to affine transformation with the pre-set theoretical region coordinates, so as to generate a new ROI mapping dictionary in the image coordinate system which completely matches the actual position of the current chip.

[0010] Further, the dynamic droplet recognition module specifically includes: Background frame acquisition, a clear image of the chip is shot as the first frame by improving the signal-to-noise ratio of the industrial camera At this time, there should be no droplet in the chip or the droplet is in the initial position; Dynamic differential detection: in the subsequent process, after the host computer control module controls the execution of a droplet moving operation, the host computer control module controls the industrial camera to shoot a current frame under the same parameters Image difference operation is performed between the current frame and the stored first frame background image to obtain a difference value image Accurate positioning of the region of interest: according to the generated ROI mapping dictionary, the target chamber region to be detected is accurately analyzed from the difference value image; Droplet image enhancement and recognition: each small region analyzed is separately subjected to image processing; first, image binarization is performed to separate the potential droplet region from the background, and then morphological processing is used to optimize the binary image Finally, the connected white regions are extracted from the processed binary image by using a contour recognition or convex hull detection algorithm, and these regions are recognized as droplets.

[0011] ​​​Further, the intelligent communication and decision module specifically comprises: Instruction analysis: the host computer sends a target string instruction to the image processing module, and the image processing module analyzes the string to obtain a list of one or more target numbers; Region traversal and decision: the image processing module finds the corresponding image region from the ROI mapping dictionary according to the target number, and then sequentially determines each region; the determination standard is: calculating the proportion of the pixel points identified as droplets in the total pixel points in the region : Wherein, Nforeground is the total number of pixels identified as foreground in the region, Ntotal is the total number of pixels in the region, and the determination standard is: Result feedback and process control: only when the droplet proportion of all specified target regions exceeds the threshold value, the image processing module returns a success signal to the host computer, and the host computer executes the next step of the process after receiving the success signal. If any target region is determined to have no droplets or the droplets have not been successfully reached, the image processing module returns a failure signal, and the host computer will repeat the current pipetting step until success or the maximum number of retries is reached.

[0012] A microfluidic chip droplet motion detection method based on machine vision, which adopts a microfluidic chip droplet motion detection system based on machine vision for detection, comprising the following steps: Step one, get the chip image, take a correction image I_calib with an exposure time of 10000 microseconds, and cut it, then mark at least two places on the chip; Read the pre-cut template images T_tl and T_tr of the two mark points, perform Gaussian blur noise reduction on the image I_calib, then use Hough circle transformation for circle detection to roughly find all possible circular regions, greatly reducing the search range of template matching; In each circular ROI, use the normalized square difference matching method for template matching; Traverse the T_tl and T_tr image templates to find the position with the highest matching degree, and achieve sub-pixel positioning through quadratic interpolation, finally get the accurate center coordinates P_tl(x1, y1) and P_tr(x2, y2) of the two mark points; Calculate the transformation parameters: let the theoretical coordinates of one mark point in the chip preset image be P_tl0(100, 100), and the theoretical coordinates of the other mark point be P_tr0(500, 100), Translation: dx = x1 - 100, dy = y1 - 100, Rotation angle: θ = arctan2((y2 - y1), (x2 - x1)); Step two, get the background frame, after position correction, the host computer control module controls the industrial camera to set the exposure time to 50000 microseconds, takes the first frame image I_background and stores it in the memory at this time, the chip chambers are all empty; Step three, move the target droplet from the current chamber to the target chamber, the industrial camera takes a current image I_current again, Then calculate the difference image: I_current and I_background are operated by absolute difference, and the difference image I_diff = cv2.absdiff(I_current, I_background) is obtained. Get the corresponding polygon region from the roi_dict in the target ROI, create a mask on the difference image I_diff according to the polygon region, and cut out the image block I_roi of the region for subsequent processing; perform Gaussian blur on I_roi, then automatically calculate the threshold T using the Otsu method and perform binaryzation to obtain I_binary; perform a corrosion operation on I_binary first to remove small white noise points; then perform an inflation operation to connect the possible broken areas inside the droplet to form a complete connected domain. Step four, use the contour finding algorithm to find all contours in I_binary, calculate the area of each contour, filter out possible noise, calculate the total pixel ratio R_area = the sum of all contour areas / the total number of pixels in the ROI region of the droplet in the ROI region, set the success threshold T_success = 0.3, if R_area >= 0.3, then determine that the liquid transfer in the corresponding chamber is successful. Step five, if the determination is successful, send a successful signal, and the host computer receives the success signal and performs the next step of operation; if the determination fails, send a failure signal, and the host computer receives the failure signal and then re-executes the operation of driving the target droplet to move to the corresponding chamber, and then triggers the software to take pictures, identify, and determine again. This cycle is repeated at most 3 times, if 3 times fail, report an exception, and manually intervene to handle.

[0013] A microfluidic chip droplet motion detection device based on machine vision, comprising a vision imaging module, an image processing module, an upper computer control module and an upper computer, further comprising: A high-precision position correction module is used for locating the mark points of the collected chip image, then the current position is calculated according to the center coordinates of the mark point positioning, then translation and rotation are performed, and then the ROI mapping is updated to generate a new ROI mapping dictionary that is completely matched with the actual position of the current chip; A dynamic droplet recognition module is used for acquiring a background frame, performing dynamic differential detection, accurately positioning a region of interest, strengthening and recognizing a droplet image after completing the position correction of the chip image, and extracting a connected white region, which is recognized as a droplet; An intelligent communication and judgment module is used for closed-loop communication between the intelligent communication and judgment module and the host computer control module; a list of one or more target numbers is parsed, the corresponding image region is found from the ROI mapping dictionary according to the target number, and then each region is sequentially judged. Only when the droplet occupancy of all specified target regions exceeds the threshold value, the algorithm returns a success signal to the host computer, otherwise a failure signal is returned, and then the current pipetting step is repeatedly executed until success or the maximum number of retries is reached.

[0014] Further, the high-precision position correction module specifically includes: Image acquisition: After moving the chip to the center of the field of view of the vision imaging module, an industrial camera with an exposure time of 10000 microseconds is used to shoot a clear image containing specific mark points on the chip; Mark point positioning: a circle detection algorithm is used to quickly coarsely position the possible region of the mark point, and then a pre-prepared mark point template image is used to perform template matching in each candidate region. The sub-pixel level center point coordinates of the two mark points are finally determined by calculating the similarity score. The template image uses a normalized square difference matching method to calculate the similarity, and the formula is: Wherein, T is the template image, I is the image to be searched, (x, y) is the search position, and the value of R(x, y) is closer to 0, indicating a higher matching degree. By finding the minimum point of R(x, y), the center coordinates Ptl(xtl, ytl) and Ptr(xtr, ytr) of the mark points can be accurately positioned at the sub-pixel level; Pose parameter calculation: the current pose of the computer chip is calculated according to the center coordinates of the two mark points; Translation amount (dx, dy): assuming that the theoretical design coordinates of the upper left mark point are Ptl0(xtl0, ytl0), the translation offset is calculated as: Rotation angle θ calculation: the vector V=Ptr−Ptl formed by the two mark points is the rotation angle with the theoretical horizontal vector V0=(1, 0), which is solved by vector cross product and dot product: ROI mapping update, the obtained (dx, dy, theta) transformation matrix is subjected to affine transformation with the pre-set theoretical region coordinates, so as to generate a new ROI mapping dictionary in the image coordinate system which completely matches the actual position of the current chip.

[0015] Further, the dynamic droplet recognition module specifically comprises: Background frame acquisition, a clear image of the chip is shot as the first frame by improving the signal-to-noise ratio of the industrial camera At this time, there is no droplet or the droplet is in the initial position in the chip; Dynamic differential detection, in the subsequent process, after the host computer control executes a droplet moving operation, the host computer control module controls the industrial camera to shoot a current frame under the same parameters Image difference operation is performed between the current frame and the stored first frame background image, and a difference value image is obtained : ; Accurate positioning of the region of interest, according to the generated ROI mapping dictionary, the target chamber region to be detected is accurately analyzed from the difference value image; Droplet image enhancement and recognition, each small region analyzed is separately subjected to image processing; first, image binarization is performed to separate the potential droplet region from the background, and then morphological processing is adopted to optimize the binary image : Finally, the connected white regions are extracted from the processed binary image by using a contour recognition or convex hull detection algorithm, and these regions are identified as droplets.

[0016] Further, the intelligent communication and judgment module specifically comprises: Instruction analysis, the host computer sends a target string instruction to the image processing module, and the image processing module analyzes the string to obtain a list of one or more target numbers; Region traversal and judgment, the image processing module finds the corresponding image region from the ROI mapping dictionary according to the target number, and then sequentially judges each region; the judgment standard is that the proportion of the pixel points identified as droplets in the total pixel points in the region is calculated : Wherein, Nforeground is the total number of pixels identified as foreground in the region, Ntotal is the total number of pixels in the region, and the judgment standard is: The result feedback and the process control, only when all the specified target area droplet proportion exceeds the threshold, the image processing module returns the success signal to the host computer, the host computer receives the success signal, and then executes the next step of the process, if any one target area is determined as no droplet or droplet is not successfully arrived, the image processing module returns the failure signal, the host computer receives the failure signal, and then repeats the current pipetting step until success or reaches the maximum number of retries.

[0017] Compared with the prior art, the advantages of the present application include: (1) The microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision provided by the present application have high precision and high robustness: the innovative "circle detection + template matching" two-stage positioning method, combined with affine transformation, can automatically compensate for the translation and rotation of the chip, fundamentally solving the recognition drift problem caused by mechanical errors, making the system insensitive to the initial placement position of the chip, and having strong anti-interference ability.

[0018] (2) The microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision provided by the present application have high sensitivity and high reliability: the "background difference" strategy is adopted instead of simply relying on single-frame image analysis, which can extremely sensitively capture the tiny droplet motion changes, effectively suppress the interference of complex background, and greatly improve the signal-to-noise ratio and accuracy of droplet recognition.

[0019] (3) The microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision provided by the present application have intelligent closed-loop control: through dictionary mapping and area proportion determination, intelligent decision-making linked with host computer instructions is realized. The determination logic is scientific (based on area proportion), the communication protocol is simple and efficient, and a complete "perception-decision-making-execution-feedback" automatic closed loop is formed.

[0020] (4) The microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision provided by the present application have strong universality and low cost: the method is completely realized based on machine vision software algorithm, without any special structural modification (such as machining electrodes) to the microfluidic chip itself, and is suitable for most microfluidic chips made of transparent materials, which significantly reduces the hardware cost and complexity of the system application. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments or prior art of the present application, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0022] Figure 1 This is a flowchart of a microfluidic chip droplet motion detection method based on machine vision in this invention; Figure 2 This is a schematic diagram of the high-precision position correction module after processing of the microfluidic chip droplet motion detection system, detection method, and detection equipment based on machine vision in this invention; Figure 3 The image processing flow diagram of the droplet recognition module of the microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision in this invention is shown. Figure 4 This is a binarized image of the droplet recognition module in the machine vision-based microfluidic chip droplet motion detection system, detection method, and detection equipment of the present invention. Figure 5 This is a difference diagram showing the image processing flow of the droplet recognition module in the machine vision-based microfluidic chip droplet motion detection system, detection method, and detection equipment of the present invention. Detailed Implementation

[0023] In view of the shortcomings of the prior art, the inventors of this invention, through long-term research and extensive practice, have proposed the technical solution of this invention. The technical solution, its implementation process, and principles will be further explained below with reference to the accompanying drawings and specific implementation examples in the embodiments of this application.

[0024] It should be noted that the embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, the present invention covers any substitutions, modifications, equivalent methods and solutions made on the spirit, principles and scope of the present invention as defined by the claims. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In the description of this application, the terms "first," "second," "third," and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "a" or "one," and similar words, do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including," and similar words, mean that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including," and their equivalents, but do not exclude other elements or objects. The terms "connected" or "linked," and similar words, are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0026] In the description of this application, the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used solely for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, when using positional terms such as "both sides," "outer side," and "upper and lower," it should be understood that they are used only for ease of understanding and description, taking into account that the structure may be oriented to other positions.

[0027] In the description of this application, unless otherwise expressly specified and limited, the technical or scientific terms used shall have the ordinary meaning understood by a person with ordinary skills in the art to which this application pertains. Terms such as “installation,” “connection,” and “joining” shall be interpreted broadly, for example, as fixed connection, detachable connection, mating connection, or integral connection. For a person skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0028] The present invention aims to introduce and explain the structural composition and the cooperation relationship between the components of a machine vision-based microfluidic chip droplet motion detection system, detection method and detection device. Unless otherwise specified, the size, material and manufacturing process of each component in the machine vision-based microfluidic chip droplet motion detection system, detection method and detection device in the present invention can be selected according to specific circumstances, and no special limitation or explanation is given here.

[0029] Furthermore, to provide the public with a better understanding of the present invention, certain specific details are described in detail in the following description of the invention. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0030] Example 1 Please see Figures 1-5A machine vision-based microfluidic chip droplet motion detection system is disclosed, comprising a vision imaging module, an image processing module, a host computer control module, and a host computer. The vision program module is preferably an industrial camera. The hardware system mainly includes: an industrial camera (such as a CMOS or CCD camera) fixed to a mechanical structure, an LED surface light source providing uniform backlighting for the chip, a platform for fixing the microfluidic chip, and a host computer that executes the image processing algorithm and communicates with the host PLC (Programmable Logic Controller) or industrial control computer. The software of this invention is an algorithm program developed based on languages ​​such as C++ and C#, and integrating machine vision libraries such as OpenCV and Halcon. It includes: The high-precision position correction module uses the image processing module to locate the marked points in the acquired chip image. Then, it calculates the current position based on the center coordinates of the marked points, performs translation and rotation, and then updates the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module parses a list of one or more target numbers from the instruction. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it judges each region in turn. Only when the droplet percentage of all specified target regions exceeds the threshold will the algorithm return a success signal to the host computer. Otherwise, it returns a failure signal and repeats the current pipetting step until it succeeds or reaches the maximum number of retries. This establishes closed-loop communication between the intelligent communication and judgment module and the host computer control module.

[0031] In this invention, the high-precision position correction module specifically includes: Image acquisition involves moving the chip to the center of the field of view of the vision imaging module (such as an industrial camera) and then using an industrial camera with an exposure time of 10,000 microseconds to capture a clear image (such as a circle or a cross) containing specific markers on the chip. Marker localization uses a circle detection algorithm to quickly and coarsely locate the possible regions of markers, and then precisely locates at least two markers on the chip image (such as the upper left and upper right markers). Next, using a pre-prepared marker template image, template matching is performed within each candidate region. By calculating a similarity score, the sub-pixel-level center coordinates of the two markers are finally determined. The similarity of the template images is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation, based on the center coordinates of two marker points, determines the current pose of the computer chip, including: The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr − Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

[0032] In this invention, the dynamic droplet recognition module specifically includes: Background frame acquisition is used to capture a clear image of the chip in an industrial camera, improving the signal-to-noise ratio, as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection: In subsequent processes, each time the host computer controls a droplet movement operation, the host computer control module controls the industrial camera to capture a frame of the current position under the same parameters. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ; Precise location of region of interest: Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map; Droplet Image Enhancement and Recognition: Each small region is processed individually. First, image binarization is performed to separate the potential droplet region from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

[0033] In this invention, the intelligent communication and judgment module specifically includes: Command parsing: The host computer sends a target string command to the image processing module. The image processing module parses the string and obtains a list of one or more target numbers. Region Traversal and Judgment: The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then judges each region sequentially. The judgment criterion is: calculating the proportion of pixels in the region that are identified as droplets out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: Results Feedback and Process Control: The image processing module returns a success signal to the host computer only when the droplet percentage of all specified target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or the maximum number of retries is reached.

[0034] A machine vision-based microfluidic chip droplet motion detection device, employing a machine vision-based microfluidic chip droplet motion detection system and detection method, includes a vision imaging module, an image processing module, a host computer control module, and a host computer, and further includes: The high-precision position correction module is used to locate the marked points in the acquired chip image, then calculate the current position based on the center coordinates of the marked points, then translate and rotate, and then update the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module establishes closed-loop communication between itself and the host computer control module. It parses a list of one or more target numbers from the instruction parsing. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it sequentially judges each region. Only when the droplet percentage of all specified target regions exceeds the threshold does the algorithm return a success signal to the host computer; otherwise, it returns a failure signal. The current pipetting step is then repeated until success or the maximum number of retries is reached.

[0035] In this invention, the high-precision position correction module specifically includes: Image acquisition involves moving the chip to the center of the vision imaging module's field of view and then using an industrial camera with a 10,000-microsecond exposure time to capture a clear image containing specific marker points on the chip. Marker point localization involves quickly and coarsely locating the possible regions of marker points using a circle detection algorithm. Then, using pre-prepared marker point template images, template matching is performed within each candidate region. By calculating similarity scores, the sub-pixel-level center point coordinates of two marker points are finally determined. The similarity of the template images is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation: The current pose of the computer chip is determined based on the center coordinates of the two marker points. The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr − Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

[0036] In this invention, the dynamic droplet recognition module specifically includes: Background frame acquisition is used to capture a clear image of the chip in an industrial camera, improving the signal-to-noise ratio, as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection is used in subsequent processes. Each time the host computer executes a droplet movement operation, the host computer control module controls the industrial camera to capture a new frame under the same parameters. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ; Precise location of the region of interest (ROI): Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map. Droplet image enhancement and recognition involves individual image processing for each small region extracted. First, image binarization is performed to separate potential droplet regions from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

[0037] In this invention, the intelligent communication and judgment module specifically includes: Instruction parsing: The host computer sends a target string instruction to the image processing module, which parses the string to obtain a list of one or more target numbers; Region traversal and determination: The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then determines each region sequentially. The determination criterion is: calculating the proportion of pixels identified as droplets within the region out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: In terms of result feedback and process control, the image processing module only returns a success signal to the host computer when the droplet percentage in all designated target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or reaches the maximum number of retries.

[0038] A machine vision-based method for detecting droplet motion in microfluidic chips, employing a machine vision-based microfluidic chip droplet motion detection system, includes the following steps: Initialization and position correction of a microfluidic chip droplet motion detection system based on machine vision (hereinafter referred to as the system). Step 1, Hardware Connection and Startup: Start the computer, camera, light source, and host computer control module. After the system starts, initialize the camera and establish a communication connection with the host computer.

[0039] Capturing the Corrected Image: The system sends a ready signal to the host computer via the communication interface. The industrial camera is set to an exposure time of 10,000 microseconds. After the host computer control module controls the robotic arm or moves the chip to the center of the industrial camera's field of view, a clear corrected image I_calib containing specific marker points on the chip is captured and cropped to remove excess blank space. These marker points are usually high-contrast graphics (such as circles or crosses) pre-made during chip design.

[0040] Step 2, Marker Identification and Pose Calculation: A combined algorithm of "circle detection + template matching" is used to accurately locate at least two markers on the chip, such as the upper left and upper right markers. Specifically, the system reads the pre-cropped template images T_tl and T_tr for the upper left and upper right markers. Gaussian blur is applied to the image I_calib for noise reduction, and then Hough circle transform is used for circle detection to roughly identify all possible circular regions, significantly narrowing the search area (ROI) for template matching. Within each circular ROI, template matching is performed using the normalized squared difference (TM_SQDIFF_NORMED) method. The T_tl and T_tr image templates are traversed to find the position with the highest matching degree (i.e., the smallest R(x,y) value), and sub-pixel localization is achieved through quadratic interpolation. Finally, the precise center coordinates P_tl(x1, y1) and P_tr(x2, y2) of the two markers are obtained.

[0041] Calculate the transformation parameters: Let the theoretical coordinates of the upper left marker in the chip design diagram be P_tl0(100, 100), and the theoretical coordinates of the upper right marker be P_tr0(500, 100). Specifically: Translation amount: dx = x1 - 100, dy = y1 - 100.

[0042] Rotation angle: θ = arctan2((y2 - y1), (x2 - x1)). (Theoretically, P_tr0 - P_tl0 = (400, 0), so the theoretical angle is 0).

[0043] Step 3: Acquire the background frame. After position correction is completed, the software controls the camera to set the exposure time to 50,000 microseconds (to obtain a brighter image), captures the first frame image I_background, and stores it in memory. At this time, all chambers of the chip are empty.

[0044] Step four, droplet movement and dynamic identification cycle: The host computer controls the actuator (such as high-voltage electrode) to apply a driving signal to the target droplet, attempting to move it from the current chamber to the target chamber.

[0045] Capture the current frame: After the driving operation is completed, the software controls the camera to capture another current image I_current.

[0046] Calculate the difference image: Perform the absolute difference operation between I_current and I_background to obtain the difference image I_diff = cv2.absdiff(I_current, I_background).

[0047] Step 5: Extract and process the target ROI: Assume the instruction string sent by the host computer is "1-2", requiring the droplet to be moved to chamber "1-2". The software obtains the polygonal region corresponding to the number "1-2" from roi_dict. On the difference image I_diff, a mask is created based on this polygonal region, and the image block I_roi of this region is extracted for subsequent processing. Gaussian blur is applied to I_roi, and then the threshold T is automatically calculated using Otsu's method, followed by binarization to obtain I_binary. I_binary is first subjected to an erosion operation (using 3x3 rectangular structuring elements) to remove tiny white noise points; then a dilation operation (using 5x5 rectangular structuring elements) is performed to connect potentially broken regions inside the droplet, forming a complete connected component.

[0048] Contour Recognition and Judgment: Use a contour finding algorithm (such as cv2.findContours) to find all contours in I_binary. Calculate the area (number of pixels) of each contour. A minimum area threshold (e.g., 50 pixels) is typically set to filter out any residual noise. Calculate the total pixel percentage of the droplet within the ROI region, R_area = sum of all contour areas / total number of pixels in the ROI region. Set a success threshold T_success = 0.3 (i.e., 30% of the area is occupied by the droplet). If R_area >= 0.3, then the pipetting in chamber "1-2" is considered successful.

[0049] Step 4: Communication and Feedback After the software completes the judgment, it sends the result to the host computer via an interface. In this example, if the judgment is successful, the string "1" is sent. Upon receiving "1", the host computer proceeds to the next step in the process. If the judgment fails (R_area < 0.3), the string "0" is sent. Upon receiving "0", the host computer will re-execute the operation of driving the droplet to move to chambers 1-2, and then trigger the software to take a picture, identify, and judge again. This loop repeats a maximum of 3 times. If all 3 attempts fail, an exception is reported, and manual intervention is required.

[0050] Understandable, This approach provides a high-precision method for correcting the initial pose of a chip, enabling rapid calculation of the chip's offset and rotation angle in the XY plane.

[0051] A dynamic droplet detection method based on image difference is provided, which can effectively highlight moving targets (droplets) and suppress complex static background interference.

[0052] A smart communication and judgment mechanism that works in collaboration with a host computer is provided. It can accurately analyze the target area according to instructions, make scientific judgments based on the proportion of droplets in the area, and finally feed the results back to the host computer to control the process.

[0053] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. It should not be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A machine vision-based droplet motion detection system for a microfluidic chip, characterized in that: Comprise: High-precision position correction module, collect chip image and mark point positioning on chip image, calculate the current position according to the center coordinates of mark point positioning, then translation and rotation, and then ROI mapping update, generate new ROI mapping dictionary which completely matches the actual position of the current chip; Dynamic droplet recognition module, after completing chip image position correction, background frame acquisition, dynamic difference detection, accurate positioning of the region of interest, droplet image enhancement and recognition, extract the connected white area, which is identified as droplet; Intelligent communication and judgment module, command analysis one or more target number list, according to the target number, find the corresponding image area from the ROI mapping dictionary, then judge each area in turn, only when the droplet proportion of all specified target area exceeds the threshold, the algorithm returns success signal to host computer, otherwise, return failure signal, then repeat the current pipetting step until success or reach the maximum retry times. 2.The machine vision-based microfluidic chip droplet motion detection system of claim 1, wherein: The high-precision position correction module specifically comprises: Image acquisition, industrial camera with 10000 microsecond exposure time is used to shoot clear image containing specific mark points on chip, which constitutes template image; Mark point positioning, through circle detection algorithm, quickly coarsely locate the possible area of mark point, accurately locate at least two mark points on chip image, then use template image to perform template matching in each candidate area, determine the sub-pixel level center point coordinates of two mark points by calculating similarity score, template image uses normalized square difference matching method to calculate similarity, the formula is: Wherein, T is template image, I is image to be searched, (x, y) is search position, the value of R(x, y) is closer to 0, indicating higher matching degree, by finding the minimum point of R(x, y), the center coordinates Ptl(xtl, ytl) and Ptr(xtr, ytr) of mark point can be accurately positioned at sub-pixel level; Pose parameter calculation, according to the center coordinates of two mark points, calculate the current pose of computer chip, including: The translation amount (dx, dy) is calculated as follows, where the theoretical design coordinates of the upper left mark point are Ptl0(xtl0, ytl0): Rotation angle θ calculation, the vector V=Ptr−Ptl formed by two mark points, and the included angle between the theoretical horizontal vector V0=(1, 0) is the rotation angle, which is solved by vector cross product and dot product: ROI mapping update, the obtained (dx, dy, θ) transformation matrix is subjected to affine transformation with the pre-set theoretical region coordinates, thereby generating new ROI mapping dictionary which completely matches the actual position of the current chip in image coordinate system. 3.The microfluidic chip droplet motion detection system based on machine vision according to claim 1, characterized in that: The dynamic droplet recognition module specifically comprises: Background frame acquisition, improve the signal-to-noise ratio of the industrial camera to shoot a clear image of the chip as the first frame At this time, there is no droplet or the droplet is in the initial position in the chip. Dynamic differential detection, in the subsequent process, after each droplet movement operation is executed, an industrial camera is used to shoot a current frame under the same parameters Image difference operation is performed between the current frame and the stored first frame background image to obtain a difference value image : ； Accurate positioning of the region of interest, according to the generated ROI mapping dictionary, accurately analyze the target chamber area to be detected from the difference image; Droplet image enhancement and recognition, the image processing is performed separately for each target chamber region resolved; first, image binarization is performed to separate potential droplet regions from the background, followed by morphological processing to optimize the binary image : Finally, use contour recognition or convex hull detection algorithm to extract the connected white area from the processed binary image, which is identified as droplet. 4.The machine vision-based microfluidic chip droplet motion detection system of claim 1, wherein: The intelligent communication and judgment module specifically comprises: host computer, image processing module, host computer control module, According to one string command, get one or more target number list; The image processing module finds the corresponding image region from the ROI mapping dictionary according to the target number, and then determines each region in turn; the determination criterion is that the proportion of the pixel points identified as liquid drops in the total pixel points in the region is calculated : Wherein, Nforeground is the total number of pixels in the region identified as foreground, Ntotal is the total number of pixels in the region, and the determination criterion is: Only when the droplet proportion of all specified target regions exceeds the threshold value, the image processing module returns a success signal to the upper computer. After the upper computer receives the success signal, it executes the next step of the process. If any target region is determined to have no droplet or the droplet has not arrived successfully, the image processing module returns a failure signal. After the upper computer receives the failure signal, it will repeat the current pipetting step until it is successful or reaches the maximum number of retries.

5. A method for detecting droplet motion of a microfluidic chip based on machine vision, characterized in that: The method comprises the following steps: Step one, obtain the chip image, take a correction image I_calib with an exposure time of 10000 microseconds, and then crop it. At least two marks are made on the chip; Read the template images T_tl and T_tr, perform Gaussian blur denoising on the image I_calib, and then use Hough circle transformation for circle detection to roughly find all possible circular regions, which greatly reduces the search range of template matching; In each circular ROI, use the normalized square difference matching method for template matching; Iterate through the T_tl template image and the T_tr template image to find the position with the highest matching degree, and use the quadratic interpolation method to achieve sub-pixel positioning. Finally, the accurate center coordinates of the two mark points P_tl(x1, y1) and P_tr(x2, y2) are obtained; Let the theoretical coordinates of one of the mark points of the template image be P_tl0(100, 100), and the theoretical coordinates of the other mark point be P_tr0(500, 100), Translation: dx = x1 - 100, dy = y1 - 100, Rotation angle: θ = arctan2((y2 - y1), (x2 - x1)); Step two, obtain the background frame, after completing the position correction, set the exposure time to 50000 microseconds, take the first frame image I_background and store it. At this time, all chambers of the chip are empty; Step three, move the target droplet from the current chamber to the target chamber, and take a current image I_current again, Perform absolute difference operation on I_current and I_background to obtain the difference image I_diff = cv2.absdiff(I_current, I_background); Get the corresponding polygon region from the roi_dict in the target ROI. Create a mask according to the polygon region on the difference image I_diff, and cut out the image block I_roi in the region for subsequent processing. Perform Gaussian blur on I_roi, then automatically calculate the threshold T using the Otsu method, and perform binaryzation to obtain I_binary. First, perform an erosion operation on I_binary to remove small white noise points. Then, perform an inflation operation to connect the possible broken areas inside the droplet to form a complete connected domain. Step four, find all contours in I_binary using contour finding algorithm, calculate the area of each contour, filter out possible residual noise, calculate the total pixel ratio of droplets in the ROI area R_area = the sum of the areas of all contours / the total number of pixels in the ROI area, set the success threshold T_success = 0.3, if R_area >= 0.3, determine that the corresponding chamber pipetting is successful; Step five, if it is determined to be successful, a success signal is sent, and the host computer receives the success signal and then performs the next step of the process; if it is determined to be unsuccessful, a failure signal is sent, and the host computer receives the failure signal and then re-executes the operation of driving the target droplet to move to the corresponding chamber, and then triggers the software to take a picture, identify, and determine again, this cycle is repeated a maximum of 3 times, if it fails 3 times, an exception is reported, and manual intervention is required. 6.A machine vision based microfluidic chip droplet motion detection device, characterized in that: Also includes: A high-precision position correction module for locating the marker points of the collected chip image, then calculating the current position according to the center coordinates of the marker point positioning, then translating and rotating, and then updating the ROI mapping to generate a new ROI mapping dictionary that completely matches the actual position of the current chip; A dynamic droplet identification module that, after completing the position correction of the chip image, performs background frame acquisition, dynamic difference detection, precise positioning of the region of interest, droplet image enhancement and identification, and extracts the connected white region, which is identified as a droplet; An intelligent communication and determination module that analyzes one or more lists of target numbers, finds the corresponding image region from the ROI mapping dictionary according to the target number, and then determines each region in turn. Only when the droplet ratio of all specified target regions exceeds the threshold value, the algorithm returns a success signal to the host computer, otherwise a failure signal is returned, and then the current pipetting step is repeated until success or the maximum number of retries is reached.

7. The machine vision based microfluidic chip droplet motion detection device of claim 6, wherein: The high-precision position correction module specifically includes: Image acquisition: an industrial camera with an exposure time of 10000 microseconds is used to capture a clear image containing specific marker points on the chip, forming a template image; Marker point positioning: a circle detection algorithm is used to quickly and roughly locate the possible area of the marker point, and at least two marker points on the chip image are precisely positioned. Then, template matching is performed in each candidate area using the template image, and the sub-pixel level center point coordinates of the two marker points are finally determined by calculating the similarity score. The template image calculates the similarity using the normalized square difference matching method, and the formula is: Where T is the template image, I is the image to be searched, (x, y) is the search position, and the value of R(x, y) is closer to 0, indicating a higher matching degree. By finding the minimum point of R(x, y), the center coordinates Ptl(xtl, ytl) and Ptr(xtr, ytr) of the marker points can be precisely located at the sub-pixel level; Pose parameter calculation: the current pose of the computer chip is calculated based on the center coordinates of the two marker points, including: The translation amount (dx, dy), assuming that the theoretical design coordinates of the upper left mark point are Ptl0(xtl0, ytl0), is calculated as: The rotation angle θ is calculated by the vector V = Ptr-Ptl formed by the two marker points, and the included angle between the vector V and the theoretical horizontal vector V0=(1,0) is the rotation angle, which is solved by vector cross product and dot product: The ROI mapping is updated, and the obtained (dx, dy, θ) transformation matrix is subjected to affine transformation with the pre-set theoretical region coordinates, so as to generate a new ROI mapping dictionary in the image coordinate system which is completely matched with the actual position of the current chip.

8. The machine vision based microfluidic chip droplet motion detection device of claim 6, wherein: The dynamic droplet recognition module specifically includes: Background frame acquisition, improve the signal-to-noise ratio of the industrial camera to shoot a clear image of the chip as the first frame At this time, there should be no droplets in the chip or the droplets are in the initial position; Dynamic differential detection, in the subsequent process, after each droplet movement operation is executed, an industrial camera is used to shoot a current frame under the same parameters Image difference operation is performed between the current frame and the stored first frame background image to obtain a difference value image : ； The ROI is accurately positioned, and the generated ROI mapping dictionary is used to accurately analyze the target chamber region to be detected from the difference image; Droplet image enhancement and recognition, the image processing is performed separately for each target chamber region resolved; first, image binarization is performed to separate potential droplet regions from the background, followed by morphological processing to optimize the binary image : Finally, the contour recognition or convex hull detection algorithm is used to extract the connected white regions from the processed binary image, and these regions are recognized as droplets.

9. The machine vision based microfluidic chip droplet motion detection device of claim 6, wherein: The intelligent communication and judgment module specifically includes: an upper computer, an image processing module, an upper computer control module, The upper computer sends a target string instruction to the image processing module, and the image processing module analyzes the string to obtain a list of one or more target numbers; The image processing module finds the corresponding image region from the ROI mapping dictionary according to the target number, and then determines each region in turn; the determination criterion is that the proportion of the pixel points identified as liquid drops in the total pixel points in the region is calculated : Wherein, Nforeground is the total number of pixels identified as foreground in the region, Ntotal is the total number of pixels in the region, and the judgment standard is: Only when the droplet proportion of all specified target regions exceeds the threshold value, the image processing module returns a success signal to the upper computer, and the upper computer receives the success signal and executes the next step of the process. If any target region is judged as no droplet or the droplet does not arrive successfully, the image processing module returns a failure signal, and the upper computer receives the failure signal and repeats the current pipetting step until success or reaches the maximum number of retries.

Images