A microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision

Abstract

The application discloses a microfluidic chip droplet motion detection system, a detection method and a detection device based on machine vision, a high-precision position correction module generates a new region of interest (ROI) mapping dictionary that is completely matched with the actual position of the current chip; a dynamic droplet identification module performs background frame acquisition, dynamic differential detection, accurate positioning of the region of interest, droplet image strengthening and identification, and identifies as a droplet; an intelligent communication and judgment module sequentially judges each region, and only when the droplet proportion of all specified target regions exceeds the threshold value, the algorithm returns a success signal to the upper computer, otherwise a failure signal is returned, and then the current pipetting step is repeatedly executed until success or the maximum number of retries is reached. The application solves the problem of recognition drift caused by changes in the chip pose (translation and rotation), and realizes automatic judgment of the pipetting success rate without manual intervention and with high reliability.

Description

Technical Field

[0001] This invention belongs to the field of droplet detection technology, specifically, it relates to a microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision. Background Technology

[0002] Microfluidic chip technology, also known as "lab-on-a-chip," is a technique for manipulating tiny fluids (typically in the nanoliter to picoliter range) within a network of channels at the micrometer scale. It has broad application prospects in fields such as biochemical analysis, medical diagnostics, and drug screening. In the automated operation of microfluidic chips, it is often necessary to drive droplets to move, merge, and split within predetermined chambers or channels within the chip using electrical, acoustic, or mechanical methods.

[0003] However, current automation processes in this field face a key technical bottleneck: how to determine, with high precision and real-time, whether a droplet has been successfully driven to its target position without contact. Common solutions and their drawbacks are as follows: 1. Detection methods based on electrical signals: This method determines whether a droplet has passed through or arrived by detecting changes in the capacitance or resistance of a channel or chamber. This method requires the integration of complex electrode structures into a chip, increasing the chip's fabrication cost and complexity. Furthermore, it is susceptible to factors such as solution ionic strength and environmental interference, resulting in poor reliability.

[0004] 2. Single Image Recognition-Based Method: This method involves capturing an image after the operation and then using simple image processing (such as thresholding) to locate the droplet. This method is highly dependent on the initial, fixed image position. If the chip undergoes even a slight translation or rotation during operation due to mechanical vibration, thermal expansion or contraction, the pre-defined detection area (ROI) will misalign with the actual chip structure, leading to recognition failure or misjudgment. The operational precision of microfluidic chips is typically at the micrometer level; therefore, any minute pose change is unacceptable.

[0005] 3. Methods relying on manual interpretation: This involves operators observing and judging the success of an experiment entirely under a microscope. This method is extremely inefficient, highly subjective, and cannot achieve high-throughput, automated experimental procedures, thus becoming a major obstacle limiting its large-scale application.

[0006] Therefore, there is an urgent need in this field for a fully automated droplet tracking and determination technology that can adapt to changes in chip position, is robust, has high precision, and does not require modification of the chip structure, in order to promote the real-world implementation of automated microfluidic chip platforms. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this paper provides a machine vision-based microfluidic chip droplet motion detection system, detection method, detection equipment, and computer medium. The aim is to solve the recognition drift problem caused by changes in chip pose (translation and rotation) and achieve highly reliable automatic determination of pipetting success rate without manual intervention.

[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention includes: a microfluidic chip droplet motion detection system based on machine vision, comprising a vision imaging module, an image processing module, a host computer control module, and a host computer; the vision program module is preferably an industrial camera; and further comprising: The high-precision position correction module uses the image processing module to locate the marked points in the acquired chip image. Then, it calculates the current position based on the center coordinates of the marked points, performs translation and rotation, and then updates the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module parses a list of one or more target numbers from the instruction. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it judges each region in turn. Only when the droplet percentage of all specified target regions exceeds the threshold will the algorithm return a success signal to the host computer. Otherwise, it returns a failure signal and repeats the current pipetting step until it succeeds or reaches the maximum number of retries. This establishes closed-loop communication between the intelligent communication and judgment module and the host computer control module.

[0009] Furthermore, the high-precision position correction module specifically includes: Image acquisition involves moving the chip to the center of the field of view of the vision imaging module (such as an industrial camera) and then using an industrial camera with an exposure time of 10,000 microseconds to capture a clear image (such as a circle or a cross) containing specific markers on the chip. Marker localization uses a circle detection algorithm to quickly and coarsely locate the possible regions of markers, and then precisely locates at least two markers on the chip image (such as the upper left and upper right markers). Next, using a pre-prepared marker template image, template matching is performed within each candidate region. By calculating a similarity score, the sub-pixel-level center coordinates of the two markers are finally determined. The similarity of the template images is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation, based on the center coordinates of two marker points, determines the current pose of the computer chip, including: The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr − Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

[0010] Furthermore, the dynamic droplet recognition module specifically includes: Background frame acquisition is used to capture a clear image of the chip in an industrial camera, improving the signal-to-noise ratio, as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection: In subsequent processes, each time the host computer controls a droplet movement operation, the host computer control module controls the industrial camera to capture a frame of the current position under the same parameters. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ; Precise location of region of interest: Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map; Droplet Image Enhancement and Recognition: Each small region is processed individually. First, image binarization is performed to separate the potential droplet region from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

[0011] Furthermore, the intelligent communication and judgment module specifically includes: Command parsing: The host computer sends a target string command to the image processing module. The image processing module parses the string and obtains a list of one or more target numbers. Region Traversal and Judgment: The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then judges each region sequentially. The judgment criterion is: calculating the proportion of pixels in the region that are identified as droplets out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: Results Feedback and Process Control: The image processing module returns a success signal to the host computer only when the droplet percentage of all specified target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or the maximum number of retries is reached.

[0012] A machine vision-based method for detecting droplet motion in microfluidic chips, employing a machine vision-based microfluidic chip droplet motion detection system, includes the following steps: Step 1: Acquire chip image. Take a corrected image I_calib with an exposure time of 10000 microseconds, crop it, and then mark the chip in at least two places. Read the template images T_tl and T_tr of the two pre-cropped marker points, perform Gaussian blur denoising on the image I_calib, and then use Hough circle transform to perform circle detection, roughly find all possible circular regions, which will greatly narrow the search range of template matching; within each circular ROI, use normalized squared difference matching method to perform template matching. Traverse the T_tl and T_tr image templates to find the position with the highest matching degree, and achieve sub-pixel localization through quadratic interpolation to finally obtain the precise center coordinates P_tl(x1, y1) and P_tr(x2, y2) of the two marker points; Calculate the transformation parameters: Let the theoretical coordinates of one marker point in the chip's preset diagram be P_tl0(100, 100), and the theoretical coordinates of another marker point be P_tr0(500, 100). Translation amounts: dx = x1 - 100, dy = y1 – 100, Rotation angle: θ = arctan2((y2 - y1), (x2 - x1)); Step 2: Acquire the background frame. After the position correction is completed, the host computer control module controls the industrial camera to set the exposure time to 50,000 microseconds, capture the first frame image I_background and store it in memory. At this time, all chambers of the chip are empty. Step 3: Move the target droplet from the current chamber to the target chamber, and the industrial camera takes another image of the current state (I_current). Next, calculate the difference image: perform the absolute difference operation between I_current and I_background to obtain the difference image I_diff = cv2.absdiff(I_current, I_background); Obtain the corresponding polygonal region from roi_dict in the target ROI. On the difference image I_diff, create a mask based on the polygonal region and extract the image block I_roi of that region for subsequent processing. Apply Gaussian blur to I_roi, then use Otsu's method to automatically calculate the threshold T and perform binarization to obtain I_binary. Perform an erosion operation on I_binary to remove tiny white noise points. Then perform a dilation operation to connect potentially broken regions inside the droplet to form a complete connected component. Step 4: Use the contour search algorithm to find all contours in I_binary, calculate the area of ​​each contour, filter out any residual noise, and calculate the total pixel percentage of the droplet in the ROI region R_area = the sum of the areas of all contours / the total number of pixels in the ROI region. Set the success threshold T_success = 0.3. If R_area >= 0.3, the corresponding chamber is considered successfully pipetted. Step 5: If the determination is successful, a success signal is sent. After receiving the success signal, the host computer will proceed to the next step in the process. If the determination fails, a failure signal is sent. After receiving the failure signal, the host computer will re-execute the operation of driving the target droplet to move to the corresponding chamber, and then trigger the software to take pictures, identify, and determine again. This cycle will be repeated a maximum of 3 times. If all 3 attempts fail, an anomaly will be reported and manual intervention will be required.

[0013] A machine vision-based microfluidic chip droplet motion detection device includes a vision imaging module, an image processing module, a host computer control module, and a host computer, and further includes: The high-precision position correction module is used to locate the marked points in the acquired chip image, then calculate the current position based on the center coordinates of the marked points, then translate and rotate, and then update the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module establishes closed-loop communication with the host computer control module. It parses a list of one or more target numbers from the instruction parsing. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it sequentially judges each region. Only when the droplet percentage of all specified target regions exceeds the threshold does the algorithm return a success signal to the host computer; otherwise, it returns a failure signal. The current pipetting step is then repeated until success or the maximum number of retries is reached.

[0014] Furthermore, the high-precision position correction module specifically includes: Image acquisition involves moving the chip to the center of the vision imaging module's field of view and then using an industrial camera with a 10,000-microsecond exposure time to capture a clear image containing specific marker points on the chip. Marker point localization involves quickly and coarsely locating the possible regions of marker points using a circle detection algorithm. Then, using pre-prepared marker point template images, template matching is performed within each candidate region. By calculating similarity scores, the sub-pixel-level center point coordinates of two marker points are finally determined. The similarity of the template images is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation: The current pose of the computer chip is determined based on the center coordinates of the two marker points. The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr − Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

[0015] Furthermore, the dynamic droplet recognition module specifically includes: Background frame acquisition is used to capture a clear image of the chip in an industrial camera, improving the signal-to-noise ratio, as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection is used in subsequent processes. Each time the host computer executes a droplet movement operation, the host computer control module controls the industrial camera to capture a new frame under the same parameters. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ; Precise location of the region of interest (ROI): Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map. Droplet image enhancement and recognition involves individual image processing for each small region extracted. First, image binarization is performed to separate potential droplet regions from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

[0016] Furthermore, the intelligent communication and judgment module specifically includes: Instruction parsing: The host computer sends a target string instruction to the image processing module, which parses the string to obtain a list of one or more target numbers; Region traversal and determination: The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then determines each region sequentially. The determination criterion is: calculating the proportion of pixels identified as droplets within the region out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: In terms of result feedback and process control, the image processing module only returns a success signal to the host computer when the droplet percentage in all designated target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or reaches the maximum number of retries.

[0017] Compared with the prior art, the advantages of the present invention include: (1) The present invention provides a microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision, which has high precision and high robustness: the innovative "circle detection + template matching" two-stage positioning method, combined with affine transformation, can automatically compensate for the translation and rotation of the chip, fundamentally solving the identification drift problem caused by mechanical error, making the system insensitive to the initial placement position of the chip and with strong anti-interference ability.

[0018] (2) The present invention provides a microfluidic chip droplet motion detection system, detection method and detection device based on machine vision, which has high sensitivity and high reliability: adopting the "background difference" strategy instead of simply relying on single frame image analysis, it can extremely sensitively capture tiny droplet motion changes, effectively suppress the interference of complex background, and greatly improve the signal-to-noise ratio and accuracy of droplet recognition.

[0019] (3) The present invention provides a microfluidic chip droplet motion detection system, detection method and detection device based on machine vision, with intelligent closed-loop control: intelligent decision-making in conjunction with host computer instructions is realized through dictionary mapping and area proportion determination. The determination logic is scientific (based on area proportion), and the communication protocol is simple and efficient, forming a complete automated closed loop of "perception-decision-execution-feedback".

[0020] (4) The present invention provides a microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision, which is highly versatile and low cost: the method is based entirely on machine vision software algorithm, without the need for any special structural modification of the microfluidic chip itself (such as processing electrodes), and is applicable to most transparent microfluidic chips, which significantly reduces the hardware cost and complexity of the system application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a microfluidic chip droplet motion detection method based on machine vision in this invention; Figure 2 This is a schematic diagram of the high-precision position correction module after processing of the microfluidic chip droplet motion detection system, detection method, and detection equipment based on machine vision in this invention; Figure 3 The image processing flow diagram of the droplet recognition module of the microfluidic chip droplet motion detection system, detection method and detection equipment based on machine vision in this invention is shown. Figure 4 This is a binarized image of the droplet recognition module in the machine vision-based microfluidic chip droplet motion detection system, detection method, and detection equipment of the present invention. Figure 5 This is a difference diagram showing the image processing flow of the droplet recognition module in the machine vision-based microfluidic chip droplet motion detection system, detection method, and detection equipment of the present invention. Detailed Implementation

[0023] In view of the shortcomings of the prior art, the inventors of this invention, through long-term research and extensive practice, have proposed the technical solution of this invention. The technical solution, its implementation process, and principles will be further explained below with reference to the accompanying drawings and specific implementation examples in the embodiments of this application.

[0024] It should be noted that the embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, the present invention covers any substitutions, modifications, equivalent methods and solutions made on the spirit, principles and scope of the present invention as defined by the claims. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] In the description of this application, the terms "first," "second," "third," and similar words do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "a" or "one," and similar words, do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "comprising" or "including," and similar words, mean that the elements or objects preceding "comprising" or "including" encompass the elements or objects listed following "comprising" or "including," and their equivalents, but do not exclude other elements or objects. The terms "connected" or "linked," and similar words, are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0026] In the description of this application, the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used solely for the convenience of describing this application and for simplification, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. Furthermore, when using positional terms such as "both sides," "outer side," and "upper and lower," it should be understood that they are used only for ease of understanding and description, taking into account that the structure may be oriented to other positions.

[0027] In the description of this application, unless otherwise expressly specified and limited, the technical or scientific terms used shall have the ordinary meaning understood by a person with ordinary skills in the art to which this application pertains. Terms such as “installation,” “connection,” and “joining” shall be interpreted broadly, for example, as fixed connection, detachable connection, mating connection, or integral connection. For a person skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0028] The present invention aims to introduce and explain the structural composition and the cooperation relationship between the components of a machine vision-based microfluidic chip droplet motion detection system, detection method and detection device. Unless otherwise specified, the size, material and manufacturing process of each component in the machine vision-based microfluidic chip droplet motion detection system, detection method and detection device in the present invention can be selected according to specific circumstances, and no special limitation or explanation is given here.

[0029] Furthermore, to provide the public with a better understanding of the present invention, certain specific details are described in detail in the following description of the invention. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0030] Example 1 Please see Figures 1-5A machine vision-based microfluidic chip droplet motion detection system is disclosed, comprising a vision imaging module, an image processing module, a host computer control module, and a host computer. The vision program module is preferably an industrial camera. The hardware system mainly includes: an industrial camera (such as a CMOS or CCD camera) fixed to a mechanical structure, an LED surface light source providing uniform backlighting for the chip, a platform for fixing the microfluidic chip, and a host computer that executes the image processing algorithm and communicates with the host PLC (Programmable Logic Controller) or industrial control computer. The software of this invention is an algorithm program developed based on languages ​​such as C++ and C#, and integrating machine vision libraries such as OpenCV and Halcon. It includes: The high-precision position correction module uses the image processing module to locate the marked points in the acquired chip image. Then, it calculates the current position based on the center coordinates of the marked points, performs translation and rotation, and then updates the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module parses a list of one or more target numbers from the instruction. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it judges each region in turn. Only when the droplet percentage of all specified target regions exceeds the threshold will the algorithm return a success signal to the host computer. Otherwise, it returns a failure signal and repeats the current pipetting step until it succeeds or reaches the maximum number of retries. This establishes closed-loop communication between the intelligent communication and judgment module and the host computer control module.

[0031] In this invention, the high-precision position correction module specifically includes: Image acquisition involves moving the chip to the center of the field of view of the vision imaging module (such as an industrial camera) and then using an industrial camera with an exposure time of 10,000 microseconds to capture a clear image (such as a circle or a cross) containing specific markers on the chip. Marker localization uses a circle detection algorithm to quickly and coarsely locate the possible regions of markers, and then precisely locates at least two markers on the chip image (such as the upper left and upper right markers). Next, using a pre-prepared marker template image, template matching is performed within each candidate region. By calculating a similarity score, the sub-pixel-level center coordinates of the two markers are finally determined. The similarity of the template images is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation, based on the center coordinates of two marker points, determines the current pose of the computer chip, including: The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr − Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

[0032] In this invention, the dynamic droplet recognition module specifically includes: Background frame acquisition is used to capture a clear image of the chip in an industrial camera, improving the signal-to-noise ratio, as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection: In subsequent processes, each time the host computer controls a droplet movement operation, the host computer control module controls the industrial camera to capture a frame of the current position under the same parameters. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ; Precise location of region of interest: Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map; Droplet Image Enhancement and Recognition: Each small region is processed individually. First, image binarization is performed to separate the potential droplet region from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

[0033] In this invention, the intelligent communication and judgment module specifically includes: Command parsing: The host computer sends a target string command to the image processing module. The image processing module parses the string and obtains a list of one or more target numbers. Region Traversal and Judgment: The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then judges each region sequentially. The judgment criterion is: calculating the proportion of pixels in the region that are identified as droplets out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: Results Feedback and Process Control: The image processing module returns a success signal to the host computer only when the droplet percentage of all specified target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or the maximum number of retries is reached.

[0034] A machine vision-based microfluidic chip droplet motion detection device, employing a machine vision-based microfluidic chip droplet motion detection system and detection method, includes a vision imaging module, an image processing module, a host computer control module, and a host computer, and further includes: The high-precision position correction module is used to locate the marked points in the acquired chip image, then calculate the current position based on the center coordinates of the marked points, then translate and rotate, and then update the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module establishes closed-loop communication between itself and the host computer control module. It parses a list of one or more target numbers from the instruction parsing. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it sequentially judges each region. Only when the droplet percentage of all specified target regions exceeds the threshold does the algorithm return a success signal to the host computer; otherwise, it returns a failure signal. The current pipetting step is then repeated until success or the maximum number of retries is reached.

[0035] In this invention, the high-precision position correction module specifically includes: Image acquisition involves moving the chip to the center of the vision imaging module's field of view and then using an industrial camera with a 10,000-microsecond exposure time to capture a clear image containing specific marker points on the chip. Marker point localization involves quickly and coarsely locating the possible regions of marker points using a circle detection algorithm. Then, using pre-prepared marker point template images, template matching is performed within each candidate region. By calculating similarity scores, the sub-pixel-level center point coordinates of two marker points are finally determined. The similarity of the template images is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation: The current pose of the computer chip is determined based on the center coordinates of the two marker points. The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr − Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

[0036] In this invention, the dynamic droplet recognition module specifically includes: Background frame acquisition is used to capture a clear image of the chip in an industrial camera, improving the signal-to-noise ratio, as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection is used in subsequent processes. Each time the host computer executes a droplet movement operation, the host computer control module controls the industrial camera to capture a new frame under the same parameters. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ; Precise location of the region of interest (ROI): Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map. Droplet image enhancement and recognition involves individual image processing for each small region extracted. First, image binarization is performed to separate potential droplet regions from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

[0037] In this invention, the intelligent communication and judgment module specifically includes: Instruction parsing: The host computer sends a target string instruction to the image processing module, which parses the string to obtain a list of one or more target numbers; Region traversal and determination: The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then determines each region sequentially. The determination criterion is: calculating the proportion of pixels identified as droplets within the region out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: In terms of result feedback and process control, the image processing module only returns a success signal to the host computer when the droplet percentage in all designated target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or reaches the maximum number of retries.

[0038] A machine vision-based method for detecting droplet motion in microfluidic chips, employing a machine vision-based microfluidic chip droplet motion detection system, includes the following steps: Initialization and position correction of a microfluidic chip droplet motion detection system based on machine vision (hereinafter referred to as the system). Step 1, Hardware Connection and Startup: Start the computer, camera, light source, and host computer control module. After the system starts, initialize the camera and establish a communication connection with the host computer.

[0039] Capturing the Corrected Image: The system sends a ready signal to the host computer via the communication interface. The industrial camera is set to an exposure time of 10,000 microseconds. After the host computer control module controls the robotic arm or moves the chip to the center of the industrial camera's field of view, a clear corrected image I_calib containing specific marker points on the chip is captured and cropped to remove excess blank space. These marker points are usually high-contrast graphics (such as circles or crosses) pre-made during chip design.

[0040] Step 2, Marker Identification and Pose Calculation: A combined algorithm of "circle detection + template matching" is used to accurately locate at least two markers on the chip, such as the upper left and upper right markers. Specifically, the system reads the pre-cropped template images T_tl and T_tr for the upper left and upper right markers. Gaussian blur is applied to the image I_calib for noise reduction, and then Hough circle transform is used for circle detection to roughly identify all possible circular regions, significantly narrowing the search area (ROI) for template matching. Within each circular ROI, template matching is performed using the normalized squared difference (TM_SQDIFF_NORMED) method. The T_tl and T_tr image templates are traversed to find the position with the highest matching degree (i.e., the smallest R(x,y) value), and sub-pixel localization is achieved through quadratic interpolation. Finally, the precise center coordinates P_tl(x1, y1) and P_tr(x2, y2) of the two markers are obtained.

[0041] Calculate the transformation parameters: Let the theoretical coordinates of the upper left marker in the chip design diagram be P_tl0(100, 100), and the theoretical coordinates of the upper right marker be P_tr0(500, 100). Specifically: Translation amount: dx = x1 - 100, dy = y1 - 100.

[0042] Rotation angle: θ = arctan2((y2 - y1), (x2 - x1)). (Theoretically, P_tr0 - P_tl0 = (400, 0), so the theoretical angle is 0).

[0043] Step 3: Acquire the background frame. After position correction is completed, the software controls the camera to set the exposure time to 50,000 microseconds (to obtain a brighter image), captures the first frame image I_background, and stores it in memory. At this time, all chambers of the chip are empty.

[0044] Step four, droplet movement and dynamic identification cycle: The host computer controls the actuator (such as high-voltage electrode) to apply a driving signal to the target droplet, attempting to move it from the current chamber to the target chamber.

[0045] Capture the current frame: After the driving operation is completed, the software controls the camera to capture another current image I_current.

[0046] Calculate the difference image: Perform the absolute difference operation between I_current and I_background to obtain the difference image I_diff = cv2.absdiff(I_current, I_background).

[0047] Step 5: Extract and process the target ROI: Assume the instruction string sent by the host computer is "1-2", requiring the droplet to be moved to chamber "1-2". The software obtains the polygonal region corresponding to the number "1-2" from roi_dict. On the difference image I_diff, a mask is created based on this polygonal region, and the image block I_roi of this region is extracted for subsequent processing. Gaussian blur is applied to I_roi, and then the threshold T is automatically calculated using Otsu's method, followed by binarization to obtain I_binary. I_binary is first subjected to an erosion operation (using 3x3 rectangular structuring elements) to remove tiny white noise points; then a dilation operation (using 5x5 rectangular structuring elements) is performed to connect potentially broken regions inside the droplet, forming a complete connected component.

[0048] Contour Recognition and Judgment: Use a contour finding algorithm (such as cv2.findContours) to find all contours in I_binary. Calculate the area (number of pixels) of each contour. A minimum area threshold (e.g., 50 pixels) is typically set to filter out any residual noise. Calculate the total pixel percentage of the droplet within the ROI region, R_area = sum of all contour areas / total number of pixels in the ROI region. Set a success threshold T_success = 0.3 (i.e., 30% of the area is occupied by the droplet). If R_area >= 0.3, then the pipetting in chamber "1-2" is considered successful.

[0049] Step 4: Communication and Feedback After the software completes the judgment, it sends the result to the host computer via an interface. In this example, if the judgment is successful, the string "1" is sent. Upon receiving "1", the host computer proceeds to the next step in the process. If the judgment fails (R_area < 0.3), the string "0" is sent. Upon receiving "0", the host computer will re-execute the operation of driving the droplet to move to chambers 1-2, and then trigger the software to take a picture, identify, and judge again. This loop repeats a maximum of 3 times. If all 3 attempts fail, an exception is reported, and manual intervention is required.

[0050] Understandable This approach provides a high-precision method for correcting the initial pose of a chip, enabling rapid calculation of the chip's offset and rotation angle in the XY plane.

[0051] A dynamic droplet detection method based on image difference is provided, which can effectively highlight moving targets (droplets) and suppress complex static background interference.

[0052] A smart communication and judgment mechanism that works in collaboration with a host computer is provided. It can accurately analyze the target area according to instructions, make scientific judgments based on the proportion of droplets in the area, and finally feed the results back to the host computer to control the process.

[0053] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. It should not be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several simple deductions or substitutions can be made without departing from the concept of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the protection scope of the present invention.

Claims

1. A microfluidic chip droplet motion detection system based on machine vision, characterized in that: include: The high-precision position correction module acquires chip images and locates the chip images by marking points. It calculates the current position based on the center coordinates of the marked points, then translates and rotates the chip, and then updates the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module parses a list of one or more target numbers. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it judges each region in turn. Only when the droplet proportion of all specified target regions exceeds the threshold will the algorithm return a success signal to the host computer. Otherwise, it returns a failure signal and repeats the current pipetting step until it succeeds or reaches the maximum number of retries. The high-precision position correction module specifically includes: Image acquisition: A clear image containing specific marker points on the chip is captured using an industrial camera with an exposure time of 10,000 microseconds to form a template image; Marker localization involves rapidly and coarsely locating the possible regions of markers using a circle detection algorithm, and then precisely locating at least two markers on the chip image. A template image is then used to perform template matching within each candidate region. By calculating a similarity score, the sub-pixel-level center point coordinates of the two markers are finally determined. The template image similarity is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation, based on the center coordinates of two marker points, determines the current pose of the computer chip, including: The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr - Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

2. The microfluidic chip droplet motion detection system based on machine vision according to claim 1, characterized in that: The dynamic droplet recognition module specifically includes: Background frame acquisition, improving the signal-to-noise ratio of the industrial camera to capture a clear image of the chip as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection involves capturing a frame of the current droplet under identical parameters using an industrial camera after each droplet movement operation in the subsequent process. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ； Precise location of the region of interest (ROI): Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map. Droplet image enhancement and recognition involves individual image processing for each identified target chamber region. First, image binarization separates the potential droplet region from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

3. The microfluidic chip droplet motion detection system based on machine vision according to claim 1, characterized in that: The intelligent communication and judgment module specifically includes: a host computer, an image processing module, and a host computer control module. Given a label string, obtain a list of one or more target numbers; The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then judges each region sequentially. The judgment criterion is: calculating the proportion of pixels in the region that are identified as droplets out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: The image processing module returns a success signal to the host computer only when the droplet percentage in all designated target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or the maximum number of retries is reached.

4. A method for detecting droplet motion in a microfluidic chip based on machine vision, characterized in that: Includes the following steps: Step 1: Acquire chip image. Take a corrected image I_calib with an exposure time of 10000 microseconds, crop it, and then mark the chip in at least two places. Read the labeled template images T_tl and T_tr, perform Gaussian blur denoising on the image I_calib, and then use Hough circle transform to perform circle detection, roughly find all possible circular regions, which will greatly narrow the search range for template matching; within each circular ROI, use normalized squared difference matching method to perform template matching. Traverse the T_tl template image and the T_tr template image to find the position with the highest matching degree, and achieve sub-pixel positioning through quadratic interpolation to finally obtain the precise center coordinates P_tl(x1, y1) and P_tr(x2, y2) of the two marker points; Let the theoretical coordinates of one marker point in the template image be P_tl0(100, 100), and the theoretical coordinates of another marker point be P_tr0(500, 100). Translation amounts: dx = x1 - 100, dy = y1 - 100, Rotation angle: θ = arctan2((y2 - y1), (x2 - x1)); Step 2: Acquire the background frame. After the position correction is completed, set the exposure time to 50,000 microseconds, capture the first frame image I_background and store it. At this time, all chambers of the chip are empty. Step 3: Move the target droplet from the current chamber to the target chamber, and take another image of the current state (I_current). Perform the absolute difference operation between I_current and I_background to obtain the difference image I_diff = cv2.absdiff(I_current, I_background); Obtain the corresponding polygonal region from roi_dict in the target ROI. On the difference image I_diff, create a mask based on the polygonal region and extract the image block I_roi of that region for subsequent processing. Apply Gaussian blur to I_roi, then use Otsu's method to automatically calculate the threshold T and perform binarization to obtain I_binary. Perform an erosion operation on I_binary to remove tiny white noise points. Then perform a dilation operation to connect potentially broken regions inside the droplet to form a complete connected component. Step 4: Use the contour search algorithm to find all contours in I_binary, calculate the area of ​​each contour, filter out any residual noise, and calculate the total pixel percentage of the droplet in the ROI region R_area = the sum of the areas of all contours / the total number of pixels in the ROI region. Set the success threshold T_success = 0.

3. If R_area >= 0.3, the corresponding chamber is considered successfully pipetted. Step 5: If the determination is successful, a success signal is sent. After receiving the success signal, the host computer will proceed to the next step in the process. If the determination fails, a failure signal is sent. After receiving the failure signal, the host computer will re-execute the operation of driving the target droplet to move to the corresponding chamber, and then trigger the software to take pictures, identify, and determine again. This cycle will be repeated a maximum of 3 times. If all 3 attempts fail, an anomaly will be reported and manual intervention will be required.

5. A microfluidic chip droplet motion detection device based on machine vision, characterized in that: Also includes: The high-precision position correction module is used to locate the marked points in the acquired chip image, then calculate the current position based on the center coordinates of the marked points, then translate and rotate, and then update the ROI mapping to generate a new region of interest (ROI) mapping dictionary that perfectly matches the current actual position of the chip. The dynamic droplet recognition module, after completing the chip image position correction, performs background frame acquisition, dynamic differential detection, precise location of region of interest, droplet image enhancement and recognition, and extracts the connected white area, which is then identified as a droplet. The intelligent communication and judgment module parses a list of one or more target numbers. Based on the target number, it finds the corresponding image region from the ROI mapping dictionary. Then, it judges each region in turn. Only when the droplet proportion of all specified target regions exceeds the threshold will the algorithm return a success signal to the host computer. Otherwise, it returns a failure signal and repeats the current pipetting step until it succeeds or reaches the maximum number of retries. The high-precision position correction module specifically includes: Image acquisition: A clear image containing specific marker points on the chip is captured using an industrial camera with an exposure time of 10,000 microseconds to form a template image; Marker localization involves rapidly and coarsely locating the possible regions of markers using a circle detection algorithm, and then precisely locating at least two markers on the chip image. A template image is then used to perform template matching within each candidate region. By calculating a similarity score, the sub-pixel-level center point coordinates of the two markers are finally determined. The template image similarity is calculated using the normalized squared difference matching method, with the following formula: Where T is the template image, I is the image to be searched, (x,y) is the search position, and the closer the value of R(x,y) is to 0, the higher the matching degree. By finding the minimum point of R(x,y), the center coordinates Ptl(xtl,ytl) and Ptr(xtr,ytr) of the marker point can be accurately located at the sub-pixel level. Pose parameter calculation, based on the center coordinates of two marker points, determines the current pose of the computer chip, including: The translation amount (dx, dy) is calculated as follows: Let the theoretical design coordinates of the upper left marker be Ptl0(xtl0, ytl0). The rotation angle θ is calculated by taking the vector V = Ptr - Ptl formed by the two marked points and finding the angle between it and the theoretical horizontal vector V0 = (1,0). This angle is obtained through the cross product and dot product of the vectors. The ROI mapping is updated by performing an affine transformation on the obtained (dx, dy, θ) transformation matrix and the pre-defined theoretical region coordinates, thereby generating a new ROI mapping dictionary in the image coordinate system that perfectly matches the actual position of the current chip.

6. The microfluidic chip droplet motion detection device based on machine vision according to claim 5, characterized in that: The dynamic droplet recognition module specifically includes: Background frame acquisition, improving the signal-to-noise ratio of the industrial camera to capture a clear image of the chip as the first frame. At this point, there should be no droplets inside the chip or the droplets should be in their initial positions. Dynamic differential detection involves capturing a frame of the current droplet under identical parameters using an industrial camera after each droplet movement operation in the subsequent process. The current frame is compared with the stored background image of the first frame using an image difference operation to obtain a difference image. : ； Precise location of the region of interest (ROI): Based on the generated ROI mapping dictionary, the target chamber region to be detected is accurately parsed from the difference map. Droplet image enhancement and recognition involves individual image processing for each identified target chamber region. First, image binarization separates the potential droplet region from the background. Then, morphological processing is used to optimize the binary image. : Finally, using contour recognition or convex hull detection algorithms, connected white regions are extracted from the processed binary image, and these regions are identified as droplets.

7. The microfluidic chip droplet motion detection device based on machine vision according to claim 5, characterized in that: The intelligent communication and judgment module specifically includes: a host computer, an image processing module, and a host computer control module. The host computer sends a target string instruction to the image processing module, which parses the string to obtain a list of one or more target numbers; The image processing module finds the corresponding image region from the ROI mapping dictionary based on the target number, and then judges each region sequentially. The judgment criterion is: calculating the proportion of pixels in the region that are identified as droplets out of the total number of pixels. : Where Nforeground is the total number of pixels identified as foreground within the region, and Ntotal is the total number of pixels in the region. The determination criteria are as follows: The image processing module returns a success signal to the host computer only when the droplet percentage in all designated target areas exceeds the threshold. After receiving the success signal, the host computer executes the next step of the process. If any target area is determined to be without droplets or the droplets fail to arrive, the image processing module returns a failure signal. After receiving the failure signal, the host computer will repeat the current pipetting step until it succeeds or the maximum number of retries is reached.

Images