Micro-droplet image recognition method based on color features and micro-droplet observation device

Abstract

The invention relates to the technical field of micro-droplet detection, solves the technical problem that in the prior art, micro-droplets in multiple working condition states cannot be accurately recognized, and particularly relates to a color feature-based micro-droplet image recognition method and a micro-droplet observation device. The micro-droplet image recognition method comprises the following steps: acquiring video image data of a micro-channel in a micro-fluidic chip in an optimal visual field of a color high-speed industrial camera, and converting the video image data into an HSV color space to obtain color images of micro-droplets with different color features; and judging the attribution of the segmented region according to the area of the micro-droplet region and the characteristics of the approximate circle. According to the method, searching and positioning can be carried out based on the color features of the micro-droplets, good robustness is achieved for other complex non-target edge features such as noise, shielding and bubbles, even multiple targets with similar shapes and areas can be accurately recognized, and the method can be well suitable for multi-channel color images.

Description

Technical Field

[0001] The present invention relates to the technical field of micro-droplet detection, and in particular to a micro-droplet image recognition method based on color features and a micro-droplet observation device. Background Art

[0002] For droplet microfluidics applications, the quality of droplet generation, including droplet size, generation frequency, droplet composition changes, and the number and size of double-layer droplet packages, directly impacts the success or failure of droplet microfluidics applications. Therefore, the detection and monitoring of droplet generation quality is particularly important. Microscopic imaging combined with image processing technology to detect and monitor droplet generation quality has become a widely used detection technology in recent years because it allows for contactless observation of droplet generation, avoiding possible contact contamination with reactants.

[0003] Currently, the main method for detecting the quality of microdroplet generation using image processing technology is image recognition based on the edge or shape features of microdroplets. However, this method is difficult to identify and process images of microdroplets with complex edge features, multiple types of microdroplets, double-layer complex microdroplets, mixed microdroplets with varying components, and microdroplets with complex backgrounds. It also often requires more complex image recognition and processing algorithms and higher computing hardware requirements. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a micro-droplet image recognition method and a micro-droplet observation device based on color features, which solves the technical problem that the existing technology cannot accurately identify micro-droplets under multiple working conditions.

[0005] To solve the above technical problems, the present invention provides the following technical solution: a method for identifying micro-droplet images based on color features, the method comprising the following steps: S1. Acquire video image data of a microchannel in a microfluidic chip within the optimal field of view of a color high-speed industrial camera, and convert the video image data into an HSV color space to obtain color images of microdroplets with different color features; S2. Segmenting the color image from the micro-droplet region and the chip background region based on the color features, and determining the ownership of the segmented region based on the area of ​​the micro-droplet region and the characteristics of the approximate circle; S3, performing post-processing on the segmented micro-droplet region to eliminate noise points generated after image segmentation; S4, identifying and marking the shape characteristics, center position and motion state information of the micro-droplet according to the post-processed micro-droplet area; S5. Identify and detect single-layer microdroplets containing a mixture of different components, single-layer microdroplets of different types with merging, and double-layer complex microdroplets.

[0006] Furthermore, the microdroplets include single-layer microdroplets and double-layer complex microdroplets, wherein the single-layer microdroplets include conventional single-layer microdroplets, different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of single-layer microdroplets that are merged, and the double-layer complex microdroplets include inner-layer microdroplets in the double-layer complex microdroplets and outer-layer microdroplets in the double-layer complex microdroplets.

[0007] Furthermore, in step S1, the specific process includes the following steps: S11, using the aqueous phase as the inner phase of the microdroplets and the oil phase as the outer phase of the microdroplets, dyeing the inner phase fluid with a water-soluble pigment to generate microdroplets with different colors or different components within the microdroplets; S12. Adjust the control platform so that the microchannels in the microfluidic chip are in the optimal field of view of the color high-speed industrial camera. At the same time, adjust the corresponding magnification and light intensity. S13. Import the video image data captured by the color high-speed industrial camera into the working computer and convert it into the HSV color space.

[0008] Furthermore, in step S2, the specific process includes the following steps: S21. Determine the value range of color saturation and brightness of the non-neutral color based on the HSV color space characteristics, and select a segmentation threshold for segmenting the micro-droplet area and the chip background area based on the value range; S22, segmenting the color image into a plurality of segmented regions based on a segmentation threshold value for a region with high color saturation and a region with low color saturation; S23, determining whether the plurality of segmented regions belong to micro-droplet regions or chip background regions; If the area of ​​several segmented regions is larger and the circularity is less than 0.7, it is the chip background area; otherwise, it is the micro-droplet area.

[0009] Furthermore, in step S22, the area with high color saturation is the dyed micro-droplet, and the area with low color saturation is the chip background, wherein segmenting the micro-droplet and the chip background includes: Segmentation of conventional single-layer microdroplets, inner-layer microdroplets in double-layer complex microdroplets, and chip background areas; segmentation of different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of merged single-layer microdroplets and chip background areas; If the chip background has no complex colors, the S and V channels of the HSV color space are used to directly segment the microdroplets with high color saturation from the chip background area with low color saturation according to the segmentation threshold; If the chip background has complex colors, the S and V channels of the HSV color space are used to directly segment the microdroplets with high color saturation from some chip background areas with low color saturation according to the segmentation threshold. Then, based on the two or more peak areas corresponding to the H channel histogram of the HSV color space, the OTSU maximum class distance method is used to obtain the optimal segmentation threshold between the two or more peak areas, and the microdroplet area and the chip background area are segmented using the optimal segmentation threshold.

[0010] Furthermore, in step S22, the outer layer micro-droplets in the double-layer complex micro-droplets and the chip background area are segmented, specifically: First, the color image is processed into a grayscale image, and the edge contour of the outer micro-droplet is obtained using the Sobel or Laplacian edge detection operator; Based on the roundness of the edge contour, it is judged whether there are microdroplets in the grayscale image. If yes, the edge contour of the microdroplet is segmented from the chip background. Otherwise, no segmentation is performed.

[0011] Furthermore, in step S4, the specific process includes the following steps: S41, traverse all contours in the micro-droplet area and calculate the roundness of the contours , the calculation formula is: ; in, represents the area of ​​the i-th contour; is the perimeter of the i-th contour; S42, according to roundness Determine whether the contour is the target micro-droplet contour area; like , then the contour is the target micro-droplet contour area, and the process goes to step S43; like , then return to step S41, is the target threshold; S43, performing ellipse fitting on the target micro-droplet contour area to obtain shape features and marking it as a micro-droplet; S44, taking the average value of the major axis and the minor axis after ellipse fitting as the approximate radius of the microdroplet, and taking the center position of the target microdroplet contour area as the center position of the microdroplet; S45, using an inter-frame difference method to analyze the video image data of the micro-droplet region frame by frame to obtain the center position of the micro-droplet on two adjacent frames of color images; S46 , obtaining the motion state information of the micro-droplet according to the center position of the micro-droplet in two adjacent color images, that is, the moving distance and direction of the center position of the same micro-droplet within one frame time.

[0012] Furthermore, in step S5, the specific process includes the following steps: S51. Performing a different component mixing test on a single-layer micro-droplet containing a mixture of different components. The specific process includes the following steps: S511, calculate the distance from the center position of any contour area in the segmented area where the component is located to the center position of the single layer micro droplet , and the radius of the monolayer droplets is ; S512, determine whether there is and only one single layer of micro-droplets satisfy ; If not, it is considered that the mixing of different components inside the monolayer microdroplet is complete; If yes, proceed to step S513; S513: Determine the area of ​​the contour area Is it less than a minimum threshold? ; like , it is considered that the mixing of different components inside the monolayer microdroplets is complete; like , it is considered that the different components inside the monolayer microdroplets are not mixed completely; S52, identifying the merger between different types of monolayer micro-droplets that have merged, the specific process includes the following steps: S521. Calculate the center position of any microdroplet in different types of monolayer microdroplets The center position of any microdroplet in another type of single-component microdroplet distance ; S522, according to distance Determine whether the two microdroplets have completed merging; If satisfied , then the two microdroplets complete the merger, is a minimum distance threshold; S53. Detecting the encapsulation and encapsulation states of the inner microdroplets and the outer microdroplets in the double-layer complex microdroplets.

[0013] Furthermore, in step S53, the specific process includes the following steps: S531. Calculate the distance between the center position of any inner micro-droplet and the center position of each outer micro-droplet. ; S532, according to distance and the approximate radius of the outer droplet Determine whether the inner layer of micro-droplets is wrapped normally; If all the distances Both are larger than the approximate radius , then the inner layer of micro-droplets is not wrapped properly; If there is only one distance Smaller than the approximate radius , it means that the inner layer of micro-droplets is wrapped normally; S533. Calculate the distance between the center position of any outer micro-droplet and the center position of each inner micro-droplet. ; S534, according to distance and the approximate radius of the outer droplet Determine the number of inner microdroplets encapsulated within the outer microdroplets; If there is a distance Smaller than the approximate radius , then the number of inner layer micro-droplets wrapped inside the outer layer micro-droplets increases by one; If the number of inner-layer micro-droplets wrapped in the outer-layer micro-droplets does not match the setting, the outer-layer micro-droplet wrapping state is abnormal.

[0014] By means of the above technical solution, the present invention provides a micro-droplet image recognition method and micro-droplet observation device based on color features, which have at least the following beneficial effects: 1. The present invention simplifies the process of micro-droplet image recognition and detection algorithm, omitting the early background removal and noise removal processes, and directly segments micro-droplets from the background based on color features, thereby reducing the algorithm difficulty. At the same time, it solves the recognition difficulty problem caused by the mutual interference between the edges of the inner and outer micro-droplets of double-layer complex micro-droplets, and proposes to use different colors on different types of micro-droplets for recognition based on color features, thereby solving the problem of multi-type micro-droplet classification that cannot be solved by traditional micro-droplet image recognition and detection methods based on edge and shape features, and can solve the micro-droplet recognition problem under more working conditions.

[0015] 2. The present invention can realize the detection of conventional single-layer microdroplets, single-layer multi-type microdroplets, mixed detection of different components inside single-layer microdroplets, combined detection of different single-layer microdroplets, and detection of double-layer complex microdroplets, and can detect the encapsulation status of inner-layer microdroplets and the number of outer-layer microdroplets in double-layer complex microdroplets.

[0016] 3. The present invention can search and locate based on the color features of micro-droplets, and has good robustness against other complex non-target edge features such as noise, occlusion, bubbles, etc. Even multiple targets with similar shapes and areas can be accurately identified. It can be well applied to multi-channel color images and is very suitable for micro-droplet recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 Schematic diagram of the structure of the micro-droplet observation device of the present invention; Figure 2 This is a result diagram of the single-layer micro-droplet recognition under a simple color background of the present invention; Figure 3 This is a result diagram of the single-layer micro-droplet recognition under a complex color background of the present invention; Figure 4 This is the result diagram of the recognition of different types of monolayer microdroplets with a complex background in the present invention; Figure 5 Result diagram of identification and detection of monolayer micro-droplet mixing of the present invention; Figure 6 This is a result diagram of the recognition and detection of the merging of single-layer micro-droplets of the present invention; Figure 7 This is a diagram showing the results of the identification and detection of double-layer complex microdroplets of the present invention. DETAILED DESCRIPTION

[0018] To make the above-mentioned objectives, features, and advantages of the present invention more clearly understood, the present invention is further described below in detail with reference to the accompanying drawings and specific embodiments. This will enable a full understanding of how this application uses technical means to solve technical problems and achieve technical effects, and to implement the invention accordingly.

[0019] Please refer to Figure 1-Figure 7 This embodiment proposes a color-based microdroplet image recognition method that can detect conventional single-layer microdroplets, multiple types of single-layer microdroplets, mixed components within a single-layer microdroplet, combined components of different single-layer microdroplets, and complex double-layer microdroplets. It can also detect the encapsulation status of inner microdroplets and the number of outer microdroplets encapsulated within a complex double-layer microdroplet. The recognition method includes the following steps: S1. Acquire video image data of a microchannel in a microfluidic chip within the optimal field of view of a color high-speed industrial camera, and convert the video image data into the HSV color space to obtain color images of microdroplets with different color characteristics; the microdroplets include single-layer microdroplets and double-layer complex microdroplets, wherein the single-layer microdroplets include conventional single-layer microdroplets, different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of single-layer microdroplets with merging, and the double-layer complex microdroplets include inner-layer microdroplets in the double-layer complex microdroplets and outer-layer microdroplets in the double-layer complex microdroplets. In step S1, the specific process includes the following steps: S11. Using the aqueous phase as the inner phase of the microdroplets and the oil phase as the outer phase of the microdroplets, the inner phase fluid is dyed with a water-soluble pigment to form a dispersed phase fluid, thereby generating microdroplets with different colors or different components within the microdroplets; wherein, different types of dispersed phases are used for dyeing, and colors with large color contrast are preferred. After dyeing, different dispersed phases will subsequently form microdroplets of different colors or different components within the microdroplets, so one color represents one microdroplet or component.

[0020] S12. Adjust the control platform so that the microchannels in the microfluidic chip are in the optimal field of view of the color high-speed industrial camera. At the same time, adjust the corresponding magnification and light intensity. S13. Import the video image data captured by the color high-speed industrial camera into the working computer and convert it into the HSV color space.

[0021] S2. Segmenting the color image from the micro-droplet region and the chip background region based on the color features, and determining the ownership of the segmented region based on the area of ​​the micro-droplet region and the characteristics of the approximate circle; in step S2, the specific process includes the following steps: S21. Determine the value range of the color saturation and brightness of the non-neutral color based on the HSV color space characteristics, and select the segmentation threshold for segmenting the micro-droplet area and the chip background area according to the value range; in this step, it can be seen from the HSV color space characteristics that the value range of the color saturation (S) of the non-neutral color is [43, 255] and the value range of the brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the range of subsequent image processing. The optimal segmentation threshold selected is [0, 43, 46]-[180, 255, 255].

[0022] S22. Segmenting the color image into a plurality of segmented regions based on a segmentation threshold value from regions with high color saturation. In step S22, the regions with high color saturation are the dyed microdroplets, and the regions with low color saturation are the chip background. Segmenting the microdroplets from the chip background includes: Segmentation of conventional single-layer microdroplets, inner-layer microdroplets in double-layer complex microdroplets, and chip background areas; segmentation of different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of merged single-layer microdroplets and chip background areas; If the chip background has no complex colors, the S and V channels of the HSV color space are used to directly segment the microdroplets with high color saturation from the chip background area with low color saturation according to the segmentation threshold; If the chip background has complex colors, the S and V channels of the HSV color space are used to directly segment the microdroplets with high color saturation from some chip background areas with low color saturation according to the segmentation threshold. Then, based on the two or more peak areas corresponding to the H channel histogram of the HSV color space, the OTSU maximum class distance method is used to obtain the optimal segmentation threshold between the two or more peak areas, and the microdroplet area and the chip background area are segmented using the optimal segmentation threshold.

[0023] In this process, the segmentation of conventional single-layer microdroplets and the inner layer of double-layer complex microdroplets is a binary classification problem based on color features. Specifically, since the principle of image segmentation is based on color features, the edge features of the chip background, such as complex textures and complex microchannels, have little effect on image segmentation. Therefore, the focus should be on whether the chip background has complex colors. If the chip background does not have complex colors, the S and V channels of the HSV color space are used, and the preferred segmentation threshold is [0,43,46]-[180,255,255]. This can directly segment the microdroplets with high color saturation from the chip background with low color saturation. If the chip background has complex colors, the S and V channels of the HSV color space are first used, preferably with a fixed segmentation threshold between [0, 43, 46] and [180, 255, 255]. This allows for direct segmentation of highly saturated microdroplets from areas of the chip with less saturation, resulting in a smoother H channel histogram in the color image. This smoothing of the H channel allows for subsequent image segmentation. In this case, the H channel histogram often exhibits two peaks: one representing the microdroplets and another representing the chip background. The OTSU maximum inter-class distance method is used to determine the optimal segmentation threshold between these two peaks, achieving complete segmentation of the microdroplets from the chip background.

[0024] Among them, the segmentation of different types of monolayer microdroplets, monolayer microdroplets containing a mixture of different components, and monolayer microdroplets of different types with mergers belongs to a multi-classification problem based on color features. That is, they contain microdroplets of different colors or different components within the microdroplets. Therefore, the segmentation method adopted is the same as when the chip background has complex colors. The difference is that the H channel histogram mostly presents multiple peak areas, and different types (N types) of microdroplets and the background each have a peak area (a total of N+1). The maximum class distance method can also be used to obtain the optimal segmentation threshold between multiple peak areas to achieve complete segmentation between different types of microdroplets and between microdroplets and the chip background area.

[0025] Similarly, the first frame of the microdroplet video is used to determine the N thresholds [T1, T2, T3, …, TN] for the multi-peak region segmentation problem. Subsequently, the color images in subsequent videos are segmented based on fixed thresholds [0, T1], [T1, T2], …, [TN, 180] to save image segmentation time. Those skilled in the art will readily appreciate that the aforementioned maximum class distance method for obtaining optimal segmentation thresholds between multi-peak regions can also be combined with recursive algorithms, genetic algorithms, and other methods to further improve image segmentation efficiency. This also includes segmenting the outer layer of a double-layer complex microdroplet from the chip background area.

[0026] Specifically, there is still the problem of segmenting the outer layer of micro-droplets in a double-layer complex micro-droplet. In response to this problem, due to the need to observe the inner layer of micro-droplets, the intermediate phase that produces the outer layer of micro-droplets is generally not dyed, so the edge features of the outer layer of micro-droplets are used for image segmentation. First, the color image is processed into a grayscale image, and the edge contour of the outer layer of micro-droplets is preferably obtained by using a Sobel or Laplacian edge detection operator. At this time, the presence of micro-droplets in the grayscale image is judged based on the roundness of the edge contour. In this way, the non-micro-droplet area is excluded by binarization and based on the roundness judgment result, and the complete segmentation of the micro-droplet edge contour and the background is achieved. In particular, here, the inner layer (color features) and outer layer (shape and edge features) micro-droplets are identified separately based on different features, which effectively avoids the problem of droplet recognition failure caused by strong interference such as the overlap of the two layers of droplet contours, and can obtain independent information on the inner and outer layer micro-droplet areas, which is convenient for subsequent identification and detection.

[0027] S23. Determine whether the segmented regions are microdroplet regions or chip background regions. If the segmented region has a larger area and a circularity less than 0.7, it is the chip background region; otherwise, it is the microdroplet region. Since the total area of ​​the chip background region is larger than the microdroplet region, the total area of ​​the segmented regions is used as the criterion to determine whether the smaller region is the microdroplet region. Furthermore, to avoid errors in more complex working conditions, the presence of a segmented region with a circularity less than 0.7 can be used as a criterion for determining the chip background region, taking into account the approximately circular shape of microdroplets.

[0028] S3. Post-process the segmented micro-droplet area to eliminate the noise points generated after image segmentation. After determining the micro-droplet area, use median filtering to remove some noise points that may be generated after image segmentation, and perform morphological corrosion and morphological dilation to further achieve small area removal. Specifically, first take the kernel as (3,3), use image morphological corrosion to eliminate the noise influence generated in image segmentation, and then use image morphological dilation to ensure that the detection size is as accurate as possible. The number of iterations of corrosion and dilation is 1.

[0029] S4. Identify and mark the shape characteristics, center position and motion state information of the micro-droplets based on the post-processed micro-droplet area; for the color images of micro-droplets in all working conditions, it is necessary to identify the shape characteristics of the micro-droplets and important information such as the center position and motion characteristics according to the following method. In addition, for single-layer micro-droplets containing a mixture of different components, the segmented images of all different components inside the single-layer micro-droplets should be summed to form an accurate micro-droplet area. For different types of single-layer micro-droplets, it is also necessary to indicate the type of micro-droplets they belong to based on the color image segmentation results between the micro-droplets. In order to eliminate the interference of irrelevant edges and shape features in complex backgrounds, realize the recognition and extraction of target micro-droplets, and the recognition and classification of micro-droplets of different components, in step S4, the specific process includes the following steps: S41, traverse all contours in the micro-droplet area and calculate the roundness of the contours , the calculation formula is: ; in, represents the area of ​​the i-th contour; is the perimeter of the i-th contour; S42, according to roundness Determine whether the contour is the target micro-droplet contour area; like , then the contour is the target micro-droplet contour area, and the process goes to step S43; like , then returns to step S41, wherein the target threshold M is set to 0.7 in this embodiment; S43, performing ellipse fitting on the target micro-droplet contour area to obtain shape features and marking it as a micro-droplet; S44, taking the average value of the major axis and the minor axis after ellipse fitting as the approximate radius of the microdroplet, and taking the center position of the target microdroplet contour area as the center position of the microdroplet; S45. Analyzing the video image data of the micro-droplet region frame by frame using an inter-frame difference method to obtain the center positions of the micro-droplets in two adjacent color image frames; wherein the minimum adjacent distance is used as a criterion for frame association, that is, the two micro-droplets with the smallest distance between the micro-droplet centers in two adjacent frames are the same micro-droplet; S46 , obtaining the motion state information of the micro-droplet according to the center position of the micro-droplet in two adjacent color images, that is, the moving distance and direction of the center position of the same micro-droplet within one frame time.

[0030] In this embodiment, the roundness feature is first used to determine whether it is the target micro-droplet contour area, and then the target micro-droplet contour area is elliptical fitted to obtain the size of the micro-droplet. At the same time, the micro-droplet morphology is detected based on the roundness of the elliptical fitting. Subsequently, the center position of the micro-droplet area is selected as the position coordinate of the micro-droplet to detect the motion state of the micro-droplet in the microchannel. The inter-frame difference method is used for the motion video of the micro-droplet, and the minimum adjacent distance is used as the criterion for frame association. That is, the two micro-droplets with the smallest distance between the micro-droplet centers between two adjacent frames are considered to be the same micro-droplet. Then, the moving distance and direction of the same micro-droplet center position within one frame time are obtained, thereby realizing the monitoring of the micro-droplet motion behavior and obtaining the micro-droplet motion state information, including motion speed, motion direction, motion status, etc.

[0031] S5. Identify and detect single-layer microdroplets containing a mixture of different components, single-layer microdroplets of different types that have merged, and double-layer complex microdroplets. For more complex microdroplet conditions, such as single-layer microdroplets containing a mixture of different components, single-layer microdroplets of different types that have merged, and double-layer complex microdroplets, additional indicators need to be detected based on the identified information according to the following method to achieve comprehensive detection and monitoring of the quality of complex microdroplet generation. In step S5, the specific process includes the following steps:

[0032] S51, performing different component mixing detection on a single-layer micro-droplet containing a mixture of different components. In order to identify the mixture of different components inside the micro-droplet, in step S51, the specific process includes the following steps: S511, calculate the distance from the center position of any contour area in the segmented area where the component is located to the center position of the single layer micro droplet , and the radius of the monolayer droplets is ; S512, determine whether there is and only one single layer of micro-droplets satisfy ; If not, it is considered that the mixing of different components inside the monolayer microdroplet is complete; If yes, proceed to step S513; S513: Determine the area of ​​the contour area Is it less than a minimum threshold? ; like , it is considered that the mixing of different components inside the monolayer microdroplets is complete; like , it is considered that the different components inside the monolayer microdroplets are not mixed completely; Taking the position and area of ​​any one or more components in the monolayer microdroplet as indicators, the center position and area of ​​all contour areas in the segmented area of ​​the component are calculated one by one. For each contour area, the distance between its center position and the center position of the monolayer microdroplet is first calculated. The center position of the single-layer micro-droplet is the center position of the target micro-droplet contour area in step S44. The radius of the single-layer micro-droplet is , determine whether there is one and only one single layer of microdroplets satisfy If not, it is considered that the different components in the single layer of micro-droplets are mixed; if so, the area of ​​the contour area is further determined. Is it less than a minimum threshold? ,like , it is considered that the mixing of different components inside the single layer of microdroplets is complete.

[0033] S52, identifying the mergers between different types of monolayer micro-droplets. To identify the mergers between different micro-droplets, in step S52, the specific process includes the following steps: S521. Calculate the center position of any microdroplet in different types of monolayer microdroplets The center position of any microdroplet in another type of single-component microdroplet distance ; S522, according to distance Determine whether the two microdroplets have completed merging; If satisfied , then the two microdroplets complete merging; Using the center position and radius of different monolayer microdroplets as indicators, for each microdroplet in one type of monolayer microdroplet, the distance between its center position and the center position of each microdroplet in another type of single-component microdroplet is calculated. , if satisfied ,in is a minimum distance threshold. If yes, it means that the two micro-droplets have completed merging.

[0034] S53: Detect the encapsulation and encapsulation status of the inner and outer microdroplets in the double-layer complex microdroplets. To eliminate interference from overlapping edges between the inner and outer droplets in the double-layer complex microdroplets, the number and size of the inner and outer microdroplets in the double-layer complex microdroplets can be simultaneously identified, as well as the microdroplets of different components in the inner layer. Furthermore, the successful encapsulation and rupture of the inner droplets can be detected, thereby achieving comprehensive detection and monitoring of the quality of the double-layer complex microdroplet generation. In step S53, the specific process includes the following steps: S531. Calculate the distance between the center position of any inner micro-droplet and the center position of each outer micro-droplet. ; S532, according to distance and the approximate radius of the outer droplet Determine whether the inner layer of micro-droplets is wrapped normally; If all the distances Both are larger than the approximate radius , then the inner layer of micro-droplets is not wrapped properly; If there is only one distance Smaller than the approximate radius , it means that the inner layer of micro-droplets is wrapped normally; S533. Calculate the distance between the center position of any outer micro-droplet and the center position of each inner micro-droplet. ; S534, according to distance and the approximate radius of the outer droplet Determine the number of inner microdroplets encapsulated within the outer microdroplets; If there is a distance Smaller than the approximate radius , then the number of inner layer micro-droplets wrapped inside the outer layer micro-droplets increases by one; If the number of inner-layer micro-droplets wrapped in the outer-layer micro-droplets does not match the setting, the outer-layer micro-droplet wrapping state is abnormal.

[0035] Specifically, the center position and approximate radius of the inner and outer layer micro-droplets are used as indicators, namely: Inner layer micro-droplet status: judged by whether there are outer layer micro-droplets. For each inner layer micro-droplet, calculate the distance between its center position and the center position of each outer layer micro-droplet and the approximate radius of the outer microdroplet Compare, if all Both greater than , it means that the inner layer of micro-droplets is not wrapped properly. If there is only one Less than , it means that the inner layer of micro-droplets are wrapped normally.

[0036] Outer layer micro-droplet status: mainly detects the number of inner layer micro-droplets wrapped inside, and for each outer layer micro-droplet, calculates the distance between its center position and the center position of each inner layer micro-droplet , which is also related to the radius of the outer micro-droplet Compare, if there is a Less than , then the number of inner-layer micro-droplets of the inner package is increased by one. Special attention should be paid to the fact that when the number of inner-layer micro-droplets of the inner package does not match the setting, the outer-layer micro-droplet packaging state is abnormal.

[0037] In practical applications, this microdroplet image recognition method can detect conventional single-layer microdroplets, multiple types of single-layer microdroplets, mixed components within single-layer microdroplets, merged microdroplets from different single-layers, and complex double-layer microdroplets. It can also detect the encapsulation status of inner microdroplets and the number of droplets encapsulated by outer microdroplets within complex double-layer microdroplets. Furthermore, due to the robustness of color features, using conventional single-layer microdroplet image recognition as an example, this method achieves a segmentation accuracy (P) of 96% and a recall rate (R) of over 95%. The deviations caused by variations in color type, light intensity, and color saturation are all within 5%.

[0038] like Figure 1 As shown, this embodiment provides a micro-droplet observation device applied to the above-mentioned micro-droplet image recognition method, including a microfluidic system and an observation system.

[0039] The microfluidic system is used to adjust the flow rate of the syringe pump and control the generation of microdroplets by the microfluidic chip. The microfluidic system consists of a syringe pump, a control platform, and a microfluidic chip. The microfluidic chip is fixed on the control platform. To facilitate observation, the material of the microfluidic chip should be a material with good light transmittance. In this embodiment, photosensitive resin is used and the molding is based on light-curing 3D printing (SLA).

[0040] The observation system is used to observe and record the microdroplets generated by the microfluidic chip. It consists of a light source and a color high-speed industrial camera. The light source is preferably a white parallel light source, and the hardware core is a color high-speed industrial camera (Hikbot MV-CH050-10UC) for observing and recording the microdroplets.

[0041] According to the micro-droplet image recognition method and micro-droplet observation device based on color features proposed in the above embodiment, for conventional single-layer micro-droplets, single-layer multi-type micro-droplets, mixed detection of different components inside single-layer micro-droplets, merging detection of different single-layer micro-droplets, detection of double-layer complex micro-droplets, and detection of the inner layer micro-droplet encapsulation state and the number of outer layer micro-droplets in double-layer complex micro-droplets, please refer to Figures 2 to 7 This embodiment can search and locate microdroplets based on their color features, and is highly robust to complex non-target edge features such as noise, occlusion, and bubbles. It can accurately identify multiple targets with similar shapes and areas, and is well suited for multi-channel color images, making it ideal for microdroplet recognition scenarios. Specific implementations for the aforementioned various microdroplet detection conditions are as follows: Example 1

[0042] Identification and detection of conventional monolayer microdroplets against a simple color background This example uses the identification of a single layer of microdroplets against a normal-color transparent chip background as an example. With the aqueous phase as the inner phase and the oil phase as the outer phase, the inner phase fluid is first dyed with a water-soluble pigment (a non-neutral color is selected) to ensure uniform dyeing of the dispersed phase fluid. The control platform is then adjusted to ensure that the microchannels in the microfluidic chip are within the optimal field of view of the color high-speed camera. The magnification and light intensity are also adjusted accordingly for subsequent microdroplet image recognition and detection. The video within the field of view of the color high-speed industrial camera is imported into the work computer and converted to the HSV color space.

[0043] Conventional single-layer micro-droplet image segmentation based on color features: Because the chip background is simple, no image preprocessing is required. The HSV color space shows that the saturation (S) of non-neutral colors ranges from [43, 255], and the value (V) ranges from [46, 255]. The colored microdroplet region must fall within this range. Therefore, the color image of the microdroplet is segmented using a low threshold of [0, 43, 46] and a high threshold of [180, 255, 255], achieving segmentation between the microdroplet region and the chip background. The area of ​​the microdroplet region and its circular approximation are used to determine the identity of the segmented region. Since the total area of ​​the chip background region is larger than the microdroplet region, the total area of ​​the segmented region is used as the criterion to determine that the smaller area is the microdroplet region. Furthermore, to avoid errors in more complex background conditions, the presence of a contour region with a circularity less than 0.7 can be used as a criterion for chip background region identification, in combination with the microdroplet's circular shape.

[0044] Image post-processing after image segmentation: First, the kernel is set to (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0045] Microdroplet feature information identification and marking: For the micro-droplet region after the image segmentation, traverse all contours in this region, because the characteristic of the micro-droplet approximate circle, adopt outline circularity greater than 0.7 as criterion, in order to identify each micro-droplet more accurately, can add minimum contour area and maximum contour area as auxiliary judgment, take the present embodiment as example, main criterion is that outline circularity is greater than 0.7, and the criterion of auxiliary judgment is that minimum contour area is greater than 100 pixels, and maximum contour area is less than 5000 pixels.For each definite micro-droplet profile, because generally speaking, micro-droplet is easily squeezed, usually shows as ellipse, therefore adopt ellipse fitting to mark micro-droplet, and with the mean value of major axis and minor axis after ellipse fitting as the approximate radius of this micro-droplet. In addition, represent the position of this micro-droplet with the contour area center position.For each frame image, all can obtain the position of all micro-droplets on the image, for the position of each micro-droplet on adjacent two frame images, calculate distance each other, think that two micro-droplets with minimum distance are same micro-droplet on the two frame images, infer the motion speed and the direction of motion of micro-droplet with this. Finally, the identification result of each micro-droplet feature information is marked next to the micro-droplet, such as Figure 2 shown. Example 2

[0046] Identification and detection of conventional monolayer microdroplets against complex color backgrounds This example uses the identification of monolayer microdroplets with poor color as an example. Using a water phase as the inner phase and an oil phase as the outer phase, the inner phase fluid is first dyed with a water-soluble pigment (a non-neutral color is selected) to ensure uniform dyeing of the dispersed phase fluid. The control platform is then adjusted to ensure that the microchannels in the microfluidic chip are within the optimal field of view of the color high-speed camera. The magnification and light intensity are also adjusted accordingly for subsequent microdroplet image recognition and detection. The video within the field of view of the color high-speed industrial camera is imported into the work computer and converted to the HSV color space.

[0047] Image preprocessing: From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the scope of subsequent image processing.

[0048] Conventional single-layer micro-droplet image segmentation based on color features: First, image segmentation is performed based on color features. For the preprocessed micro-droplet color image, its HSV channel is separated. At this time, the H channel of the micro-droplet color image is a typical bimodal problem. The OTSU maximum class distance is used to determine its optimal segmentation threshold to achieve the segmentation of the micro-droplet area and the chip background area. Secondly, the area of ​​the micro-droplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the micro-droplet area, the total area of ​​the segmented area is used as the criterion to determine the smaller area as the micro-droplet area. In addition, in order to avoid errors in more complex working conditions, combined with the shape characteristics of the micro-droplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as the criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine its optimal segmentation threshold T based on the first frame image. For each subsequent frame, the threshold T is directly used for image segmentation to save the program running time for each subsequent frame.

[0049] Image post-processing after image segmentation: First, the kernel is set to (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0050] Microdroplet feature information identification and marking: For the micro-droplet region after the image segmentation, traverse all contours in this region, because the characteristic of the micro-droplet approximate circle, adopt outline circularity greater than 0.7 as criterion, in order to identify each micro-droplet more accurately, can add minimum contour area and maximum contour area as auxiliary judgment, take the present embodiment as example, main criterion is that outline circularity is greater than 0.7, and the criterion of auxiliary judgment is that minimum contour area is greater than 100 pixels, and maximum contour area is less than 5000 pixels.For each definite micro-droplet profile, because generally speaking, micro-droplet is easily squeezed, usually shows as ellipse, therefore adopt ellipse fitting to mark micro-droplet, and with the mean value of major axis and minor axis after ellipse fitting as the approximate radius of this micro-droplet. In addition, represent the position of this micro-droplet with the contour area center position.For each frame image, all can obtain the position of all micro-droplets on the image, for the position of each micro-droplet on adjacent two frame images, calculate distance each other, think that two micro-droplets with minimum distance are same micro-droplet on the two frame images, infer the motion speed and the direction of motion of micro-droplet with this. Finally, the identification result of each micro-droplet feature information is marked next to the micro-droplet, such as Figure 3 shown. Example 3

[0051] Identification and detection of various types of monolayer microdroplets, various types of monolayer microdroplets such as Figure 4The difficulty lies in the severe squeezing of the two different types of microdroplets, the complex channel edges, and the high level of interference, which makes identification difficult.

[0052] This example uses the identification of two monolayer microdroplets as an example, with an aqueous phase as the inner phase and an oil phase as the outer phase. First, the two inner phase fluids are dyed with water-soluble pigments (using non-neutral colors with large color differences, in this case green and red). However, the color is poor in this case. To ensure uniform dyeing of the dispersed phase fluid, the control platform is adjusted so that the microchannels in the microfluidic chip are within the optimal field of view of the color high-speed camera. The magnification and light intensity are also adjusted accordingly for subsequent microdroplet image recognition and detection. The video within the field of view of the color high-speed industrial camera is imported into the work computer and converted to the HSV color space.

[0053] Image preprocessing based on color features: From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the scope of subsequent image processing.

[0054] Micro-droplet image segmentation based on color features: First, image segmentation is performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. At this time, the H channel of the microdroplet color image is a three-peak problem. The OTSU maximum class distance is used to determine the optimal segmentation thresholds T1 and T2, respectively. These thresholds are used to achieve segmentation between different types of microdroplets and between the microdroplet area and the chip background area. Secondly, the area of ​​the microdroplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the microdroplet area, the total area of ​​the segmented area is used as the criterion to determine that the smaller area is the microdroplet area. In addition, to avoid errors in more complex working conditions, combined with the shape characteristics of the microdroplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as the criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine the optimal segmentation thresholds T1 and T2 based on the first frame image. For each subsequent frame, the thresholds [0, 43, 46]–[T1,255,55], [T1,255,255]–[T2,255,255], [T2,255,255]–[180,255,255] are directly used for image segmentation to save the program running time for each subsequent frame.

[0055] Image post-processing after image segmentation: First, the kernel is set to (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0056] Microdroplet feature information identification and marking: For the micro-droplet area after image segmentation, all the contours of the area are traversed. Due to the characteristic of micro-droplets being approximately circular, the contour circularity greater than 0.7 is used as the criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgments. Taking this embodiment as an example, the main criterion is that the contour circularity is greater than 0.7, and the auxiliary judgment criteria are that the minimum contour area is greater than 20 pixels and the maximum contour area is less than 500 pixels. For each determined micro-droplet contour, since micro-droplets are generally easily squeezed and often appear as ellipses, ellipse fitting is used to mark the micro-droplets, and the average value of the major axis and the minor axis after ellipse fitting is used as the approximate radius of the micro-droplet, and the center position of the contour area represents the position of the micro-droplet. In addition, for the contours in different segmented images, different colors are used to mark them in the original image, and the type of droplet to which they belong is indicated, and finally as shown in FIG. Figure 4 Similarly, for each frame of image, the positions of all microdroplets in the image can be obtained. For the position of each microdroplet in two adjacent frames of image, the distance between them is calculated. The two microdroplets with the smallest distance between the two frames of image are considered to be the same microdroplet, and the movement speed and direction of the microdroplets can be inferred from this. Example 4

[0057] Identification and detection of mixing of different components within a monolayer microdroplet The difficulty in this implementation case lies in the irregular and rapidly changing edges of the component regions during the component mixing process. Without color features, it is impossible to accurately extract the component regions, which is impossible with edge detection.

[0058] This example uses the identification of a mixture of two different components within a monolayer of microdroplets as an example. With the aqueous phase as the inner phase and the oil phase as the outer phase, the inner phase fluid is first dyed with a water-soluble pigment (using non-neutral colors, green and purple in this example) to ensure uniform dyeing of the dispersed phase fluid. The control platform is then adjusted to ensure that the microchannels in the microfluidic chip are within the optimal field of view of the color high-speed camera. The magnification and light intensity are also adjusted accordingly for subsequent microdroplet image recognition and detection. The video within the field of view of the color high-speed industrial camera is imported into the work computer and converted to the HSV color space.

[0059] Image preprocessing based on color features: From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the scope of subsequent image processing.

[0060] Segmentation of target droplets and background based on color features: First, image segmentation is performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. At this time, the H channel of the microdroplet color image is a three-peak problem. The OTSU maximum class distance is used to determine the optimal segmentation thresholds T1 and T2, respectively. This threshold is used to achieve segmentation between different components within the microdroplet and between the microdroplet area and the chip background area. Secondly, the area of ​​the microdroplet area is used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the microdroplet area, the total area of ​​the segmented area is used as the criterion to determine that the smaller area is the microdroplet area. In addition, in order to be able to identify the complete microdroplet contour later, the segmentation images of all different color component areas are summed to obtain a complete microdroplet area segmentation image. At the same time, in order to avoid errors in more complex working conditions, combined with the shape characteristics of the microdroplet that are approximately circular, the existence of a contour area with a roundness less than 0.7 can be used as the criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine the optimal segmentation thresholds T1 and T2 based on the first frame image. For each subsequent frame, the thresholds [0,43,46]–[T1,255,255], [T1,255,255]–[T2,255,255], [T2,255,255]–[180,255,255] are directly used for image segmentation to save the program running time for each subsequent frame.

[0061] Image post-processing after image segmentation: First, the kernel is set to (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0062] Identification and marking of microdroplet characteristic information: For the microdroplet region after image segmentation, the segmented images of the different components are first merged to obtain the actual single-layer microdroplet region segmentation image. All contours of the region are traversed. Due to the characteristic of microdroplets being approximately circular, a contour circularity greater than 0.7 is used as a criterion. To more accurately identify each microdroplet, a minimum contour area and a maximum contour area are added as auxiliary criteria. In this embodiment, the primary criterion is a contour circularity greater than 0.7, while auxiliary criteria are a minimum contour area greater than 100 pixels and a maximum contour area less than 1000 pixels. For each determined microdroplet contour, since microdroplets are generally easily squeezed and often appear elliptical, ellipse fitting is used to mark the microdroplet. The average value of the major and minor axes after ellipse fitting is used as the approximate radius of the microdroplet. In addition, the center of the contour region represents the position of the microdroplet. For each frame of image, the positions of all microdroplets in the image can be obtained. For the position of each microdroplet in two adjacent frames of image, the distance between them is calculated. The two microdroplets with the smallest distance between the two frames of image are considered to be the same microdroplet, thereby inferring the movement speed and direction of the microdroplets.

[0063] Detection of micro-droplet mixing: Take the purple component area in the single layer of microdroplets and detect the center position, radius and area of ​​all contours Traverse all the microdroplet contours in the microdroplet area and calculate the distance between the center position of each purple component contour area and the center position of the microdroplet area , determine the distance With the radius of the microdroplet The size of Both greater than , indicating that the micro-droplet mixing is complete, if there is Less than , indicating that the mixing of the micro-droplets is not complete. Mark the droplets that have completed mixing on the original image, such as Figure 5 shown. Example 5

[0064] Identification and detection of merging of micro-droplets from different monolayers. Droplet merging is an edge overlap problem, which is a detection target that is difficult to achieve with traditional edge feature-based detection algorithms. However, the method proposed in this embodiment is simpler and can accurately identify it.

[0065] This example uses the merging and identification of two different monolayer microdroplets as an example, with the aqueous phase as the inner phase and the oil phase as the outer phase. First, the inner phase fluid is dyed with a water-soluble pigment (a non-neutral color is selected; in this example, green and purple are used) to ensure uniform dyeing of the dispersed phase fluid. The control platform is adjusted so that the microchannels in the microfluidic chip are within the optimal field of view of the color high-speed camera. The magnification and light intensity are also adjusted accordingly for subsequent microdroplet image recognition and detection. The video within the field of view of the color high-speed industrial camera is imported into the work computer and converted to the HSV color space.

[0066] Image preprocessing based on color features: From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the scope of subsequent image processing.

[0067] Segmentation of target droplets and background based on color features: First, image segmentation is performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. At this time, the H channel of the microdroplet color image is a three-peak problem. The OTSU maximum class distance is used to determine the optimal segmentation thresholds T1 and T2, respectively. These thresholds are used to achieve segmentation between different types of microdroplets and between the microdroplet area and the chip background area. Secondly, the area of ​​the microdroplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the microdroplet area, the total area of ​​the segmented area is used as the criterion to determine that the smaller area is the microdroplet area. In addition, to avoid errors in more complex working conditions, combined with the shape characteristics of the microdroplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as the criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine the optimal segmentation thresholds T1 and T2 based on the first frame image. For each subsequent frame, the thresholds [0, 43, 46]–[T1, 255, 255], [T1, 255, 255]–[T2, 255, 255], [T2, 255, 255]–[180, 255, 255] are directly used for image segmentation to save the program running time for each subsequent frame.

[0068] Acquisition of microdroplet characteristic information: For two kinds of little droplet zones after the image segmentation, all traverse all contours in respective regions, because the characteristic of little droplet approximate circle, adopt outline circularity greater than 0.7 as criterion, in order to identify each little droplet more accurately, can add minimum contour area and maximum contour area as auxiliary judgment, take present embodiment as example, main criterion is that outline circularity is greater than 0.7, and the criterion of auxiliary judgment is that minimum contour area is greater than 20 pixels, and maximum contour area is less than 500 pixels.For each definite little droplet profile, because generally speaking, little droplet is easily squeezed, usually show as ellipse, therefore adopt ellipse fitting to mark little droplet, and with the mean value of major axis and minor axis after ellipse fitting as the approximate radius of this little droplet.In addition, represent the position of this little droplet with outline region center position.For each frame image, all can obtain the position of all little droplets on the image, for the position of each little droplet on adjacent two frame images, calculate distance each other, think that two little droplets with minimum distance are same little droplet on the two frame images, infer motion speed and the direction of motion of little droplet with this.

[0069] Image post-processing after image segmentation: First, the kernel is set to (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0070] Detection of microdroplet merging: Using the center position and radius of different monolayer microdroplets as indicators, for each microdroplet in one type of monolayer microdroplet, the distance between its center position and the center position of each microdroplet in another type of single-component microdroplet is calculated. , if satisfied , then is a minimum threshold. If yes, it means that the two micro-droplets have completed merging. In this embodiment, . Then mark it in the original image. Figure 6 shown. Example 6

[0071] Identification and detection of double-layer complex micro-droplets This embodiment takes a double-layer complex micro-droplet with different numbers of micro-droplets wrapped inside as an example, and the background is complex and the color is poor, wherein there are complex droplet edge features, and the edges of the inner and outer droplets overlap, so it is difficult to accurately identify them using edge features. The aqueous phase is the inner and outer phases, and the oil phase is the intermediate phase. First, the inner phase fluid should be dyed with a water-soluble pigment (a non-neutral color is selected, and brown is used in this embodiment) to ensure that the dispersed phase fluid is evenly dyed, and the control platform is adjusted so that the microchannel in the microfluidic chip should be in the optimal field of view of the color high-speed camera. At the same time, the corresponding magnification and light intensity should be adjusted for subsequent micro-droplet image recognition and detection. The video within the field of view of the color high-speed industrial camera is imported into the working computer and converted into the HSV color space.

[0072] Identify inner microdroplets based on color features: First, preprocessing is performed. According to the HSV color space characteristics, the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The inner micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the scope of subsequent image processing.

[0073] Image segmentation is then performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. The H channel of the microdroplet color image is a typical bimodal problem. The OTSU maximum inter-class distance is used to determine the optimal segmentation threshold, achieving segmentation between the microdroplet region and the chip background region. The microdroplet region area and the characteristics of the approximate circle are then used to determine the identity of the segmented region. Since the total area of ​​the chip background region is larger than the microdroplet region, the total area of ​​the segmented region is used as the criterion to determine the smaller area as the microdroplet region. Furthermore, to avoid errors in more complex working conditions, the presence of a contour area with a circularity less than 0.7 can be used as a criterion for the chip background region, in combination with the microdroplet's approximate circular shape. For video detection, the optimal segmentation threshold T can be determined based on the first frame of the image using the above method. This threshold T is then directly used for image segmentation in each subsequent frame, saving program runtime for each subsequent frame.

[0074] Identify outer microdroplets based on edge features: Based on the original image (non-HSV color space), after grayscale conversion, the Sobel edge operator is used to identify the edges in the micro-droplet image (Sobel and Laplacian operators are preferred, which are better than the Canny operator). Based on the high roundness of the micro-droplet edges, all contours are traversed, and the contour roundness greater than 0.7 is used as the criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgments. Taking this embodiment as an example, the main judgment criterion is the contour roundness greater than 0.7, and the auxiliary judgment criteria are the minimum contour area greater than 500 pixels and the maximum contour area less than 10,000 pixels. In addition, due to the characteristics of droplet microfluidics, if the microfluidic chip has stains or the observation surface of the microfluidic chip is uneven, it may cause unstable edge recognition (for example, the detected microdroplet edge is incomplete and is judged not to be a microdroplet outline). Hough circle detection can be used for supplementary recognition. The microdroplet outline identified by Hough circle detection is compared with the microdroplet outline identified by edge detection. If the position deviation between the two is less than Min pixels, the edge detection result is retained. If the position deviation between the two is greater than the approximate radius of the edge detected microdroplet, it is considered that the edge detection missed the detection, and the outline detected by Hough circle detection is supplemented as a microdroplet identification result. In this embodiment, Hough circle detection is not required for auxiliary recognition.

[0075] Image post-processing after image segmentation: First, the kernel is set to (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0076] Inner and outer micro-droplet status detection: In this embodiment, since the inner and outer layer micro-droplet detection processes are completely independent and do not affect each other, completely separate inner layer micro-droplet information and outer layer micro-droplet information can be obtained. Based on the distance between the inner layer micro-droplet and the outer layer micro-droplet, the encapsulation state of the inner layer micro-droplet and the number of droplets contained in the outer layer micro-droplet are determined. First, for each inner layer micro-droplet, the distance between it and each outer layer micro-droplet is calculated. , and compare it with the outer micro-droplet radius Compare, if there is a Less than the corresponding , then the wrapping state is normal, otherwise it is abnormal. Initially, the number of droplets inside each outer micro-droplet is set to zero. For each outer micro-droplet, calculate its distance to each inner micro-droplet. , and compare it with the outer micro-droplet radius Compare, if there is a Less than the corresponding , then the number of droplets wrapped by the outer micro-droplet is increased by one until all the inner micro-droplets are traversed, and the number of droplets wrapped inside the outer micro-droplet is output. Figure 7 shown.

[0077] Those skilled in the art will appreciate that all or part of the steps in the above-mentioned embodiment methods can be accomplished by instructing the relevant hardware through a program. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to in detail. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For relevant parts, please refer to the partial description of the method embodiments.

[0079] The above embodiments provide a detailed introduction to the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for micro-droplet image recognition based on color features, characterized in that: The method comprises the following steps: S1, obtaining video image data of a microchannel in a microfluidic chip within the optimal field of view of a color high-speed industrial camera, and converting the video image data into an HSV color space to obtain a color image of microdroplets with different color features; S2, segmenting the color image from the micro-droplet region and the chip background region based on the color feature, and determining the ownership of the segmented region according to the area of ​​the micro-droplet region and the characteristics of the approximate circle; S3, performing post-processing on the segmented micro-droplet region to eliminate noise points generated after image segmentation; S4, identifying and marking the shape characteristics, center position and motion state information of the micro-droplet according to the post-processed micro-droplet area; S5. Identify and detect single-layer microdroplets containing a mixture of different components, single-layer microdroplets of different types that have merged, and double-layer complex microdroplets.

2. The micro-droplet image recognition method according to claim 1, characterized in that: The microdroplets include single-layer microdroplets and double-layer complex microdroplets, wherein the single-layer microdroplets include conventional single-layer microdroplets, different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of single-layer microdroplets that are merged, and the double-layer complex microdroplets include inner-layer microdroplets in the double-layer complex microdroplets and outer-layer microdroplets in the double-layer complex microdroplets.

3. The micro-droplet image recognition method according to claim 1, characterized in that: In step S1, the specific process includes the following steps: S11, using the water phase as the inner phase of the microdroplets and the oil phase as the outer phase of the microdroplets, dyeing the inner phase fluid with a water-soluble pigment to generate microdroplets with different colors or different components in the microdroplets; S12. Adjust the control platform so that the microchannel in the microfluidic chip is in the best field of view of the color high-speed industrial camera, and adjust the corresponding magnification and light intensity; S13, importing the video image data captured by the color high-speed industrial camera into the working computer and converting it into the HSV color space.

4. The micro-droplet image recognition method according to claim 1, characterized in that: In step S2, the specific process includes the following steps: S21, determining the value range of color saturation and brightness of the non-neutral color based on the HSV color space characteristics, and selecting a segmentation threshold for segmenting the micro-droplet area and the chip background area according to the value range; S22, segmenting the area with high color saturation and the area with low color saturation in the color image according to the segmentation threshold to form a plurality of segmented areas; S23, determining whether a number of segmented regions belong to micro-droplet regions or chip background regions; If the area of ​​several segmented regions is larger and the segmented region with a circularity less than 0.7 is the chip background region; otherwise, it is the micro-droplet region.

5. The micro-droplet image recognition method according to claim 4, characterized in that: In step S22, the area with high color saturation is the dyed micro-droplet, and the area with low color saturation is the chip background, wherein segmenting the micro-droplet and the chip background includes: Segmentation of conventional single-layer microdroplets, inner-layer microdroplets in double-layer complex microdroplets and chip background areas, as well as segmentation of different types of single-layer microdroplets, single-layer microdroplets containing different components mixed inside, and different types of single-layer microdroplets with merging and chip background areas; If the chip background has no complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from the chip background area with low color saturation according to the segmentation threshold; If the chip background has complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from some chip background areas with low color saturation according to the segmentation threshold. Then, according to the two or more peak areas corresponding to the H channel histogram of the HSV color space, the OTSU maximum class spacing method is used to obtain the optimal segmentation threshold between the two or more peak areas, and the micro-droplet area and the chip background area are segmented with the optimal segmentation threshold.

6. The micro-droplet image recognition method according to claim 4, characterized in that: In step S22, the outer layer of the double-layer complex micro-droplets and the chip background area are segmented, specifically: Firstly, the color image is processed into a grayscale image, and the edge contour of the outer layer of micro-droplets is obtained by using the Sobel or Laplacian edge detection operator; Based on the roundness of the edge contour, it is judged whether there are microdroplets in the grayscale image. If yes, the edge contour of the microdroplet is segmented from the chip background. Otherwise, no segmentation is performed.

7. The micro-droplet image recognition method according to claim 1, characterized in that: In step S4, the specific process includes the following steps: S41, traverse all contours in the micro-droplet area and calculate the roundness of the contours , the calculation formula is: ； in, represents the area of ​​the i-th contour; is the perimeter of the i-th contour; S42, according to roundness Determine whether the contour is a target micro-droplet contour area; like , then the contour is the target micro-droplet contour area, and the process goes to step S43; like , then return to step S41, is the target threshold; S43, performing ellipse fitting on the target micro-droplet contour area to obtain shape features, and marking it as a micro-droplet; S44, taking the average value of the major axis and the minor axis after the ellipse fitting as the approximate radius of the micro-droplet, and taking the central position of the target micro-droplet contour area as the central position of the micro-droplet; S45, using an inter-frame difference method to analyze the video image data of the micro-droplet region frame by frame to obtain the center position of the micro-droplet on two adjacent color image frames; S46, obtaining the motion state information of the micro-droplet according to the center position of the micro-droplet in two adjacent color images, that is, the moving distance and direction of the center position of the same micro-droplet within one frame time.

8. The micro-droplet image recognition method according to claim 1, characterized in that: In step S5, the specific process includes the following steps: S51, performing a different component mixing detection on a single layer of micro-droplets containing different components, the specific process includes the following steps: S511, calculating the distance from the center position of any contour area in the segmented area where the component is located to the center position of the single layer micro droplet , and the radius of the monolayer droplets is ; S512: Determine whether there is one and only one single layer of micro-droplets satisfy ; If not, it is considered that the mixing of different components inside the monolayer microdroplets is complete; If yes, proceed to step S513; S513: Determine the area of ​​the contour area Is it less than a minimum threshold? ; like , it is considered that the mixing of different components inside the monolayer microdroplets is complete; like , it is considered that the different components inside the monolayer microdroplets are not mixed completely; S52, identifying the merger between different types of monolayer micro-droplets that have merged, the specific process includes the following steps: S521. Calculate the center position of any microdroplet in different types of monolayer microdroplets The center position of any microdroplet in another type of single-component microdroplet Distance ; S522, according to the distance Determine whether two micro-droplets have completed merging; If satisfied , then the two microdroplets complete merging, is a minimum distance threshold; S53, detecting the encapsulation and encapsulation states of the inner layer micro-droplets and the outer layer micro-droplets in the double-layer complex micro-droplets.

9. The micro-droplet image recognition method according to claim 8, characterized in that: In step S53, the specific process includes the following steps: S531, calculate the distance between the center position of any inner layer micro-droplet and the center position of each outer layer micro-droplet ; S532, according to distance and the approximate radius of the outer droplet Determine whether the inner layer micro-droplets are wrapped normally; If all the distances Greater than the approximate radius , then the inner layer of micro-droplets is not wrapped properly; If there is only one distance Smaller than the approximate radius , it means that the inner layer of micro-droplets is wrapped normally; S533, calculate the distance between the center position of any outer layer micro-droplet and the center position of each inner layer micro-droplet ; S534, according to distance and the approximate radius of the outer droplet Determine the number of inner layer micro-droplets enclosed within the outer layer micro-droplets; If there is a distance Smaller than the approximate radius , then the number of inner layer micro-droplets wrapped inside the outer layer micro-droplets increases by one; If the number of inner-layer micro-droplets wrapped inside the outer-layer micro-droplets does not match the setting, the outer-layer micro-droplet wrapping state is abnormal.

10. A micro-droplet observation device applied to the micro-droplet image recognition method according to any one of claims 1 to 9, characterized in that: include: A microfluidic system, which is used to adjust the flow rate of the injection pump and control the microfluidic chip to generate microdroplets. The microfluidic system consists of a injection pump, a control platform and a microfluidic chip; the microfluidic chip is fixed on the control platform; The observation system is used to observe and record the microdroplets generated by the microfluidic chip. The observation system consists of a light source and a color high-speed industrial camera.

Images