A micro-droplet image recognition method based on color features and a micro-droplet observation device

Abstract

The present invention relates to the field of micro-droplet detection technology, solves the technical problem that the prior art cannot accurately identify micro-droplets under multiple working conditions, and particularly relates to a micro-droplet image recognition method and a micro-droplet observation device based on color features. The micro-droplet image recognition method includes: obtaining video image data of a microchannel in a microfluidic chip in the best field of view of a color high-speed industrial camera, and converting the video image data into an HSV color space to obtain a color image of micro-droplets with different color features; judging the ownership of a segmented area based on the area of ​​the micro-droplet area and the features of an approximate circle. The present invention can search and locate based on the color features of micro-droplets, has good robustness to other complex non-target edge features such as noise, occlusion, and bubbles, and can accurately identify multiple targets with similar shapes and areas, and can be well applied to multi-channel color images.

Description

Technical Field

[0001] The present invention relates to the technical field of micro-droplet detection, and in particular to a micro-droplet image recognition method based on color features and a micro-droplet observation device. Background Art

[0002] For droplet microfluidics application scenarios, the quality of microdroplet generation, including the size of microdroplet generation, generation frequency, droplet component changes, number and size of double-layer droplet packages, etc., directly affects the success or failure of droplet microfluidics application. Therefore, the detection and monitoring of the quality of microdroplet generation is particularly important. Microscopic imaging combined with image processing technology to detect and monitor the quality of microdroplet generation has become a widely used detection technology in recent years because it can observe microdroplet generation without contact, avoiding possible contact contamination of reactants, etc.

[0003] At present, the main method for detecting the quality of micro-droplet generation by image processing technology is based on image recognition of the edge or shape characteristics of micro-droplets. However, this method is difficult to identify and process images of micro-droplets with complex edge features, multiple types of micro-droplets, double-layer complex micro-droplets, mixed micro-droplets with varying components, and micro-droplet images with complex backgrounds, and often requires more complex image recognition and processing algorithms and higher computing hardware requirements. Summary of the invention

[0004] In view of the deficiencies of the prior art, the present invention provides a micro-droplet image recognition method and a micro-droplet observation device based on color features, which solves the technical problem that the prior art cannot accurately identify micro-droplets under multiple working conditions.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for identifying micro-droplet images based on color features, the method comprising the following steps:

[0006] S1, obtaining video image data of a microchannel in a microfluidic chip within the optimal field of view of a color high-speed industrial camera, and converting the video image data into an HSV color space to obtain a color image of microdroplets with different color features;

[0007] S2, segmenting the color image from the micro-droplet region and the chip background region based on the color feature, and determining the ownership of the segmented region according to the area of ​​the micro-droplet region and the characteristics of the approximate circle;

[0008] S3, performing post-processing on the segmented micro-droplet region to eliminate noise points generated after image segmentation;

[0009] S4, identifying and marking the shape characteristics, center position and motion state information of the micro-droplet according to the post-processed micro-droplet area;

[0010] S5. Identify and detect single-layer microdroplets containing a mixture of different components, single-layer microdroplets of different types that have merged, and double-layer complex microdroplets.

[0011] Furthermore, the microdroplets include single-layer microdroplets and double-layer complex microdroplets, wherein the single-layer microdroplets include conventional single-layer microdroplets, different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of single-layer microdroplets that are merged, and the double-layer complex microdroplets include inner-layer microdroplets in the double-layer complex microdroplets and outer-layer microdroplets in the double-layer complex microdroplets.

[0012] Furthermore, in step S1, the specific process includes the following steps:

[0013] S11, using the water phase as the inner phase of the microdroplets and the oil phase as the outer phase of the microdroplets, dyeing the inner phase fluid with a water-soluble pigment to generate microdroplets with different colors or different components in the microdroplets;

[0014] S12. Adjust the control platform so that the microchannel in the microfluidic chip is in the best field of view of the color high-speed industrial camera, and adjust the corresponding magnification and light intensity;

[0015] S13, importing the video image data captured by the color high-speed industrial camera into the working computer and converting it into the HSV color space.

[0016] Furthermore, in step S2, the specific process includes the following steps:

[0017] S21, determining the value range of color saturation and brightness of the non-neutral color based on the HSV color space characteristics, and selecting a segmentation threshold for segmenting the micro-droplet area and the chip background area according to the value range;

[0018] S22, segmenting the area with high color saturation and the area with low color saturation in the color image according to the segmentation threshold to form a plurality of segmented areas;

[0019] S23, determining whether a number of segmented regions belong to micro-droplet regions or chip background regions;

[0020] If the area of ​​several segmented regions is larger and the segmented region with a circularity less than 0.7 is the chip background region; otherwise, it is the micro-droplet region.

[0021] Further, in step S22, the area with high color saturation is the dyed micro-droplet, and the area with low color saturation is the chip background, wherein segmenting the micro-droplet and the chip background includes:

[0022] Segmentation of conventional single-layer microdroplets, inner-layer microdroplets in double-layer complex microdroplets and chip background areas, as well as segmentation of different types of single-layer microdroplets, single-layer microdroplets containing different components mixed inside, and different types of single-layer microdroplets with merging and chip background areas;

[0023] If the chip background has no complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from the chip background area with low color saturation according to the segmentation threshold;

[0024] If the chip background has complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from some chip background areas with low color saturation according to the segmentation threshold. Then, according to the two or more peak areas corresponding to the H channel histogram of the HSV color space, the OTSU maximum class spacing method is used to obtain the optimal segmentation threshold between the two or more peak areas, and the micro-droplet area and the chip background area are segmented with the optimal segmentation threshold.

[0025] Furthermore, in step S22, the outer layer of the double-layer complex micro-droplets and the chip background area are segmented, specifically:

[0026] Firstly, the color image is processed into a grayscale image, and the edge contour of the outer layer of micro-droplets is obtained by using the Sobel or Laplacian edge detection operator;

[0027] Based on the roundness of the edge contour, it is judged whether there are microdroplets in the grayscale image. If yes, the edge contour of the microdroplet is segmented from the chip background. Otherwise, no segmentation is performed.

[0028] Furthermore, in step S4, the specific process includes the following steps:

[0029] S41, traverse all contours in the micro-droplet area and calculate the roundness of the contours , the calculation formula is:

[0030] ;

[0031] in, represents the area of ​​the i-th contour; is the perimeter of the i-th contour;

[0032] S42, according to roundness Determine whether the contour is a target micro-droplet contour area;

[0033] like , then the contour is the target micro-droplet contour area, and the process goes to step S43;

[0034] like , then return to step S41, is the target threshold;

[0035] S43, performing ellipse fitting on the target micro-droplet contour area to obtain shape features, and marking it as a micro-droplet;

[0036] S44, taking the average value of the major axis and the minor axis after the ellipse fitting as the approximate radius of the micro-droplet, and taking the central position of the target micro-droplet contour area as the central position of the micro-droplet;

[0037] S45, using an inter-frame difference method to analyze the video image data of the micro-droplet region frame by frame to obtain the center position of the micro-droplet on two adjacent color image frames;

[0038] S46, obtaining the motion state information of the micro-droplet according to the center position of the micro-droplet in two adjacent color images, that is, the moving distance and direction of the center position of the same micro-droplet within one frame time.

[0039] Furthermore, in step S5, the specific process includes the following steps:

[0040] S51, performing a different component mixing detection on a single layer of micro-droplets containing different components, the specific process includes the following steps:

[0041] S511, calculating the distance from the center position of any contour area in the segmented area where the component is located to the center position of the single layer micro droplet , and the radius of the monolayer droplets is ;

[0042] S512: Determine whether there is and only one single layer of micro-droplets satisfy ;

[0043] If not, it is considered that the mixing of different components inside the monolayer microdroplets is complete;

[0044] If yes, proceed to step S513;

[0045] S513: Determine the area of ​​the contour area Is it less than a minimum threshold? ;

[0046] like , it is considered that the mixing of different components inside the monolayer microdroplets is complete;

[0047] like , it is considered that the different components inside the monolayer microdroplets are not mixed completely;

[0048] S52, identifying the merger between different types of monolayer micro-droplets that have merged, the specific process includes the following steps:

[0049] S521. Calculate the center position of any microdroplet in different types of monolayer microdroplets The center position of any microdroplet in another type of single-component microdroplet Distance ;

[0050] S522, according to the distance Determine whether two micro-droplets have completed merging;

[0051] If satisfied , then the two microdroplets complete merging, is a minimum distance threshold;

[0052] S53, detecting the encapsulation and encapsulation states of the inner layer micro-droplets and the outer layer micro-droplets in the double-layer complex micro-droplets.

[0053] Furthermore, in step S53, the specific process includes the following steps:

[0054] S531, calculate the distance between the center position of any inner layer micro-droplet and the center position of each outer layer micro-droplet ;

[0055] S532, according to distance and the approximate radius of the outer droplet Determine whether the inner layer micro-droplets are wrapped normally;

[0056] If all the distances Greater than the approximate radius , then the inner layer of micro-droplets is not wrapped normally;

[0057] If there is only one distance Smaller than the approximate radius , it means that the inner layer of micro-droplets is wrapped normally;

[0058] S533, calculate the distance between the center position of any outer layer micro-droplet and the center position of each inner layer micro-droplet ;

[0059] S534, according to distance and the approximate radius of the outer droplet Determine the number of inner layer micro-droplets enclosed within the outer layer micro-droplets;

[0060] If there is a distance Smaller than the approximate radius , then the number of inner layer micro-droplets wrapped inside the outer layer micro-droplets increases by one;

[0061] If the number of inner-layer micro-droplets wrapped inside the outer-layer micro-droplets does not match the setting, the outer-layer micro-droplet wrapping state is abnormal.

[0062] By means of the above technical solution, the present invention provides a method for micro-droplet image recognition based on color features and a micro-droplet observation device, which have at least the following beneficial effects:

[0063] 1. The present invention simplifies the process of micro-droplet image recognition and detection algorithm, omits the previous background removal and noise removal process, directly segments micro-droplets from the background based on color features, reduces the algorithm difficulty, and solves the recognition difficulty problem caused by the mutual interference between the edges of the inner and outer micro-droplets of the double-layer complex micro-droplets. It also proposes to use different colors on different types of micro-droplets to identify based on color features, solves the problem of multi-type micro-droplet classification that cannot be solved by traditional micro-droplet image recognition and detection methods based on edge and shape features, and can solve the micro-droplet recognition under more working conditions.

[0064] 2. The present invention can realize the detection of conventional single-layer microdroplets, single-layer multi-type microdroplets, mixed detection of different components inside single-layer microdroplets, combined detection of different single-layer microdroplets, detection of double-layer complex microdroplets, and detection of the encapsulation state of inner-layer microdroplets in double-layer complex microdroplets and the number of encapsulated outer-layer microdroplets.

[0065] 3. The present invention can search and locate based on the color features of micro-droplets, and has good robustness against other complex non-target edge features such as noise, occlusion, bubbles, etc. Even multiple targets with similar shapes and areas can be accurately identified. It can be well applied to multi-channel color images and is very suitable for micro-droplet recognition scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0067] Figure 1 It is a structural schematic diagram of the micro-droplet observation device of the present invention;

[0068] Figure 2 This is a result diagram of the single-layer micro-droplet recognition under a simple color background of the present invention;

[0069] Figure 3 This is a result diagram of the single-layer micro-droplet recognition under a complex color background of the present invention;

[0070] Figure 4 This is a result diagram of the identification of different types of single-layer micro-droplets with a complex background according to the present invention;

[0071] Figure 5 This is a result diagram of the identification and detection of the mixing of single-layer micro-droplets of the present invention;

[0072] Figure 6 This is a result diagram of the recognition and detection of the merging of single-layer micro-droplets of the present invention;

[0073] Figure 7 This is a result diagram of the recognition and detection of double-layer complex microdroplets of the present invention. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods, so that the implementation process of how the present application uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0075] Please refer to Figure 1-Figure 7 This embodiment proposes a micro-droplet image recognition method based on color features, which can realize the detection of conventional single-layer micro-droplets, single-layer multi-type micro-droplets, mixed detection of different components inside single-layer micro-droplets, combined detection of different single-layer micro-droplets, detection of double-layer complex micro-droplets, and detection of the encapsulation state of the inner micro-droplets in the double-layer complex micro-droplets and the number of encapsulated outer micro-droplets. The recognition method includes the following steps:

[0076] S1, obtain video image data of the microchannel in the microfluidic chip in the optimal field of view of the color high-speed industrial camera, and convert the video image data into the HSV color space to obtain color images of microdroplets with different color characteristics; the microdroplets include single-layer microdroplets and double-layer complex microdroplets, wherein the single-layer microdroplets include conventional single-layer microdroplets, different types of single-layer microdroplets, single-layer microdroplets containing different components mixed inside, and different types of single-layer microdroplets with mergers, and the double-layer complex microdroplets include inner-layer microdroplets in the double-layer complex microdroplets and outer-layer microdroplets in the double-layer complex microdroplets. In step S1, the specific process includes the following steps:

[0077] S11. With the aqueous phase as the inner phase of the microdroplets and the oil phase as the outer phase of the microdroplets, the inner phase fluid is dyed with a water-soluble pigment to form dispersed phase fluid, so as to generate microdroplets with different colors or different components in the microdroplets; wherein, different types of dispersed phases are used for dyeing respectively, and colors with large color contrast are preferred. After coloring, different dispersed phases will subsequently form microdroplets with different colors or different components in the microdroplets, so one color represents one microdroplet or component.

[0078] S12. Adjust the control platform so that the microchannel in the microfluidic chip is in the best field of view of the color high-speed industrial camera, and adjust the corresponding magnification and light intensity;

[0079] S13, importing the video image data captured by the color high-speed industrial camera into the working computer and converting it into the HSV color space.

[0080] S2, segmenting the color image into a micro-droplet region and a chip background region based on color features, and determining the ownership of the segmented region according to the area of ​​the micro-droplet region and the features of the approximate circle; in step S2, the specific process includes the following steps:

[0081] S21. Determine the value range of color saturation and brightness of non-neutral colors based on the HSV color space characteristics, and select the segmentation threshold for segmenting the micro-droplet area and the chip background area according to the value range; in this step, it can be seen from the HSV color space characteristics that the value range of color saturation (S) of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, [0, 43, 46] is used as the low threshold and [180, 255, 255] is used as the high threshold to segment the micro-droplet color image as a preprocessing to narrow the scope of subsequent image processing. The optimal segmentation threshold selected is [0, 43, 46]-[180, 255, 255].

[0082] S22, segmenting the area with high color saturation and the area with low color saturation in the color image according to the segmentation threshold to form a plurality of segmented areas; in step S22, the area with high color saturation is the dyed micro-droplet, and the area with low color saturation is the chip background, wherein segmenting the micro-droplet and the chip background includes:

[0083] Segmentation of conventional single-layer microdroplets, inner-layer microdroplets in double-layer complex microdroplets and chip background areas, as well as segmentation of different types of single-layer microdroplets, single-layer microdroplets containing different components mixed inside, and different types of single-layer microdroplets with merging and chip background areas;

[0084] If the chip background has no complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from the chip background area with low color saturation according to the segmentation threshold;

[0085] If the chip background has complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from some chip background areas with low color saturation according to the segmentation threshold. Then, according to the two or more peak areas corresponding to the H channel histogram of the HSV color space, the OTSU maximum class spacing method is used to obtain the optimal segmentation threshold between the two or more peak areas, and the micro-droplet area and the chip background area are segmented with the optimal segmentation threshold.

[0086] In this process, the segmentation of conventional single-layer micro-droplets and inner-layer micro-droplets in double-layer complex micro-droplets belongs to a binary classification problem based on color features. Specifically, since the principle of image segmentation is based on color features, the edge features of the chip background, such as complex textures and complex micro-channels, have little effect on image segmentation. Therefore, it is necessary to pay attention to whether the chip background has complex colors. If the chip background has no complex colors, the S and V channels of the HSV color space are used, and the preferred segmentation threshold is [0,43,46]-[180,255,255], which can directly segment the micro-droplets with high color saturation from the chip background with low color saturation;

[0087] If the chip background has complex colors, first use the S and V channels of the HSV color space, preferably with a fixed segmentation threshold of [0,43,46]-[180,255,255], which can directly segment the microdroplets with high color saturation from some chip areas with low color saturation, making the H channel histogram in the color image smoother, and then smooth the H channel for subsequent image segmentation. At this time, the H channel histogram mostly presents two peak areas, one peak area represented by the microdroplets and the other peak area represented by the chip background. The OTSU maximum class spacing method is used to obtain the optimal segmentation threshold between the double peak areas to achieve complete segmentation of the microdroplets and the chip background.

[0088] Among them, the segmentation of different types of monolayer microdroplets, monolayer microdroplets containing different components mixed inside, and different types of monolayer microdroplets with mergers belongs to the multi-classification problem based on color features, that is, microdroplets containing different colors or different components in microdroplets. Therefore, the segmentation method adopted is the same as when the chip background has complex colors. The difference is that the H channel histogram mostly presents multiple peak areas, and different types (N types) of microdroplets and the background each have a peak area (a total of N+1). The maximum class spacing method can also be used to obtain the optimal segmentation threshold between multiple peak areas to achieve complete segmentation between different types of microdroplets and between microdroplets and chip background areas.

[0089] Similarly, the first frame image in the micro-droplet video is used to determine the N thresholds [T1, T2, T3, ..., TN] of the multi-peak region segmentation problem, and then the color image in the subsequent video is segmented based on the fixed thresholds [0, T1], [T1, T2], ..., [TN, 180] to save image segmentation time. It is easy for technicians in this field to understand that the method of obtaining the optimal segmentation threshold between multi-peak regions using the maximum class spacing method mentioned above can also be combined with a recursive algorithm, a genetic algorithm, etc. to further improve the efficiency of image segmentation. It also includes segmenting the outer layer of the double-layer complex micro-droplets and the chip background area.

[0090] Specifically, there is still the problem of segmenting the outer layer of micro-droplets in the double-layer complex micro-droplets. In view of this problem, due to the need to observe the inner layer of micro-droplets, the intermediate phase that produces the outer layer of micro-droplets is generally not dyed, so the edge features of the outer layer of micro-droplets are used for image segmentation. First, the color image is processed into a grayscale image, and the edge contour of the outer layer of micro-droplets is preferably obtained by using the Sobel or Laplacian edge detection operator. At this time, based on the roundness of the edge contour, it is judged whether there are micro-droplets in the grayscale image, so that the non-micro-droplet area is excluded by binarization and based on the roundness judgment result, and the complete segmentation of the edge contour of the micro-droplet and the background is achieved. In particular, here, the inner layer (color features) and outer layer (shape and edge features) micro-droplets are identified separately based on different features, which effectively avoids the problem of droplet recognition failure caused by strong interference such as the overlap of the two layers of droplet contours, and can obtain independent information on the inner and outer layer micro-droplet areas, which is convenient for subsequent identification and detection.

[0091] S23. Determine whether a number of segmented areas are micro-droplet areas or chip background areas; if a segmented area among the segmented areas is larger and has a roundness less than 0.7, it is the chip background area; otherwise, it is the micro-droplet area. Among them, since the total area of ​​the chip background area is larger than the micro-droplet area, the total area of ​​the segmented areas is used as a criterion to determine that the smaller area is the micro-droplet area. In addition, in order to avoid errors in the case of more complex working conditions, combined with the shape characteristics of the micro-droplets that are approximately circular, the existence of a segmented area with a roundness less than 0.7 can be used as a criterion for the chip background area for auxiliary judgment.

[0092] S3. Post-process the segmented micro-droplet area to eliminate the noise points generated after image segmentation. After determining the micro-droplet area, use median filtering to remove some noise points that may be generated after image segmentation, and perform morphological corrosion and morphological dilation to further achieve small area removal. Specifically, first take the Kernel as (3,3), use image morphological corrosion to eliminate the noise influence generated in image segmentation, and then use image morphological dilation to ensure that the detection size is as accurate as possible, and the number of iterations of corrosion and dilation are both 1.

[0093] S4. Identify and mark the shape characteristics, center position and motion state information of the micro-droplets based on the post-processed micro-droplet area; for color images of micro-droplets in all working conditions, it is necessary to identify important information such as the shape characteristics, center position and motion characteristics of the micro-droplets according to the following method. In addition, for single-layer micro-droplets containing a mixture of different components, the segmented images of all different components inside the single-layer micro-droplets should be summed to form an accurate micro-droplet area. For different types of single-layer micro-droplets, it is also necessary to indicate the type of micro-droplets to which they belong based on the color image segmentation results between the micro-droplets. In order to eliminate the interference of irrelevant edges and shape features in complex backgrounds, realize the recognition and extraction of target micro-droplets, and the recognition and classification of micro-droplets of different components, in step S4, the specific process includes the following steps:

[0094] S41, traverse all contours in the micro-droplet area and calculate the roundness of the contours , the calculation formula is:

[0095] ;

[0096] in, represents the area of ​​the i-th contour; is the perimeter of the i-th contour;

[0097] S42, according to roundness Determine whether the contour is a target micro-droplet contour area;

[0098] like , then the contour is the target micro-droplet contour area, and the process goes to step S43;

[0099] like , then return to step S41, wherein the target threshold M is 0.7 in this embodiment;

[0100] S43, performing ellipse fitting on the target micro-droplet contour area to obtain shape features, and marking it as a micro-droplet;

[0101] S44, taking the average value of the major axis and the minor axis after the ellipse fitting as the approximate radius of the micro-droplet, and taking the central position of the target micro-droplet contour area as the central position of the micro-droplet;

[0102] S45, using the inter-frame difference method to analyze the video image data of the micro-droplet region frame by frame to obtain the center position of the micro-droplet on two adjacent color image frames; wherein the minimum adjacent distance is used as the criterion for frame association, that is, the two micro-droplets with the smallest distance between the center positions of the micro-droplets in two adjacent frames are the same micro-droplet;

[0103] S46, obtaining the motion state information of the micro-droplet according to the center position of the micro-droplet in two adjacent color images, that is, the moving distance and direction of the center position of the same micro-droplet within one frame time.

[0104] In this embodiment, the roundness feature is first used to determine whether it is the target micro-droplet contour area, and then the target micro-droplet contour area is ellipse-fitted to obtain the size of the micro-droplet. At the same time, the micro-droplet morphology is detected based on the roundness of the ellipse fitting. Subsequently, the center position of the micro-droplet area is selected as the position coordinate of the micro-droplet to detect the motion state of the micro-droplet in the microchannel. The inter-frame difference method is used for the motion video of the micro-droplet, and the minimum adjacent distance is used as the criterion for frame association, that is, the two micro-droplets with the smallest distance between the center positions of the micro-droplets between two adjacent frames are considered to be the same micro-droplet, and then the moving distance and direction of the center position of the same micro-droplet within one frame time are obtained, so as to realize the monitoring of the motion behavior of the micro-droplet and obtain the motion state information of the micro-droplet, including the motion speed, motion direction, motion situation, etc.

[0105] S5, identify and detect single-layer micro-droplets containing a mixture of different components, single-layer micro-droplets of different types that are merged, and double-layer complex micro-droplets; for more complex micro-droplet conditions such as single-layer micro-droplets containing a mixture of different components, single-layer micro-droplets of different types that are merged, and double-layer complex micro-droplets, it is also necessary to detect additional indicators based on the identified information according to the following method to achieve comprehensive detection and monitoring of the quality of complex micro-droplet generation. In step S5, the specific process includes the following steps:

[0106] S51, performing different component mixing detection on a single layer of micro-droplets containing different components; in order to identify the mixing of different components inside the micro-droplets, in step S51, the specific process includes the following steps:

[0107] S511, calculating the distance from the center position of any contour area in the segmented area where the component is located to the center position of the single layer micro droplet , and the radius of the monolayer droplets is ;

[0108] S512: Determine whether there is one and only one single layer of micro-droplets satisfy ;

[0109] If not, it is considered that the mixing of different components inside the monolayer microdroplets is complete;

[0110] If yes, proceed to step S513;

[0111] S513: Determine the area of ​​the contour area Is it less than a minimum threshold? ;

[0112] like , it is considered that the mixing of different components inside the monolayer microdroplets is complete;

[0113] like , it is considered that the different components inside the monolayer microdroplets are not mixed completely;

[0114] Taking the position and area of ​​any one or more components in the different components inside the monolayer microdroplet as indicators, the center position and area of ​​all contour areas in the segmented area of ​​the component are calculated one by one. For each contour area, the distance between its center position and the center position of the monolayer microdroplet is first calculated. The center position of the single layer of micro-droplets is the center position of the target micro-droplet contour area in step S44. The radius of the single layer of micro-droplets is , to determine whether there is one and only one monolayer of microdroplets satisfy If not, it is considered that the different components in the single-layer micro-droplet are mixed; if so, the area of ​​the contour area is further determined. Is it less than a minimum threshold? ,like , it is considered that the mixing of different components inside the single layer of microdroplets is complete.

[0115] S52, identifying the merger between different types of single-layer micro-droplets that have merged. In order to realize the identification of the merger between different micro-droplets, in step S52, the specific process includes the following steps:

[0116] S521. Calculate the center position of any microdroplet in different types of monolayer microdroplets The center position of any microdroplet in another type of single-component microdroplet Distance ;

[0117] S522, according to the distance Determine whether two micro-droplets have completed merging;

[0118] If satisfied , then the two micro-droplets complete merging;

[0119] Taking the center position and radius of different monolayer microdroplets as indicators, for each microdroplet in one type of monolayer microdroplets, the distance between its center position and the center position of each microdroplet in another type of single-component microdroplets is calculated. , if satisfied ,in is a minimum distance threshold. If yes, it means that the two micro-droplets have completed merging.

[0120] S53, the wrapped and wrapped states of the inner layer micro-droplets and the outer layer micro-droplets in the double-layer complex micro-droplets are detected; in order to eliminate the interference of the overlapped edges of the inner and outer layer droplets of the double-layer complex micro-droplets, the number and size of the inner and outer layer micro-droplets in the double-layer complex micro-droplets can be simultaneously identified, as well as the micro-droplets of different components in the inner layer, and the successful wrapping and rupture of the inner layer droplets can be detected, so as to achieve comprehensive detection and monitoring of the quality of the double-layer complex micro-droplet generation. In step S53, the specific process includes the following steps:

[0121] S531, calculate the distance between the center position of any inner layer micro-droplet and the center position of each outer layer micro-droplet ;

[0122] S532, according to distance and the approximate radius of the outer droplet Determine whether the inner layer micro-droplets are wrapped normally;

[0123] If all the distances Greater than the approximate radius , then the inner layer of micro-droplets is not wrapped normally;

[0124] If there is only one distance Smaller than the approximate radius , it means that the inner layer of micro-droplets is wrapped normally;

[0125] S533, calculate the distance between the center position of any outer layer micro-droplet and the center position of each inner layer micro-droplet ;

[0126] S534, according to distance and the approximate radius of the outer droplet Determine the number of inner layer micro-droplets enclosed within the outer layer micro-droplets;

[0127] If there is a distance Smaller than the approximate radius , then the number of inner layer micro-droplets wrapped inside the outer layer micro-droplets increases by one;

[0128] If the number of inner-layer micro-droplets wrapped inside the outer-layer micro-droplets does not match the setting, the outer-layer micro-droplet wrapping state is abnormal.

[0129] Specifically, the center position and approximate radius of the inner and outer layer micro-droplets are used as indicators, that is:

[0130] Inner layer micro-droplet status: judged by whether there are outer layer micro-droplets. For each inner layer micro-droplet, calculate the distance between its center position and the center position of each outer layer micro-droplet , and the approximate radius of the outer microdroplet Compare, if all Both greater than , it means that the inner layer of micro-droplets is not wrapped properly. If there is only one Less than , it means that the inner layer of micro-droplets are wrapped normally.

[0131] Outer layer micro-droplet status: mainly detect the number of inner layer micro-droplets wrapped inside, and for each outer layer micro-droplet, calculate the distance between its center position and the center position of each inner layer micro-droplet , which is also related to the radius of the outer micro-droplet Compare, if there is a Less than , the number of the inner layer micro-droplets of the inner package increases by one. It is particularly noted that when the number of the inner layer micro-droplets of the inner package does not match the setting, the outer layer micro-droplet packaging state is abnormal.

[0132] In practical applications, this micro-droplet image recognition method can realize the detection of conventional single-layer micro-droplets, single-layer multi-type micro-droplets, mixed detection of different components inside single-layer micro-droplets, combined detection of different single-layer micro-droplets, detection of double-layer complex micro-droplets, and detection of the encapsulation state of inner micro-droplets in double-layer complex micro-droplets and the number of droplets encapsulated by outer micro-droplets. In addition, due to the robustness of the color feature itself, taking the conventional single-layer micro-droplet image recognition as an example, the segmentation accuracy (P) of this method can reach 96%, and the recall rate (R) can reach more than 95%. The deviations caused by changes in the three working conditions of color type, light intensity, and color saturation are all within 5%.

[0133] like Figure 1 As shown, this embodiment provides a micro-droplet observation device applied to the above-mentioned micro-droplet image recognition method, including a microfluidic system and an observation system.

[0134] The microfluidic system is used to adjust the flow rate of the injection pump and control the microfluidic chip to generate microdroplets. The microfluidic system consists of a injection pump, a control platform and a microfluidic chip. The microfluidic chip is fixed on the control platform. For easy observation, the material of the microfluidic chip should be a material with good light transmittance. In this embodiment, photosensitive resin is used, and the molding is based on light-curing 3D printing (SLA).

[0135] The observation system is used to observe and record the microdroplets generated by the microfluidic chip. The observation system consists of a light source and a color high-speed industrial camera. The light source is preferably a white parallel light source, and the hardware core is a color high-speed industrial camera (HikRobot MV-CH050-10UC), which is used to observe and record the microdroplets.

[0136] According to the micro-droplet image recognition method and micro-droplet observation device based on color features proposed in the above embodiments, conventional single-layer micro-droplets, single-layer multi-type micro-droplets, mixed detection of different components inside single-layer micro-droplets, merging detection of different single-layer micro-droplets, detection of double-layer complex micro-droplets, and detection of the encapsulation state of the inner micro-droplets in the double-layer complex micro-droplets and the number of encapsulated outer micro-droplets are detected. Please refer to Figures 2 to 7 This embodiment can search and locate based on the color features of micro-droplets, and has good robustness to other complex non-target edge features such as noise, occlusion, bubbles, etc. Even multiple targets with similar shapes and areas can be accurately identified, which can be well applied to multi-channel color images and is very suitable for micro-droplet recognition scenarios. The specific implementation methods for the above-mentioned various micro-droplet working condition detection are as follows: Embodiment 1

[0137] Identification and detection of conventional monolayer microdroplets against a simple color background

[0138] This embodiment takes the recognition of a single layer of micro-droplets under a transparent chip background with normal color as an example, with the water phase as the inner phase and the oil phase as the outer phase. First, the inner phase fluid should be dyed with a water-soluble pigment (a non-neutral color should be selected) to ensure that the dispersed phase fluid is dyed uniformly, and the control platform is adjusted so that the microchannel in the microfluidic chip should be in the best field of view of the color high-speed camera. At the same time, the corresponding magnification and light intensity should be adjusted for subsequent micro-droplet image recognition and detection. The video in the field of view of the color high-speed industrial camera is imported into the working computer and converted into the HSV color space.

[0139] Conventional single-layer micro-droplet image segmentation based on color features:

[0140] Since the chip background is simple, no image preprocessing is required. It can be directly known from the HSV color space characteristics that the color saturation (S) of non-neutral colors is in the range of [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, [0, 43, 46] is used as the low threshold and [180, 255, 255] is used as the high threshold to segment the color image of the micro-droplet area and the chip background area. The area of ​​the micro-droplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the micro-droplet area, the total area of ​​the segmented area is used as the criterion to determine that the smaller area is the micro-droplet area. In addition, in order to avoid errors in the case of more complex working conditions, combined with the shape characteristics of the micro-droplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as the criterion for the chip background area for auxiliary judgment.

[0141] Image post-processing after image segmentation:

[0142] First, the kernel is taken as (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0143] Microdroplet feature information identification and marking:

[0144] For the micro-droplet region after the image segmentation, all the contours of this region are traversed. Due to the characteristic of the micro-droplet approximate circle, the contour roundness is greater than 0.7 as the criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgment. Taking the present embodiment as an example, the main criterion is that the contour roundness is greater than 0.7, and the criterion of auxiliary judgment is that the minimum contour area is greater than 100 pixels, and the maximum contour area is less than 5000 pixels. For each determined micro-droplet contour, due to the fact that micro-droplets are easily squeezed under normal circumstances, they are often shown as elliptical, so ellipse fitting is adopted to mark micro-droplets, and the mean value of the major axis and the minor axis after the ellipse fitting is used as the approximate radius of this micro-droplet. In addition, the position of this micro-droplet is represented by the contour area center position. For each frame image, the position of all micro-droplets on the image can be obtained. For the position of each micro-droplet on the adjacent two frames of images, the distance between each other is calculated, and it is believed that the two micro-droplets with the minimum distance on the two frames of images are the same micro-droplet, so as to infer the motion speed and motion direction of the micro-droplet. Finally, the identification result of each micro-droplet feature information is marked next to the micro-droplet, such as Figure 2 shown. Embodiment 2

[0145] Recognition and detection of conventional monolayer microdroplets against complex color background

[0146] This embodiment takes the identification of monolayer micro-droplets with poor color as an example, with the water phase as the inner phase and the oil phase as the outer phase. First, the inner phase fluid should be dyed with water-soluble pigments (non-neutral colors should be selected) to ensure uniform dyeing of the dispersed phase fluid. The control platform is adjusted so that the microchannels in the microfluidic chip should be in the best field of view of the color high-speed camera. At the same time, the corresponding magnification and light intensity should be adjusted for subsequent micro-droplet image recognition and detection. The video in the field of view of the color high-speed industrial camera is imported into the working computer and converted into the HSV color space.

[0147] Image preprocessing:

[0148] From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, [0, 43, 46] is used as the low threshold and [180, 255, 255] is used as the high threshold to segment the micro-droplet color image as a preprocessing to narrow the scope of subsequent image processing.

[0149] Conventional single-layer micro-droplet image segmentation based on color features:

[0150] First, image segmentation is performed based on color features. For the preprocessed micro-droplet color image, its HSV channel is separated. At this time, the H channel of the micro-droplet color image is a typical bimodal problem. The OTSU maximum class spacing is used to determine its optimal segmentation threshold to achieve the segmentation of the micro-droplet area and the chip background area. Secondly, the area of ​​the micro-droplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the micro-droplet area, the total area of ​​the segmented area is used as a criterion to determine that the smaller area is the micro-droplet area. In addition, in order to avoid errors in the case of more complex working conditions, combined with the shape characteristics of the micro-droplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as a criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine its optimal segmentation threshold T based on the first frame of the image, and the threshold T is directly used for image segmentation for each subsequent frame to save the program running time for each subsequent frame.

[0151] Image post-processing after image segmentation:

[0152] First, the kernel is taken as (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0153] Microdroplet feature information identification and marking:

[0154] For the micro-droplet region after the image segmentation, all the contours of this region are traversed. Due to the characteristic of the micro-droplet approximate circle, the contour roundness is greater than 0.7 as the criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgment. Taking the present embodiment as an example, the main criterion is that the contour roundness is greater than 0.7, and the criterion of auxiliary judgment is that the minimum contour area is greater than 100 pixels, and the maximum contour area is less than 5000 pixels. For each determined micro-droplet contour, due to the fact that micro-droplets are easily squeezed under normal circumstances, they are often shown as elliptical, so ellipse fitting is adopted to mark micro-droplets, and the mean value of the major axis and the minor axis after the ellipse fitting is used as the approximate radius of this micro-droplet. In addition, the position of this micro-droplet is represented by the contour area center position. For each frame image, the position of all micro-droplets on the image can be obtained. For the position of each micro-droplet on the adjacent two frames of images, the distance between each other is calculated, and it is believed that the two micro-droplets with the minimum distance on the two frames of images are the same micro-droplet, so as to infer the motion speed and motion direction of the micro-droplet. Finally, the identification result of each micro-droplet feature information is marked next to the micro-droplet, such as Figure 3 shown. Embodiment 3

[0155] Identification and detection of various types of monolayer microdroplets, such as Figure 4 The difficulty lies in the serious squeezing of the two different types of microdroplets, the complex channel edges and the large amount of interference, which makes identification difficult.

[0156] This embodiment takes the recognition of two monolayer micro-droplets as an example, with the water phase as the inner phase and the oil phase as the outer phase. First, the two inner phase fluids should be dyed with water-soluble pigments (non-neutral colors with large color difference are selected, green and red are used in this case), and the color in this case is not good. Ensure that the dispersed phase fluid is dyed uniformly, adjust the control platform so that the microchannel in the microfluidic chip should be in the best field of view of the color high-speed camera, and adjust the corresponding magnification and light intensity for subsequent micro-droplet image recognition detection. Import the video in the field of view of the color high-speed industrial camera into the working computer and convert it to the HSV color space.

[0157] Image preprocessing based on color features:

[0158] From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, [0, 43, 46] is used as the low threshold and [180, 255, 255] is used as the high threshold to segment the micro-droplet color image as a preprocessing to narrow the scope of subsequent image processing.

[0159] Micro-droplet image segmentation based on color features:

[0160] First, image segmentation is performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. At this time, the H channel of the microdroplet color image is a three-peak problem. The OTSU maximum class distance is used to determine the optimal segmentation thresholds, which are T1 and T2 respectively. This threshold is used to achieve segmentation between different types of microdroplets and between the microdroplet area and the chip background area. Secondly, the area of ​​the microdroplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the microdroplet area, the total area of ​​the segmented area is used as a criterion to determine that the smaller area is the microdroplet area. In addition, in order to avoid errors in more complex working conditions, combined with the shape characteristics of the microdroplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as a criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine the optimal segmentation thresholds T1 and T2 based on the first frame image. For each subsequent frame, the thresholds [0, 43, 46]–[T1,255,55], [T1,255,255]–[T2,255,255], [T2,255,255]–[180,255,255] are directly used for image segmentation to save the program running time for each subsequent frame.

[0161] Image post-processing after image segmentation:

[0162] First, the kernel is taken as (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0163] Microdroplet feature information identification and marking:

[0164] For the micro-droplet area after image segmentation, all contours of the area are traversed. Due to the characteristic of micro-droplets being approximate circles, the contour roundness greater than 0.7 is used as a criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgments. Taking this embodiment as an example, the main criterion is that the contour roundness is greater than 0.7, and the auxiliary judgment criteria are that the minimum contour area is greater than 20 pixels and the maximum contour area is less than 500 pixels. For each determined micro-droplet contour, since micro-droplets are generally easily squeezed and often appear as ellipses, ellipse fitting is used to mark the micro-droplets, and the average value of the major axis and the minor axis after ellipse fitting is used as the approximate radius of the micro-droplet, and the center position of the contour area represents the position of the micro-droplet. In addition, for contours in different segmented images, different colors are used to mark them in the original image, and the type of droplet to which they belong is indicated. Finally, as shown in the figure, Figure 4Similarly, for each frame of image, the positions of all micro-droplets on the image can be obtained. For the position of each micro-droplet on two adjacent frames of image, the distance between them is calculated. It is considered that the two micro-droplets with the smallest distance between the two frames of image are the same micro-droplet, so as to infer the movement speed and direction of the micro-droplet. Embodiment 4

[0165] Identification and detection of mixing of different components inside monolayer microdroplets

[0166] The difficulty of this implementation case is that the edges of the component regions are irregular and change rapidly during the component mixing process. If the color feature is not used, it is impossible to accurately extract the component regions. This is impossible to achieve with edge detection.

[0167] This embodiment takes the identification of the mixture of two different components inside a monolayer micro-droplet as an example, with the water phase as the inner phase and the oil phase as the outer phase. First, the inner phase fluid should be dyed with a water-soluble pigment (non-neutral colors are selected, green and purple are used in this embodiment) to ensure that the dispersed phase fluid is dyed uniformly, and the control platform is adjusted so that the microchannel in the microfluidic chip should be in the best field of view of the color high-speed camera. At the same time, the corresponding magnification and light intensity should be adjusted for subsequent micro-droplet image recognition and detection. The video in the field of view of the color high-speed industrial camera is imported into the working computer and converted into the HSV color space.

[0168] Image preprocessing based on color features:

[0169] From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, [0, 43, 46] is used as the low threshold and [180, 255, 255] is used as the high threshold to segment the micro-droplet color image as a preprocessing to narrow the scope of subsequent image processing.

[0170] Segmentation of target droplets and background based on color features:

[0171] First, image segmentation is performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. At this time, the H channel of the microdroplet color image is a three-peak problem. The OTSU maximum class distance is used to determine the optimal segmentation thresholds, which are T1 and T2 respectively. This threshold is used to achieve segmentation between different components inside the microdroplet and between the microdroplet area and the chip background area. Secondly, the area of ​​the microdroplet area is used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the microdroplet area, the total area of ​​the segmented area is used as the criterion to determine that the smaller area is the microdroplet area. In addition, in order to be able to identify the complete microdroplet contour later, all the segmented images of the different color component areas are summed to obtain a complete microdroplet area segmentation image. At the same time, in order to avoid errors in the case of more complex working conditions, combined with the shape characteristics of the microdroplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as the criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine the optimal segmentation thresholds T1 and T2 based on the first frame image. For each subsequent frame, the thresholds [0,43,46]–[T1,255,255], [T1,255,255]–[T2,255,255], [T2,255,255]–[180,255,255] are directly used for image segmentation to save the program running time for each subsequent frame.

[0172] Image post-processing after image segmentation:

[0173] First, the kernel is taken as (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0174] Identification and marking of micro-droplet characteristic information:

[0175] For the micro-droplet area after image segmentation, first merge the segmented images of different components to obtain the actual single-layer micro-droplet area segmented image, traverse all the contours of the area, and because the micro-droplet is approximately circular, the contour roundness greater than 0.7 is used as a criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgments. Taking this embodiment as an example, the main criterion is that the contour roundness is greater than 0.7, and the auxiliary judgment criterion is that the minimum contour area is greater than 100 pixels and the maximum contour area is less than 1000 pixels. For each determined micro-droplet contour, because micro-droplets are generally easily squeezed and often appear as ellipses, ellipse fitting is used to mark the micro-droplets, and the average value of the major axis and the minor axis after ellipse fitting is used as the approximate radius of the micro-droplet. In addition, the center position of the contour area represents the position of the micro-droplet. For each frame of image, the positions of all micro-droplets on the image can be obtained. For the position of each micro-droplet in two adjacent frames of image, the distance between them is calculated. It is considered that the two micro-droplets with the smallest distance between the two frames of image are the same micro-droplet, thereby inferring the movement speed and direction of the micro-droplets.

[0176] Detection of micro-droplet mixing:

[0177] Take the purple component area in the single layer of microdroplets and detect the center position, radius and area of ​​all contours Traverse all the microdroplet contours in the microdroplet area and calculate the distance between the center position of each purple component contour area and the center position of the microdroplet area , determine the distance The radius of the microdroplet The size of Both greater than , indicating that the micro-droplet mixing is complete. If there is Less than , indicating that the mixing of the micro-droplets is not complete. Mark the droplets that have been mixed on the original image, such as Figure 5 shown. Embodiment 5

[0178] Identification and detection of merging of micro-droplets of different monolayers. Droplet merging belongs to the problem of edge overlap, which is a detection target that is difficult to achieve with traditional edge feature-based detection algorithms. However, the method proposed in this embodiment is simpler and can accurately identify.

[0179] This embodiment takes the merging and identification of two different monolayer micro-droplets as an example, with the water phase as the inner phase and the oil phase as the outer phase. First, the inner phase fluid should be dyed with water-soluble pigments (non-neutral colors are selected, and green and purple are used in this embodiment) to ensure that the dispersed phase fluid is dyed evenly. The control platform is adjusted so that the microchannel in the microfluidic chip should be in the best field of view of the color high-speed camera. At the same time, the corresponding magnification and light intensity should be adjusted for subsequent micro-droplet image recognition and detection. The video in the field of view of the color high-speed industrial camera is imported into the working computer and converted into the HSV color space.

[0180] Image preprocessing based on color features:

[0181] From the HSV color space characteristics, we can see that the color saturation (S) value range of non-neutral colors is [43, 255] and the value range of brightness (V) is [46, 255]. The micro-droplet area after coloring must be in this area. Therefore, [0, 43, 46] is used as the low threshold and [180, 255, 255] is used as the high threshold to segment the micro-droplet color image as a preprocessing to narrow the scope of subsequent image processing.

[0182] Segmentation of target droplets and background based on color features:

[0183] First, image segmentation is performed based on color features. For the preprocessed microdroplet color image, its HSV channel is separated. At this time, the H channel of the microdroplet color image is a three-peak problem. The OTSU maximum class distance is used to determine the optimal segmentation thresholds, which are T1 and T2 respectively. This threshold is used to achieve segmentation between different types of microdroplets and between the microdroplet area and the chip background area. Secondly, the area of ​​the microdroplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the microdroplet area, the total area of ​​the segmented area is used as a criterion to determine that the smaller area is the microdroplet area. In addition, in order to avoid errors in more complex working conditions, combined with the shape characteristics of the microdroplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as a criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine the optimal segmentation thresholds T1 and T2 based on the first frame image. For each subsequent frame, the thresholds [0,43,46]–[T1,255,255], [T1,255,255]–[T2,255,255], [T2,255,255]–[180,255,255] are directly used for image segmentation to save the program running time for each subsequent frame.

[0184] Microdroplet characteristic information acquisition:

[0185] For two kinds of micro-droplet regions after the image segmentation, all the contours of the respective regions are traversed. Due to the characteristic of the micro-droplet approximate circle, the contour roundness is adopted to be greater than 0.7 as the criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgment. Taking the present embodiment as an example, the main criterion is that the contour roundness is greater than 0.7, and the criterion of auxiliary judgment is that the minimum contour area is greater than 20 pixels, and the maximum contour area is less than 500 pixels. For each determined micro-droplet contour, due to the fact that micro-droplet is easily squeezed under normal circumstances, it is usually shown as an ellipse, so ellipse fitting is adopted to mark micro-droplet, and the mean value of the major axis and the minor axis after the ellipse fitting is used as the approximate radius of the micro-droplet. In addition, the position of the micro-droplet is represented by the contour region center position. For each frame image, the position of all micro-droplets on the image can be obtained. For the position of each micro-droplet on the adjacent two frames of images, the distance between each other is calculated, and it is believed that the two micro-droplets with the minimum distance on the two frames of images are the same micro-droplet, so as to infer the motion speed and the motion direction of the micro-droplet.

[0186] Image post-processing after image segmentation:

[0187] First, the kernel is taken as (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0188] Detection of microdroplet merging:

[0189] Taking the center position and radius of different monolayer microdroplets as indicators, for each microdroplet in one type of monolayer microdroplets, the distance between its center position and the center position of each microdroplet in another type of single-component microdroplets is calculated. , if satisfied , then is a minimum threshold. If yes, it means that the two micro-droplets have merged. In this embodiment, . Then mark it in the original image. Finally, Figure 6 shown. Embodiment 6

[0190] Identification and detection of double-layer complex micro-droplet detection

[0191] This embodiment takes a double-layer complex micro-droplet with different numbers of micro-droplets wrapped inside as an example, and the background is complex and the color is poor. It has complex droplet edge features, and the edges of the inner and outer droplets overlap, so it is difficult to accurately identify them using edge features. The water phase is the inner and outer phases, and the oil phase is the intermediate phase. First, the inner phase fluid should be dyed with a water-soluble pigment (a non-neutral color is selected, and brown is used in this implementation case) to ensure that the dispersed phase fluid is evenly dyed, and the control platform is adjusted so that the microchannel in the microfluidic chip should be in the best field of view of the color high-speed camera. At the same time, the corresponding magnification and light intensity should be adjusted for subsequent micro-droplet image recognition and detection. The video in the field of view of the color high-speed industrial camera is imported into the working computer and converted into the HSV color space.

[0192] Identify inner layer droplets based on color features:

[0193] First, preprocessing is performed. From the characteristics of the HSV color space, it can be seen that the color saturation (S) of non-neutral colors is in the range of [43, 255] and the value range of brightness (V) is [46, 255]. The inner micro-droplet area after coloring must be in this area. Therefore, the micro-droplet color image is segmented with [0, 43, 46] as the low threshold and [180, 255, 255] as the high threshold as preprocessing to narrow the scope of subsequent image processing.

[0194] Then, image segmentation is performed based on color features. For the preprocessed micro-droplet color image, its HSV channel is separated. At this time, the H channel of the micro-droplet color image is a typical bimodal problem. The OTSU maximum class spacing is used to determine its optimal segmentation threshold to achieve the segmentation of the micro-droplet area and the chip background area. Secondly, the area of ​​the micro-droplet area and the characteristics of the approximate circle are used to determine the ownership of the segmented area. Since the total area of ​​the chip background area is larger than the micro-droplet area, the total area of ​​the segmented area is used as a criterion to determine that the smaller area is the micro-droplet area. In addition, in order to avoid errors in the case of more complex working conditions, combined with the shape characteristics of the micro-droplet approximate circle, the existence of a contour area with a roundness less than 0.7 can be used as a criterion for the chip background area for auxiliary judgment. For video detection, the above method can be used to determine its optimal segmentation threshold T based on the first frame of the image, and the threshold T is directly used for image segmentation for each subsequent frame to save the program running time for each subsequent frame.

[0195] Identify outer layer micro-droplets based on edge features:

[0196] Based on the original image (non-HSV color space), after graying, the Sobel edge operator is used to identify the edges in the micro-droplet image (Sobel and Laplacian operators are preferred, and the effect is better than the Canny operator). Based on the high roundness of the micro-droplet edge, all contours are traversed, and the contour roundness greater than 0.7 is used as the criterion. In order to more accurately identify each micro-droplet, the minimum contour area and the maximum contour area can be added as auxiliary judgments. Taking this embodiment as an example, the main criterion is that the contour roundness is greater than 0.7, and the auxiliary judgment criterion is that the minimum contour area is greater than 500 pixels and the maximum contour area is less than 10000 pixels. In addition, due to the characteristics of droplet microfluidics, if there are stains on the microfluidic chip, or the observation surface of the microfluidic chip is not flat, etc., it may cause unstable edge recognition (such as the detected microdroplet edge is incomplete and is judged not to be a microdroplet contour). Hough circle detection can be used for supplementary recognition. The microdroplet contour identified in the Hough circle detection and the microdroplet contour identified by the edge detection are compared. If the position deviation of the two is less than Min pixels, the edge detection result is retained. If there is a position deviation of the two that is greater than the approximate radius of the edge detection microdroplet, it is considered that the edge detection is missed, and the contour of the Hough circle detection is supplemented as a microdroplet recognition result. In this embodiment, Hough circle detection auxiliary recognition is not required for the time being.

[0197] Image post-processing after image segmentation:

[0198] First, the kernel is taken as (3,3), and image morphological erosion is used to eliminate the influence of noise generated in image segmentation. Then, image morphological dilation is used to ensure that the detection size is as accurate as possible. The number of iterations of erosion and dilation is 1.

[0199] Inner and outer micro-droplet status detection:

[0200] In this embodiment, since the inner and outer layer micro-droplet detection processes are completely independent and do not affect each other, completely separate inner layer micro-droplet information and outer layer micro-droplet information can be obtained. Based on the distance between the inner layer micro-droplet and the outer layer micro-droplet, the wrapped state of the inner layer micro-droplet and the number of droplets contained in the outer layer micro-droplet are judged. First, for each inner layer micro-droplet, the distance between it and each outer layer micro-droplet is calculated. , and compare it with the radius of the outer micro-droplet Compare, if there is a Less than the corresponding , then the wrapping state is normal, otherwise it is abnormal. Initially, the number of droplets inside each outer layer micro-droplet is set to zero. For each outer layer micro-droplet, calculate its distance to each inner layer micro-droplet. , and compare it with the radius of the outer micro-droplet Compare, if there is a Less than the corresponding , then the number of droplets encapsulated by the outer layer micro-droplet increases by one until all inner layer micro-droplets are traversed, and the number of droplets encapsulated inside the outer layer micro-droplet is output. Figure 7 shown.

[0201] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, so the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0202] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the above embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0203] The above implementation methods have been described in detail. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for micro-droplet image recognition based on color features, characterized in that: The method comprises the following steps: S1, obtaining video image data of a microchannel in a microfluidic chip within the optimal field of view of a color high-speed industrial camera, and converting the video image data into an HSV color space to obtain a color image of microdroplets with different color features; S2, segmenting the color image into a micro-droplet region and a chip background region based on color features, and determining the ownership of the segmented region according to the area of ​​the micro-droplet region and the features of the approximate circle. The specific process includes the following steps: S21, determining the value range of color saturation and brightness of the non-neutral color based on the HSV color space characteristics, and selecting a segmentation threshold for segmenting the micro-droplet area and the chip background area according to the value range; S22, segmenting the area with high color saturation and the area with low color saturation in the color image according to the segmentation threshold to form a plurality of segmented areas; S23, determining whether a number of segmented regions belong to micro-droplet regions or chip background regions; If the area of ​​several segmented regions is larger and the segmented region with a circularity less than 0.7 is the chip background region; otherwise, it is the micro-droplet region; S3, performing post-processing on the segmented micro-droplet region to eliminate noise points generated after image segmentation; S4, identifying and marking the shape characteristics, center position and motion state information of the micro-droplet according to the post-processed micro-droplet area, and the specific process includes the following steps: S41, traverse all contours in the micro-droplet area and calculate the roundness of the contours , the calculation formula is: ； in, represents the area of ​​the i-th contour; is the perimeter of the i-th contour; S42, according to roundness Determine whether the contour is a target micro-droplet contour area; like , then the contour is the target micro-droplet contour area, and the process goes to step S43; like , then return to step S41, is the target threshold; S43, performing ellipse fitting on the target micro-droplet contour area to obtain shape features, and marking it as a micro-droplet; S44, taking the average value of the major axis and the minor axis after the ellipse fitting as the approximate radius of the micro-droplet, and taking the central position of the target micro-droplet contour area as the central position of the micro-droplet; S45, using an inter-frame difference method to analyze the video image data of the micro-droplet region frame by frame to obtain the center position of the micro-droplet on two adjacent color image frames; S46, obtaining the motion state information of the micro-droplet according to the center position of the micro-droplet on two adjacent color images, that is, the moving distance and direction of the center position of the same micro-droplet within one frame time; S5. Identify and detect single-layer microdroplets containing a mixture of different components, single-layer microdroplets of different types with merging, and double-layer complex microdroplets. The specific process includes the following steps: S51, performing a different component mixing detection on a single layer of micro-droplets containing different components, the specific process includes the following steps: S511, calculating the distance from the center position of any contour area in the segmented area where the component is located to the center position of the single layer micro droplet , and the radius of the monolayer droplets is ; S512: Determine whether there is one and only one single layer of micro-droplets satisfy ; If not, it is considered that the mixing of different components inside the monolayer microdroplets is complete; If yes, proceed to step S513; S513: Determine the area of ​​the contour area Is it less than a minimum threshold? ; like , it is considered that the mixing of different components inside the monolayer microdroplets is complete; like , it is considered that the different components inside the monolayer microdroplets are not mixed completely; S52, identifying the merger between different types of monolayer micro-droplets that have merged, the specific process includes the following steps: S521. Calculate the center position of any microdroplet in different types of monolayer microdroplets The center position of any microdroplet in another type of single-component microdroplet Distance ; S522, according to the distance Determine whether two micro-droplets have completed merging; If satisfied , then the two microdroplets complete merging, is a minimum distance threshold; S53, detecting the encapsulation and encapsulation states of the inner layer micro-droplets and the outer layer micro-droplets in the double-layer complex micro-droplets.

2. The micro-droplet image recognition method according to claim 1, characterized in that: The microdroplets include single-layer microdroplets and double-layer complex microdroplets, wherein the single-layer microdroplets include conventional single-layer microdroplets, different types of single-layer microdroplets, single-layer microdroplets containing a mixture of different components, and different types of single-layer microdroplets that are merged, and the double-layer complex microdroplets include inner-layer microdroplets in the double-layer complex microdroplets and outer-layer microdroplets in the double-layer complex microdroplets.

3. The micro-droplet image recognition method according to claim 1, characterized in that: In step S1, the specific process includes the following steps: S11, using the water phase as the inner phase of the microdroplets and the oil phase as the outer phase of the microdroplets, dyeing the inner phase fluid with a water-soluble pigment to generate microdroplets with different colors or different components in the microdroplets; S12. Adjust the control platform so that the microchannel in the microfluidic chip is in the best field of view of the color high-speed industrial camera, and adjust the corresponding magnification and light intensity; S13, importing the video image data captured by the color high-speed industrial camera into the working computer and converting it into the HSV color space.

4. The micro-droplet image recognition method according to claim 1, characterized in that: In step S22, the area with high color saturation is the dyed micro-droplet, and the area with low color saturation is the chip background, wherein segmenting the micro-droplet and the chip background includes: Segmentation of conventional single-layer microdroplets, inner-layer microdroplets in double-layer complex microdroplets and chip background areas, as well as segmentation of different types of single-layer microdroplets, single-layer microdroplets containing different components mixed inside, and different types of single-layer microdroplets with merging and chip background areas; If the chip background has no complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from the chip background area with low color saturation according to the segmentation threshold; If the chip background has complex colors, the S and V channels of the HSV color space are used to directly segment the micro-droplets with high color saturation from some chip background areas with low color saturation according to the segmentation threshold. Then, according to the two or more peak areas corresponding to the H channel histogram of the HSV color space, the OTSU maximum class spacing method is used to obtain the optimal segmentation threshold between the two or more peak areas, and the micro-droplet area and the chip background area are segmented with the optimal segmentation threshold.

5. The micro-droplet image recognition method according to claim 1, characterized in that: In step S22, the outer layer of the double-layer complex micro-droplets and the chip background area are segmented, specifically: Firstly, the color image is processed into a grayscale image, and the edge contour of the outer layer of micro-droplets is obtained by using the Sobel or Laplacian edge detection operator; Based on the roundness of the edge contour, it is judged whether there are microdroplets in the grayscale image. If yes, the edge contour of the microdroplet is segmented from the chip background. Otherwise, no segmentation is performed.

6. The micro-droplet image recognition method according to claim 1, characterized in that: In step S53, the specific process includes the following steps: S531, calculate the distance between the center position of any inner layer micro-droplet and the center position of each outer layer micro-droplet ; S532, according to distance and the approximate radius of the outer droplet Determine whether the inner layer micro-droplets are wrapped normally; If all the distances Greater than the approximate radius , then the inner layer of micro-droplets is not wrapped properly; If there is only one distance Smaller than the approximate radius , it means that the inner layer of micro-droplets is wrapped normally; S533, calculate the distance between the center position of any outer layer micro-droplet and the center position of each inner layer micro-droplet ; S534, according to distance and the approximate radius of the outer droplet Determine the number of inner layer micro-droplets enclosed within the outer layer micro-droplets; If there is a distance Smaller than the approximate radius , then the number of inner layer micro-droplets wrapped inside the outer layer micro-droplets increases by one; If the number of inner-layer micro-droplets wrapped inside the outer-layer micro-droplets does not match the setting, the outer-layer micro-droplet wrapping state is abnormal.

7. A micro-droplet observation device applied to the micro-droplet image recognition method according to any one of claims 1 to 6, characterized in that: include: A microfluidic system, which is used to adjust the flow rate of the injection pump and control the microfluidic chip to generate microdroplets. The microfluidic system consists of a injection pump, a control platform and a microfluidic chip; the microfluidic chip is fixed on the control platform; The observation system is used to observe and record the microdroplets generated by the microfluidic chip. The observation system consists of a light source and a color high-speed industrial camera.

Images