Liquid dispensing process monitoring and endpoint identification method and system, storage medium and terminal

Abstract

The present invention provides a method and system for monitoring and identifying an endpoint of a liquid separation process, a storage medium and a terminal, comprising the following steps: obtaining a liquid separation process video based on an RGB color space; performing edge detection on each liquid separation process image of the liquid separation process video; extracting a mask image of a liquid separation container based on the liquid separation process image after edge detection; scaling each liquid separation process image of the liquid separation process video to an RGB image of a preset size; matching the RGB image and the mask image to obtain a region of interest of the RGB image; calculating the image entropy of each region of interest, constructing a time series of the image entropy, and selecting the moment corresponding to the maximum value of the image entropy as the liquid separation endpoint. The method and system for monitoring and identifying an endpoint of a liquid separation process, a storage medium and a terminal of the present invention realize unattended liquid separation operation process monitoring and endpoint identification in a laboratory based on an edge detection algorithm and an image entropy algorithm, can capture most of the details of the phenomenon changes in a chemical reaction process, and have accurate results and efficient calculations.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a liquid dispensing process monitoring and endpoint recognition method and system, a storage medium and a terminal. Background Art

[0002] In the process of new drug synthesis and development, operations such as extraction and separation, precipitation, dissolution and precipitation, and chromatographic analysis (thin layer chromatography, column chromatography) require long-term monitoring by the human eye. Among them, the separation process is not only extremely time-consuming due to different degrees of emulsification, but also highly dependent on subjective experience for accuracy. The successful experience of using visual recognition of heterogeneous liquid interfaces in the fields of beverage filling and reaction engineering shows that machine vision has high application potential in assisting pharmaceutical scientists to achieve unattended separation operations.

[0003] Wu Ze et al. proposed a liquid level state recognition algorithm, which can quickly and accurately identify the changes in the liquid state in the industrial reactor, thereby realizing the separation of the liquid level in the industrial reactor. Among them, by analyzing the liquid level data in the industrial reactor, the liquid level data is divided into five categories. The five classic feature extraction algorithms used are Local Binary Pattern (LBP), Histogram of Oriented Gradient (HOG), Local Phase Quantization (LPQ), Haar and color histogram. The classifier algorithms used are Support Vector Machine (SVM) classification algorithm, Random Forest classification algorithm and K-Nearest Neighbor (KNN) KNN classification algorithm.

[0004] In the study of the principal component analysis (PCA) dimensionality reduction classification model based on the HOG features of liquid surface images, first, when the HOG feature extraction algorithm is used to extract liquid surface features, the dimension of the liquid surface data features extracted by the algorithm is too high, so the PCA method is used to reduce the dimension of the HOG features; then the reduced-dimensional features are compared with the features formed by the LBP, LPQ, and Haar feature extraction algorithms in the above three classifiers, and it is found that the recognition rate of the PCA-reduced HOG features in the KNN classifier is as high as 93.15%. Finally, the influence of PCA contribution rate on the HOG feature extraction method is studied.

[0005] When improving the color histogram feature extraction algorithm based on the adaptive threshold method, the traditional color histogram feature extraction algorithm is first used to obtain 256-dimensional features, but this feature does not perform well in various classifiers; then the 256-dimensional features are set by setting the threshold to obtain 256-dimensional binary features. The recognition rate is significantly improved when the three classifier algorithms are used to classify the binary features, but the setting of the threshold has a great influence on the recognition accuracy; finally, the adaptive threshold method is used to form the binary features. The color histogram feature extraction algorithm improved based on the adaptive threshold method has a high recognition accuracy in both the random forest classification algorithm and the SVM classifier algorithm. At the same time, experimental comparison shows that the improved color histogram feature extraction algorithm not only has a high accuracy for all liquid level state data recognition, but also maintains a high recognition accuracy with less training data.

[0006] Zhu Pengchang et al. conducted a series of studies on the free surface recognition problem in liquid tanks using image segmentation technology. The results of the evaluation index test of the free surface segmentation showed that the watershed algorithm based on Gaussian difference filtering can efficiently and accurately identify flat and smooth free liquid surfaces with an accuracy of more than 99%, but there are certain deviations in the recognition of liquid surfaces with blurred surfaces and local micro-fragmentation. The watershed algorithm based on Sobel operator filtering can recognize liquid surfaces with local micro-fragmentation, blurred surfaces, bubbles, etc. with an accuracy of more than 98%, but it has certain limitations in the recognition of complex and broken liquid surfaces.

[0007] In view of the particularity of broken liquid surface recognition, Zhu Pengchang and others developed a breaking wave recognition model based on the U-net network. This segmentation technology has the characteristics of good stability and high accuracy in broken liquid surface recognition, and the accuracy can reach more than 95%. The intelligent recognition technology based on the U-net network model can not only make up for the shortcomings of traditional algorithms, but also ensure the recognition accuracy of the liquid surface. Therefore, in fluid mechanics experiments, the intelligent recognition technology based on the U-net network model has broad application prospects.

[0008] Zhu Pengchang et al. applied the Hilbert transform method based on empirical mode decomposition (EMD) to analyze the natural frequency and other parameters of the liquid in the cabin. The original signal was decomposed by EMD to remove interference signals such as noise, and the natural frequency and damping ratio were solved by Hilbert transform. Comparative analysis shows that the first-order natural frequency corresponding to the experimental value is smaller than the theoretical value, about 0.8-0.9f1. Due to the noise reduction effect of the EMD method, the first-order natural frequency value corresponding to the experimental value is smaller than the first-order natural frequency value solved by FFT. The damping ratio corresponding to the IMF component increases as the IMF correlation decreases.

[0009] In summary, the liquid dispensing process recognition algorithm based on machine vision technology mainly relies on detecting the liquid surface to judge the liquid dispensing process, which has the following shortcomings:

[0010] (1) Most liquid level detection algorithms require complex filtering and image morphological operations, which makes it impossible for the algorithm itself to detect the liquid level in real time;

[0011] (2) The most obvious change in the liquid separation process occurs at the very complex liquid-liquid interface. In most cases, the algorithm recognition accuracy is not high, and the position of the liquid-liquid interface cannot be accurately identified when the liquid surface changes are not obvious;

[0012] (3) The liquid level detection algorithm focuses more on the position of the liquid level in the glass bottle and is unable to determine the end point of liquid dispensing. Summary of the invention

[0013] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a method and system for monitoring the liquid dispensing process and identifying the endpoint, a storage medium and a terminal, which can realize unattended laboratory liquid dispensing operation process monitoring and endpoint identification based on edge detection algorithm and image entropy algorithm, and can capture most of the phenomenon change details of the chemical reaction process accurately and efficiently.

[0014] To achieve the above-mentioned object and other related objects, the present invention provides a method for monitoring a liquid dispensing process and identifying an endpoint, comprising the following steps: obtaining a liquid dispensing process video based on an RGB color space; performing edge detection on each liquid dispensing process image of the liquid dispensing process video; extracting a mask image of a liquid dispensing container based on the liquid dispensing process image after edge detection; scaling each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size; matching the RGB image and the mask image to obtain a region of interest of the RGB image; calculating the image entropy of the region of interest of each RGB image, constructing a time series of the image entropy, and selecting the moment corresponding to the maximum value of the image entropy as the liquid dispensing endpoint.

[0015] In one embodiment of the present invention, edge detection of each liquid dispensing process image of the liquid dispensing process video includes the following steps:

[0016] De-framing the liquid dispensing process video based on the RGB color space to form an RGB image;

[0017] Performing filtering processing on the RGB image based on Gaussian filtering;

[0018] Extract the image gradient of the filtered RGB image based on the Sobel filter;

[0019] A high threshold and a low threshold are set. When the image gradient is greater than the high threshold, the corresponding pixel point is retained as a strong edge pixel point; when the image gradient is between the high threshold and the low threshold, the corresponding pixel point is retained as a weak edge pixel point; if there is a strong edge pixel point in the neighborhood of a pixel point, the pixel point is retained.

[0020] In one embodiment of the present invention, the Gaussian filter uses a 3×3 Gaussian kernel; the Sobel filter uses a 3×3 Sobel filter.

[0021] In one embodiment of the present invention, extracting a mask image of a liquid dispensing container based on a liquid dispensing process image after edge detection includes the following steps:

[0022] Perform closing operation on the liquid separation process image after edge detection;

[0023] Perform geodesic dilation on the liquid separation process image after the closing operation to obtain a main body binary image;

[0024] Holes are filled in the binary main image to obtain a mask image of the liquid separation container.

[0025] In one embodiment of the present invention, scaling each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size comprises the following steps:

[0026] Scaling the width of the liquid dispensing process image to preset pixels;

[0027] The liquid separation process image after width scaling is length-scaled so that the image ratio of the liquid separation process image after length scaling is consistent with that of the liquid separation process image, so as to obtain the RGB image.

[0028] In one embodiment of the present invention, constructing a time series of image entropy includes constructing a curve graph showing changes in image entropy over time.

[0029] The present invention provides a liquid dispensing process monitoring and endpoint recognition system, comprising a video acquisition module, an edge detection module, a mask extraction module, an image conversion module, a matching module and a monitoring and recognition module;

[0030] The video acquisition module is used to acquire the liquid dispensing process video based on the RGB color space;

[0031] The edge detection module is used to perform edge detection on each liquid dispensing process image of the liquid dispensing process video;

[0032] The mask extraction module is used to extract the mask image of the liquid dispensing container based on the liquid dispensing process image after edge detection;

[0033] The image conversion module is used to scale each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size;

[0034] The matching module is used to match the RGB image and the mask image to obtain the region of interest of the RGB image;

[0035] The monitoring and identification module is used to calculate the image entropy of the region of interest of each RGB image, construct a time series of the image entropy, and select the moment corresponding to the maximum value of the image entropy as the end point of liquid separation.

[0036] In one embodiment of the present invention, the edge detection module includes a frame removal module, a filtering module, a gradient extraction module and a dual threshold screening module;

[0037] The de-framing module is used to de-frame the liquid dispensing process video based on the RGB color space to form an RGB image;

[0038] The filtering module is used to perform filtering processing on the RGB image based on Gaussian filtering;

[0039] The gradient extraction module is used to extract the image gradient of the filtered RGB image based on the Sobel filter;

[0040] The dual threshold screening module is used to set a high threshold and a low threshold. When the image gradient is greater than the high threshold, the corresponding pixel point is retained as a strong edge pixel point; when the image gradient is between the high threshold and the low threshold, the corresponding pixel point is retained as a weak edge pixel point; if there is a strong edge pixel point in the neighborhood of a pixel point, the pixel point is retained.

[0041] In one embodiment of the present invention, the Gaussian filter uses a 3×3 Gaussian kernel; the Sobel filter uses a 3×3 Sobel filter.

[0042] In one embodiment of the present invention, the mask extraction module includes a closing operation module, a geodesic dilation module and a hole filling module;

[0043] The closing operation module is used to perform a closing operation on the liquid separation process image after edge detection;

[0044] The geodesic expansion module is used to perform geodesic expansion on the liquid separation process image after the closing operation to obtain a main body binary image;

[0045] The hole filling module is used to fill the holes of the main binary image to obtain the mask image of the liquid separation container.

[0046] In one embodiment of the present invention, the image conversion module includes a width scaling module and a length scaling module;

[0047] The width scaling module is used to scale the width of the liquid dispensing process image to preset pixels;

[0048] The length scaling module is used to length scale the liquid separation process image after width scaling, so that the image ratio of the liquid separation process image after length scaling is consistent with that of the liquid separation process image, so as to obtain the RGB image.

[0049] In one embodiment of the present invention, when the monitoring and identification module constructs the time series of the image entropy, it constructs a curve graph showing the change of the image entropy over time.

[0050] The present invention provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned liquid separation process monitoring and endpoint identification method is implemented.

[0051] The present invention provides a liquid dispensing process monitoring and endpoint identification terminal, comprising: a processor and a memory;

[0052] The memory is used to store computer programs;

[0053] The processor is used to execute the computer program stored in the memory, so that the liquid dispensing process monitoring and endpoint identification terminal executes the above-mentioned liquid dispensing process monitoring and endpoint identification method.

[0054] The present invention provides a liquid dispensing process monitoring and endpoint identification system, comprising a video acquisition device and the above-mentioned liquid dispensing process monitoring and endpoint identification terminal;

[0055] The video acquisition terminal is used to acquire the liquid dispensing process video based on the RGB color space and send it to the liquid dispensing process monitoring and endpoint identification terminal.

[0056] As described above, the method and system for monitoring the liquid dispensing process and identifying the endpoint, the storage medium and the terminal of the present invention have the following beneficial effects:

[0057] (1) Based on edge detection algorithm and image entropy algorithm, unattended laboratory dispensing operation process monitoring can be realized, which can capture most of the phenomenon change details of the chemical reaction process;

[0058] (2) The computational complexity is low and meets the real-time requirements;

[0059] (3) High accuracy, capable of achieving temporal matching between the end point of liquid separation and the maximum value of image entropy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Shown is a flow chart of a method for monitoring the liquid dispensing process and identifying the endpoint in one embodiment of the present invention;

[0061] Figure 2It is a schematic diagram showing a mask extraction of the present invention in one embodiment;

[0062] Figure 3 Shown is a schematic diagram of phase interface changes during a liquid separation process in one embodiment of the liquid separation process monitoring and endpoint identification method of the present invention;

[0063] Figure 4 Display as Figure 3 RGB color space color histogram and grayscale image color histogram during the liquid separation process;

[0064] Figure 5 Display as Figure 3 Schematic diagram of the curve change of image entropy during the liquid separation process;

[0065] Figure 6 Shown is a schematic structural diagram of a liquid dispensing process monitoring and endpoint identification system in one embodiment of the present invention;

[0066] Figure 7 Shown is a schematic structural diagram of a liquid dispensing process monitoring and endpoint identification terminal in one embodiment of the present invention;

[0067] Figure 8 It is a schematic structural diagram of another embodiment of the liquid dispensing process monitoring and endpoint identification system of the present invention.

[0068] Component number description

[0069] 61 Video acquisition module

[0070] 62 Edge Detection Module

[0071] 63 Mask Extraction Module

[0072] 64 Image conversion module

[0073] 65 Matching module

[0074] 66 Monitoring and identification module

[0075] 71 Processor

[0076] 72 Memory

[0077] 81 Video acquisition device

[0078] 82 Liquid dispensing process monitoring and endpoint identification terminal DETAILED DESCRIPTION

[0079] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0080] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0081] The liquid dispensing process monitoring and endpoint identification method and system, storage medium and terminal of the present invention combine edge detection algorithm and image entropy algorithm to realize unattended liquid dispensing operation process monitoring in the laboratory, with low computational complexity and good real-time performance, meeting the needs of actual application scenarios and being extremely practical.

[0082] like Figure 1 As shown, in one embodiment, the method for monitoring the liquid separation process and identifying the endpoint of the present invention comprises the following steps:

[0083] Step S1, obtaining a liquid dispensing process video based on the RGB color space.

[0084] Specifically, the video of the liquid separation process is collected based on the video collection device, and is sent to the liquid separation process monitoring and endpoint identification terminal of the present invention in a wired or wireless manner.

[0085] In one embodiment of the present invention, the video acquisition device uses a camera to directly shoot the video of the liquid separation process, or uses an intelligent terminal with network communication function to indirectly provide the video of the liquid separation process transmitted from the network. Wherein, the intelligent terminal refers to a smart phone, a tablet computer, a PDA, a PC, and other terminal devices with data processing functions. Generally, an intelligent terminal refers to a terminal device with an independent operating system, on which users can install software, games and other programs provided by third-party service providers, and through such programs to continuously expand the functions of handheld devices, and can achieve wireless network access through mobile communication networks.

[0086] Step S2: performing edge detection on each liquid dispensing process image of the liquid dispensing process video.

[0087] Specifically, edge detection is performed on each frame of the liquid dispensing process image in the liquid dispensing process video, so as to identify a portion of the image where a large change occurs in the brightness region, which is composed of pixel points with extreme gradients.

[0088] In one embodiment of the present invention, edge detection of each liquid dispensing process image of the liquid dispensing process video includes the following steps:

[0089] 21) De-framing the liquid separation process video based on the RGB color space to form an RGB image.

[0090] 22) Perform filtering processing on the RGB image based on Gaussian filtering.

[0091] Specifically, the RGB image is blurred by using Gaussian filtering to suppress image noise. The larger the Gaussian kernel, the blurrier the output image. Preferably, the present invention uses a Gaussian kernel of size 3×3, so that image noise can be eliminated while retaining image details.

[0092] 23) Extract the image gradient of the filtered RGB image based on the Sobel filter.

[0093] Specifically, the image gradient refers to the rate of change of a certain pixel in the image in the x and y directions (compared with adjacent pixels), which is a two-dimensional vector consisting of two components, the change in the x-axis and the change in the y-axis. Therefore, the image edge can be obtained through the image gradient. Preferably, the Sobel filter uses a 3×3 Sobel filter.

[0094] 24) Setting a high threshold and a low threshold, when the image gradient is greater than the high threshold, the corresponding pixel point is retained as a strong edge pixel point; when the image gradient is between the high threshold and the low threshold, the corresponding pixel point is retained as a weak edge pixel point; if there is a strong edge pixel point in the neighborhood of a pixel point, the pixel point is retained.

[0095] Specifically, in order to further reduce noise points and connect edge points, the present invention uses a double threshold (high threshold and low threshold) screening method to screen the image gradient. The high threshold is greater than the low threshold, and only strong edge pixels, weak edge pixels, and pixels with strong edge pixels in the neighborhood are retained.

[0096] Preferably, the neighborhood adopts any one of the following: 4-neighborhood, diagonal neighborhood and 8-neighborhood. Specifically, for the nine-square grid centered on pixel P, the four pixels covered by a "plus sign" are called the 4-neighborhood of the central pixel, recorded as N4(P); the four pixels in the corners are the diagonal neighborhood, recorded as ND(P); all the surrounding 8 pixels are called the 8-neighborhood of the central pixel, recorded as N8(P).

[0097] Step S3: extracting a mask image of the liquid dispensing container based on the liquid dispensing process image after edge detection.

[0098] Specifically, the image mask is to use a selected image, graphic or object to block the processed image (all or part) to control the image processing area or processing process. Figure 2 As shown, extracting the mask image of the liquid dispensing container based on the liquid dispensing process image after edge detection includes the following steps:

[0099] 31) Perform a closing operation on the liquid separation process image after edge detection.

[0100] Specifically, the closing operation can make the contour smoother, eliminate narrow discontinuities and long thin gaps, eliminate small holes, and fill breaks in the contour.

[0101] 32) Perform geodesic dilation on the liquid separation process image after the closing operation to eliminate the edge noise part and obtain the main body binary image.

[0102] 33) Filling holes in the binary main image to obtain a mask image of the liquid separation container.

[0103] Step S4: scaling each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size.

[0104] Specifically, in order to reduce the computational complexity, each liquid dispensing process image of the liquid dispensing process video needs to be scaled to an RGB image of a preset size.

[0105] In one embodiment of the present invention, scaling each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size comprises the following steps:

[0106] 41) Scaling the width of the liquid dispensing process image to preset pixels.

[0107] Specifically, the width of the liquid separation process image is first scaled to a preset pixel. The preset pixel should be able to take into account both complete image information and low computational complexity. Preferably, the preset pixel is 200 pixels.

[0108] 42) Length-scaling the width-scaled liquid separation process image so that the length-scaled liquid separation process image has the same image ratio as the liquid separation process image to obtain the RGB image.

[0109] Specifically, after the width is scaled, the length is scaled again, so that the image ratio of the finally scaled liquid separation process image is consistent with that of the initial liquid separation process image, that is, the length and width are scaled in equal proportion.

[0110] Step S5: Match the RGB image and the mask image to obtain a region of interest (ROI) of the RGB image.

[0111] Specifically, for each RGB image, its corresponding mask image is obtained, and the RGB image and the mask image are matched, and the matched area is the region of interest of the RGB image, thereby eliminating the influence of lighting conditions and human operations in the picture. When performing the matching operation, the mask image is directly overlaid on the RGB image, and the overlapped part is the region of interest of the RGB image, and the calculation process is equivalent to the dot product of two matrices.

[0112] Step S6: Calculate the image entropy of the region of interest of each RGB image, construct a time series of the image entropy, and select the moment corresponding to the maximum value of the image entropy as the end point of liquid separation.

[0113] Specifically, for each region of interest in the RGB image, the image entropy is calculated. Entropy is a measure of the degree of disorder of a system. It was first applied to thermodynamics and later introduced into information theory by Shannon. Information entropy is used to describe the uncertainty of the information source. The greater the uncertainty of the variable, the greater the entropy value. Image entropy is a characteristic statistical form that reflects the average amount of information in the image.

[0114] After the global information entropy of each frame of the liquid dispensing process image in the liquid dispensing process video is calculated, a time series of image entropy is constructed based on the time sequence. Preferably, a curve graph showing the change of image entropy over time can be used. By analyzing the time series of the image entropy, the moment corresponding to the maximum value of the image entropy is obtained, and the moment is determined as the end point of liquid dispensing.

[0115] The following is a further description of the method for monitoring the liquid separation process and identifying the endpoint of the present invention through specific embodiments.

[0116] like Figure 3 As shown, in this embodiment, the four liquid dispensing process video data sets used are derived from local laboratory recording and network transmission. The backgrounds, lighting conditions, and glassware shapes of the four liquid dispensing process videos are different, which is convenient for verifying the robustness of the liquid dispensing process monitoring and endpoint recognition method of the present invention.

[0117] Figure 4The RGB color space color histogram and grayscale image color histogram of the four stages of the liquid separation process are shown. Among them, the horizontal axis represents the pixel value 0-255 normalized to the value between 0-1, and the vertical axis represents the image entropy. It can be seen from the figure that when the solution is highly mixed at the beginning of the liquid separation operation, the pixel values ​​of each point in the image domain are concentrated, the image details are less, and the image entropy value is low. As the liquid separation operation proceeds, the heterogeneous liquid in the liquid separation instrument gradually stratifies due to density differences, the image pixel values ​​show a multi-peak distribution, the image details are rich, and the image entropy value increases. Therefore, after the heterogeneous liquid is stratified and reaches the end point of liquid separation, the image entropy reaches an extreme value when the solution no longer changes.

[0118] In order to improve the calculation rate and achieve the purpose of real-time detection, after scaling each dispensing process image, the calculation is performed frame by frame according to the time series to obtain the following: Figure 5 The image entropy change curves of the above four videos are shown. As can be seen from the figure, the image entropy of the four videos shows an upward trend as the separation process proceeds. When the separation process approaches the end point, the liquid phase interface changes slowly. Affected by changes in lighting conditions, the image entropy value will fluctuate in a small range, resulting in a certain degree of lag between the actual separation end point and the maximum image entropy value in the first three videos with longer separation processes. Since the fourth video has a shorter overall time and the video frame is accelerated, the actual separation end point and the image entropy maximum point are relatively consistent in time, but there are still small fluctuations near the image entropy maximum value. According to analysis, the mismatch between the image entropy maximum value of the color RGB image and the actual separation end point is mainly affected by the mixing of multi-channel features in the RGB color space during the separation process.

[0119] like Figure 6 As shown, in one embodiment, the liquid dispensing process monitoring and endpoint identification system of the present invention includes a video acquisition module 61, an edge detection module 62, a mask extraction module 63, an image conversion module 64, a matching module 65 and a monitoring and identification module 66.

[0120] The video acquisition module 61 is used to acquire a video of the liquid dispensing process based on the RGB color space.

[0121] Specifically, the video of the liquid separation process is collected based on the video collection device, and is sent to the liquid separation process monitoring and endpoint identification terminal of the present invention in a wired or wireless manner.

[0122] In one embodiment of the present invention, the video acquisition device uses a camera to directly shoot the video of the liquid separation process, or uses an intelligent terminal with network communication function to indirectly provide the video of the liquid separation process transmitted from the network. Wherein, the intelligent terminal refers to a smart phone, a tablet computer, a PDA, a PC, and other terminal devices with data processing functions. Generally, an intelligent terminal refers to a terminal device with an independent operating system, on which users can install software, games and other programs provided by third-party service providers, and through such programs to continuously expand the functions of handheld devices, and can achieve wireless network access through mobile communication networks.

[0123] The edge detection module 62 is connected to the video acquisition module 61 and is used to perform edge detection on each liquid dispensing process image of the liquid dispensing process video.

[0124] Specifically, edge detection is performed on each frame of the liquid dispensing process image in the liquid dispensing process video, so as to identify a portion of the image where a large change occurs in the brightness region, which is composed of pixel points with extreme gradients.

[0125] In one embodiment of the present invention, the edge detection module includes a frame removal module, a filtering module, a gradient extraction module and a dual threshold screening module.

[0126] The de-framing module is used to de-frame the liquid dispensing process video based on the RGB color space to form an RGB image.

[0127] The filtering module is connected to the frame removal module and is used to perform filtering processing on the RGB image based on Gaussian filtering.

[0128] Specifically, the RGB image is blurred by using Gaussian filtering to suppress image noise. The larger the Gaussian kernel, the blurrier the output image. Preferably, the present invention uses a Gaussian kernel of size 3×3, so that image noise can be eliminated while retaining image details.

[0129] The gradient extraction module is connected to the filtering module and is used to extract the image gradient of the filtered RGB image based on a Sobel filter.

[0130] Specifically, the image gradient refers to the rate of change of a certain pixel in the image in the x and y directions (compared with adjacent pixels), which is a two-dimensional vector consisting of two components, the change in the x-axis and the change in the y-axis. Therefore, the image edge can be obtained through the image gradient. Preferably, the Sobel filter uses a 3×3 Sobel filter.

[0131] The dual threshold screening module is connected to the gradient extraction module for setting a high threshold and a low threshold. When the image gradient is greater than the high threshold, the corresponding pixel point is retained as a strong edge pixel point; when the image gradient is between the high threshold and the low threshold, the corresponding pixel point is retained as a weak edge pixel point; if there is a strong edge pixel point in the neighborhood of a pixel point, the pixel point is retained.

[0132] Specifically, in order to further reduce noise points and connect edge points, the present invention uses a double threshold (high threshold and low threshold) screening method to screen the image gradient. The high threshold is greater than the low threshold, and only strong edge pixels, weak edge pixels, and pixels with strong edge pixels in the neighborhood are retained.

[0133] Preferably, the neighborhood adopts any one of the following: 4-neighborhood, diagonal neighborhood and 8-neighborhood. Specifically, for the nine-square grid centered on pixel P, the four pixels covered by a "plus sign" are called the 4-neighborhood of the central pixel, recorded as N4(P); the four pixels in the corners are the diagonal neighborhood, recorded as ND(P); all the surrounding 8 pixels are called the 8-neighborhood of the central pixel, recorded as N8(P).

[0134] The mask extraction module 63 is connected to the edge detection module 62 and is used to extract the mask image of the liquid dispensing container based on the liquid dispensing process image after edge detection.

[0135] Specifically, the image mask is to use a selected image, graphic or object to block the processed image (all or part) to control the image processing area or processing process. In one embodiment of the present invention, the mask extraction module includes a closing operation module, a geodesic dilation module and a hole filling module.

[0136] The closing operation module is used to perform a closing operation on the liquid separation process image after edge detection.

[0137] Specifically, the closing operation can make the contour smoother, eliminate narrow discontinuities and long thin gaps, eliminate small holes, and fill breaks in the contour.

[0138] The geodesic expansion module is connected to the closing operation module and is used to perform geodesic expansion on the liquid separation process image after the closing operation to obtain a main body binary image.

[0139] The hole filling module is connected to the geodesic expansion module and is used to fill the holes of the main binary image to obtain the mask image of the liquid separation container.

[0140] The image conversion module 64 is connected to the video acquisition module 61 and is used to scale each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size.

[0141] Specifically, in order to reduce the computational complexity, each liquid dispensing process image of the liquid dispensing process video needs to be scaled to an RGB image of a preset size.

[0142] In one embodiment of the present invention, the image conversion module includes a width scaling module and a length scaling module;

[0143] The width scaling module is used to scale the width of the liquid dispensing process image to preset pixels.

[0144] Specifically, the width of the liquid separation process image is first scaled to a preset pixel. The preset pixel should be able to take into account both complete image information and low computational complexity. Preferably, the preset pixel is 200 pixels.

[0145] The length scaling module is connected to the width scaling module and is used to length scale the liquid separation process image after width scaling so that the image ratio of the liquid separation process image after length scaling is consistent with that of the liquid separation process image, so as to obtain the RGB image.

[0146] Specifically, after the width is scaled, the length is scaled again, so that the image ratio of the finally scaled liquid separation process image is consistent with that of the initial liquid separation process image, that is, the length and width are scaled in equal proportion.

[0147] The matching module 65 is connected to the mask extraction module 63 and the image conversion module 64, and is used to match the RGB image with the mask image to obtain the region of interest of the RGB image.

[0148] Specifically, for each RGB image, its corresponding mask image is obtained, and the RGB image and the mask image are matched, and the matched area is the region of interest of the RGB image, thereby eliminating the influence of lighting conditions and human operations in the picture. When performing the matching operation, the mask image is directly overlaid on the RGB image, and the overlapped part is the region of interest of the RGB image, and the calculation process is equivalent to the dot product of two matrices.

[0149] The monitoring and identification module 66 is connected to the matching module 65 and is used to calculate the image entropy of the region of interest of each RGB image, construct a time series of the image entropy, and select the moment corresponding to the maximum value of the image entropy as the end point of liquid separation.

[0150] Specifically, for each region of interest in the RGB image, the image entropy is calculated. Entropy is a measure of the degree of disorder of a system. It was first applied to thermodynamics and later introduced into information theory by Shannon. Information entropy is used to describe the uncertainty of the information source. The greater the uncertainty of the variable, the greater the entropy value. Image entropy is a characteristic statistical form that reflects the average amount of information in the image.

[0151] After the global information entropy of each frame of the liquid dispensing process image in the liquid dispensing process video is calculated, a time series of image entropy is constructed based on the time sequence. Preferably, a curve graph showing the change of image entropy over time can be used. By analyzing the time series of the image entropy, the moment corresponding to the maximum value of the image entropy is obtained, and the moment is determined as the end point of liquid dispensing.

[0152] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the x module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0153] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processors that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0154] The storage medium of the present invention stores a computer program, which implements the above-mentioned liquid separation process monitoring and endpoint identification method when executed by the processor. The storage medium includes: ROM, RAM, disk, U disk, memory card or CD and other media that can store program codes.

[0155] like Figure 7 As shown, in one embodiment, the terminal for monitoring the liquid dispensing process and identifying the endpoint of the present invention includes: a processor 71 and a memory 72 .

[0156] The memory 72 is used to store computer programs.

[0157] The memory 72 includes: ROM, RAM, disk, USB flash drive, memory card or optical disk and other media that can store program codes.

[0158] The processor 71 is connected to the memory 72 and is used to execute the computer program stored in the memory 72 so that the liquid dispensing process monitoring and endpoint identification terminal executes the above-mentioned liquid dispensing process monitoring and endpoint identification method.

[0159] Preferably, the processor 71 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0160] like Figure 8 As shown, in one embodiment, the liquid dispensing process monitoring and endpoint identification system of the present invention includes a video acquisition device 81 and the above-mentioned liquid dispensing process monitoring and endpoint identification terminal 82.

[0161] The video acquisition terminal 81 is connected to the liquid dispensing process monitoring and endpoint identification terminal 82 , and is used to acquire the liquid dispensing process video based on the RGB color space and send it to the liquid dispensing process monitoring and endpoint identification terminal 82 .

[0162] Preferably, the video acquisition device 81 can be a camera, an intelligent terminal with a network communication function, etc., so as to directly or indirectly obtain the video of the liquid separation process.

[0163] In summary, the method and system for monitoring the liquid dispensing process and identifying the endpoint, the storage medium and the terminal of the present invention realize the monitoring of the liquid dispensing operation process in the laboratory unattended based on the edge detection algorithm and the image entropy algorithm, and can capture most of the details of the phenomenon changes in the chemical reaction process; the computational complexity is low, meeting the real-time requirements; the accuracy is high, and the time matching of the liquid dispensing endpoint and the maximum value of the image entropy can be achieved. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0164] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for monitoring a liquid dispensing process and identifying an endpoint, characterized in that: The following steps are involved: Get the dispensing process video based on RGB color space; Performing edge detection on each liquid dispensing process image of the liquid dispensing process video; Extracting a mask image of the liquid dispensing container based on the liquid dispensing process image after edge detection; Scaling each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size; Matching the RGB image and the mask image to obtain a region of interest of the RGB image; Calculate the image entropy of the region of interest of each RGB image, construct the time series of image entropy, and select the moment corresponding to the maximum value of image entropy as the end point of liquid separation; Performing edge detection on each liquid dispensing process image of the liquid dispensing process video comprises the following steps: De-framing the liquid dispensing process video based on the RGB color space to form an RGB image; Performing filtering processing on the RGB image based on Gaussian filtering; Extract the image gradient of the filtered RGB image based on the Sobel filter; A high threshold and a low threshold are set. When the image gradient is greater than the high threshold, the corresponding pixel is retained as a strong edge pixel; when the image gradient is between the high threshold and the low threshold, the corresponding pixel is retained as a weak edge pixel; if a strong edge pixel exists in the neighborhood of a pixel, the pixel is retained; the neighborhood adopts a 4-neighborhood, a diagonal neighborhood or an 8-neighborhood; Extracting the mask image of the dispensing container based on the dispensing process image after edge detection includes the following steps: Perform closing operation on the liquid separation process image after edge detection; Perform geodesic dilation on the liquid separation process image after the closing operation to obtain a main body binary image; Filling holes in the binary main image to obtain a mask image of the liquid dispensing container; Scaling each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size comprises the following steps: Scaling the width of the liquid dispensing process image to preset pixels; Scaling the liquid separation process image after the width scaling so that the image ratio of the liquid separation process image after the length scaling is consistent with that of the liquid separation process image, so as to obtain the RGB image; Also included is constructing a time series of image entropy, including constructing a graph of image entropy changing over time.

2. The method for monitoring the liquid separation process and identifying the endpoint according to claim 1, characterized in that: The Gaussian filter uses a 3×3 Gaussian kernel; the Sobel filter uses a 3×3 Sobel filter.

3. A liquid dispensing process monitoring and endpoint identification system, characterized in that: It includes a video acquisition module, an edge detection module, a mask extraction module, an image conversion module, a matching module and a monitoring recognition module; The video acquisition module is used to acquire the liquid dispensing process video based on the RGB color space; The edge detection module is used to perform edge detection on each liquid dispensing process image of the liquid dispensing process video; The mask extraction module is used to extract the mask image of the liquid dispensing container based on the liquid dispensing process image after edge detection; The image conversion module is used to scale each liquid dispensing process image of the liquid dispensing process video into an RGB image of a preset size; The matching module is used to match the RGB image and the mask image to obtain the region of interest of the RGB image; The monitoring and identification module is used to calculate the image entropy of the region of interest of each RGB image, construct a time series of the image entropy, and select the moment corresponding to the maximum value of the image entropy as the end point of liquid separation; The edge detection module includes a frame removal module, a filtering module, a gradient extraction module and a dual threshold screening module; The de-framing module is used to de-frame the liquid dispensing process video based on the RGB color space to form an RGB image; The filtering module is used to perform filtering processing on the RGB image based on Gaussian filtering; The gradient extraction module is used to extract the image gradient of the filtered RGB image based on the Sobel filter; The dual threshold screening module is used to set a high threshold and a low threshold. When the image gradient is greater than the high threshold, the corresponding pixel point is retained as a strong edge pixel point; when the image gradient is between the high threshold and the low threshold, the corresponding pixel point is retained as a weak edge pixel point; if a strong edge pixel point exists in the neighborhood of a pixel point, the pixel point is retained; the neighborhood adopts a 4-neighborhood, a diagonal neighborhood or an 8-neighborhood; The mask extraction module includes a closing operation module, a geodesic dilation module and a hole filling module; The closing operation module is used to perform a closing operation on the liquid separation process image after edge detection; The geodesic expansion module is used to perform geodesic expansion on the liquid separation process image after the closing operation to obtain a main body binary image; The hole filling module is used to fill holes in the binary image of the main body to obtain a mask image of the liquid dispensing container; The image conversion module includes a wide scaling module and a long scaling module; The width scaling module is used to scale the width of the liquid dispensing process image to preset pixels; The length scaling module is used to length scale the liquid dispensing process image after width scaling so that the image ratio of the liquid dispensing process image after length scaling is consistent with that of the liquid dispensing process image, so as to obtain the RGB image; When the monitoring and identification module constructs the time series of image entropy, it constructs a curve graph showing the change of image entropy over time.

4. The liquid dispensing process monitoring and endpoint identification system according to claim 3, characterized in that: The Gaussian filter uses a 3×3 Gaussian kernel; the Sobel filter uses a 3×3 Sobel filter.

5. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for monitoring the liquid dispensing process and identifying the endpoint described in any one of claims 1 to 2 is implemented.

6. A liquid separation process monitoring and endpoint identification terminal, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the liquid dispensing process monitoring and endpoint identification terminal executes the liquid dispensing process monitoring and endpoint identification method according to any one of claims 1 to 2.

7. A liquid dispensing process monitoring and endpoint identification system, characterized in that: It includes a video acquisition device and the liquid separation process monitoring and endpoint identification terminal as described in claim 6; The video acquisition terminal is used to acquire the liquid dispensing process video based on the RGB color space and send it to the liquid dispensing process monitoring and endpoint identification terminal.

Images