Liquid drop volume image calibration method and system based on machine learning

Abstract

The invention provides a liquid drop volume image calibration method and system based on machine learning. The method comprises the following steps: obtaining known data of a plurality of training objects; processing known data of a plurality of training objects, and training through a machine learning method to obtain an optimized droplet volume prediction model; and inputting the known data of the to-be-measured liquid drop into the tuning liquid drop volume prediction model for prediction, so as to obtain the predicted volume value of the to-be-measured liquid drop through a machine learning method, and the method has higher precision and stability compared with traditional image calibration.

Description

Technical Field

[0001] This application belongs to the field of pipetting technology, and specifically relates to a method and system for calibrating droplet volume images based on machine learning. Background Art

[0002] In many fields such as scientific experiments, biotechnology, clinical diagnosis, and drug research and development, the accurate pipetting and dispensing of trace liquids are key steps in the experimental process. Traditional contact pipetting techniques, such as using pipettes or capillaries, have risks such as sample residue, cross-contamination, and pipette tip blockage, which may all have a negative impact on experimental results. Therefore, with the progress of technology and the improvement of experimental requirements, non-contact pipetting techniques have emerged. For example, ultrasonic pipetting technology ejects droplets from the liquid surface through acoustic manipulation technology to achieve precise liquid transfer; or pneumatic micro-droplet generation technology discharges liquid from small holes through air pressure to form micro-droplets, etc.

[0003] With the continuous development of non-contact pipetting technology, the pipetting accuracy has also been continuously improved. However, in the prior art, there are still certain problems with the method for calibrating the volume of pipetted droplets. Specifically: The pipetting droplet calibration methods include image calibration method and fluorescence calibration method. The traditional image calibration method assumes that the droplet is an ideal sphere, but the actual droplet is not a sphere in the strict sense, which will make the traditional image calibration method lack accuracy. Moreover, affected by the shooting angle and lens distortion, the error will be further amplified.

[0004] Although the fluorescence calibration method can convert the tiny change in liquid volume into a huge change in fluorescence intensity to calibrate the acoustic liquid volume, it will have higher accuracy than the traditional image calibration method. However, fluorescence quenching during the calibration process of the fluorescence calibration method will make the calibration result inaccurate, and this method has complex and cumbersome steps, takes a long time, and requires a large amount of manpower and material resources, which limits its practical application to a certain extent.

[0005] Content of the Application

[0006] In order to overcome the above-mentioned defects of the prior art, this application proposes a method and system for calibrating droplet volume images based on machine learning, which has higher accuracy and stability compared with traditional image calibration.

[0007] Specifically, it is realized through the following technical solutions:

[0008] A method for calibrating droplet volume images based on machine learning, comprising:

[0009] Obtaining known data of multiple training objects;

[0010] Processing the known data of multiple said training objects and training through machine learning methods to obtain an optimized droplet volume prediction model;

[0011] Input the known data of the liquid droplets to be measured into the optimized liquid droplet volume prediction model for prediction, so as to obtain the predicted volume value of the liquid droplets to be measured through machine learning methods.

[0012] In a specific embodiment, the method for calibrating the liquid droplet volume based on machine learning is applied to a pipetting device.

[0013] In a specific embodiment, the known data of multiple training objects includes video data of multiple spheres and video data of multiple liquid droplets;

[0014] Processing the known data of multiple training objects and training through machine learning methods to obtain an optimized liquid droplet volume prediction model includes:

[0015] Extract images from the video data of multiple spheres and the video data of multiple liquid droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple liquid droplets;

[0016] Use multiple first image sequences as first sample data and perform a first training on the first sample data through machine learning methods to obtain a liquid droplet volume prediction model; use multiple second image sequences as second sample data and perform a second training on the second sample data through machine learning methods to optimize the liquid droplet volume prediction model and obtain an optimized liquid droplet volume prediction model.

[0017] In a specific embodiment, the machine learning method used for training includes the combination of a three-dimensional residual neural network and a long short-term memory network, or the machine learning method used for training includes the combination of a convolutional neural network and a fully connected network.

[0018] In a specific embodiment, the parameter values of the spheres include the diameter of the spheres or the area values of the spheres; the parameter values of the liquid droplets include the diameter of the liquid droplets or the area values of the liquid droplets;

[0019] And / or, the data volume of the first sample data is greater than the data volume of the second sample data.

[0020] In a specific embodiment, the extracting images from the video data of multiple spheres and the video data of multiple liquid droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple liquid droplets includes:

[0021] Extract images from the video data of multiple spheres and the video data of multiple liquid droplets to obtain the images to be calibrated in the video data of each sphere and the images to be calibrated in the video data of each liquid droplet;

[0022] Normalize the calibration images of all spheres and the calibration images of all droplets, and set labels for the respective calibration images according to the parameter values of the spheres and the parameter values of the droplets, so as to obtain a first image sequence with the parameter values of the spheres calibrated and a second image sequence with the parameter values of the droplets calibrated.

[0023] In a specific embodiment, the extracting images from the video data of multiple spheres and the video data of multiple droplets to obtain the calibration images in the video data of each sphere and the calibration images in the video data of each droplet includes:

[0024] Extract images from the video data of multiple spheres and the video data of multiple droplets to obtain multiple consecutive frames of images in the video data of each sphere and multiple consecutive frames of images in the video data of each droplet;

[0025] Based on the multiple consecutive frames of images of all spheres and the multiple consecutive frames of images of all droplets, select the regions of interest of all spheres and the regions of interest of all droplets;

[0026] Perform binarization processing on the multiple consecutive frames of images of all spheres and the multiple consecutive frames of images of all droplets, and generate the calibration images of the spheres and the calibration images of the droplets according to the selected regions of interest of the spheres and the regions of interest of the droplets.

[0027] In a specific embodiment, the obtaining the known data of multiple training objects includes obtaining the video data of multiple spheres and obtaining the video data of multiple droplets;

[0028] The obtaining the video data of multiple spheres includes:

[0029] Build a first platform with a pipeline, a photographing device, a light source, and a measuring device, push multiple spheres into the pipeline, and the pipeline is configured to allow the spheres to freely fall when opened;

[0030] Adjust the positions of the photographing device, the light source, and the measuring device so that the photographing device can photograph the falling trajectory of the spheres and the measuring device;

[0031] Open the pipeline to allow multiple spheres to freely fall to obtain the video data of multiple spheres;

[0032] And / or, the obtaining the video data of multiple droplets includes:

[0033] Build a second platform with an ultrasonic transducer, a liquid reservoir, a target plate, a light source, a photographing device, and a measuring device, and the ultrasonic transducer is configured to move the droplets in the liquid reservoir to the target plate when working;

[0034] Adjust the positions of the photographing device, the light source, and the measuring device so that the photographing device can photograph the movement trajectory of the droplet and the measuring device;

[0035] Adjust the excitation parameters of the ultrasonic transducer to eject multiple droplets to obtain video data of multiple droplets.

[0036] In a specific embodiment, the step of taking multiple first image sequences as first sample data and performing a first training on the first sample data by means of machine learning to obtain a droplet volume prediction model includes:

[0037] Take multiple first image sequences as first sample data and divide them into a first training set, a first validation set, and a first test set according to a first preset ratio;

[0038] Train the first training set by means of machine learning and verify the training process based on the first validation set to obtain a first model to be tested; input the first test set into the first model to be tested for testing, and adjust the first model to be tested based on the test results to obtain the droplet volume prediction model;

[0039] The step of taking multiple second image sequences as second sample data and performing a second training on the second sample data by means of machine learning to optimize the droplet volume prediction model to obtain an optimized droplet volume prediction model includes:

[0040] Take multiple second image sequences as second sample data and divide them into a second training set, a second validation set, and a second test set according to a second preset ratio;

[0041] Train the second training set by means of machine learning and verify the training process based on the second validation set to optimize the droplet volume prediction model to obtain a second model to be tested; input the second test set into the second model to be tested for testing, and adjust the second model to be tested based on the test results to obtain the optimized droplet volume prediction model.

[0042] A droplet volume image calibration system based on machine learning, comprising:

[0043] An acquisition module, configured to acquire known data of multiple training objects;

[0044] A training module, configured to process the known data of multiple training objects and perform training by means of machine learning to obtain an optimized droplet volume prediction model;

[0045] A prediction module for inputting known data of a to-be-tested droplet into the optimized droplet volume prediction model for prediction, so as to obtain a predicted volume value of the to-be-tested droplet through machine learning methods.

[0046] This application has at least the following beneficial effects:

[0047] This application provides a method and system for calibrating droplet volume images based on machine learning. The method includes: obtaining known data of multiple training objects; processing the known data of multiple training objects and training through machine learning methods to obtain an optimized droplet volume prediction model; inputting the known data of the to-be-tested droplet into the optimized droplet volume prediction model for prediction, so as to obtain a predicted volume value of the to-be-tested droplet through machine learning methods. Thus, the spatio-temporal information between droplet images is successfully captured, and the change rules of droplets at different time points are deeply learned. This method has higher accuracy and stability than traditional image calibration.

[0048] Furthermore, the method for calibrating droplet volume images based on machine learning in this application is applied to pipetting devices such as ultrasonic pipetting devices, piezoelectric-driven pipettes, and pneumatic pipetting devices. It can be seen that this method is not only applicable to calibrating droplet volume in ultrasonic pipetting technology, but also applicable to calibrating droplet volume in other technologies, and its application scope is wide.

[0049] Furthermore, the known data of multiple training objects in this application includes video data of multiple spheres and video data of multiple droplets; processing the known data of multiple training objects and training through machine learning methods to obtain an optimized droplet volume prediction model includes: extracting images from the video data of multiple spheres and the video data of multiple droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple droplets, effectively extracting the spatial and temporal features of the images; using multiple first image sequences as first sample data and training the first sample data through machine learning methods to obtain a droplet volume prediction model; using multiple second image sequences as second sample data and training the second sample data through machine learning methods for secondary training to optimize the droplet volume prediction model and obtain an optimized droplet volume prediction model. Since this application predicts the volume of pipetting droplets, spheres and droplet data with similar droplet shapes are used for machine learning, thus successfully capturing the spatio-temporal information between droplet images, deeply learning the change rules of droplets at different time points, and then obtaining a droplet predicted volume with higher accuracy and stronger stability than the traditional image calibration method through machine learning methods. Description of the Drawings

[0050] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 It is the first flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0052] Figure 2 It is the second flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0053] Figure 3 It is the third flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0054] Figure 4 It is the fourth flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0055] Figure 5 It is the fifth flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0056] Figure 6 It is the sixth flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0057] Figure 7 It is the seventh flowchart of the droplet volume image calibration method based on machine learning in the present application;

[0058] Figure 8 It is the module schematic diagram of the droplet volume image calibration system based on machine learning in the present application;

[0059] Figure 9 It is the schematic diagram of the data processing flow in Embodiment 1 of the present application;

[0060] Figure 10 It is the structural schematic diagram of the pre-trained machine learning model.

[0061] Reference numerals:

[0062] 1 - Acquisition module; 2 - Training module; 3 - Prediction module. Detailed implementation manners

[0063] In the following, various embodiments of the present application will be described more comprehensively. The present application can have various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit the various embodiments of the present application to the specific embodiments disclosed herein, but the present application should be understood to cover all adjustments, equivalents, and / or alternative solutions that fall within the spirit and scope of the various embodiments of the present application.

[0064] In the following, the term "comprising" or "may comprise" that can be used in various embodiments of the present application indicates the presence of the disclosed functions, operations, or elements, and does not limit the addition of one or more functions, operations, or elements. In addition, as used in various embodiments of the present application, the terms "comprising", "having", and their cognates are only intended to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be construed as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or the possibility of adding one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0065] As Figure 1 shown, a machine learning-based calibration method for droplet volume images includes:

[0066] S1: Obtain known data of multiple training objects;

[0067] S2: Process the known data of multiple training objects and train them through machine learning methods to obtain a tuned droplet volume prediction model;

[0068] S3: Input the known data of the droplet to be measured into the tuned droplet volume prediction model for prediction, so as to obtain the predicted volume value of the droplet to be measured through machine learning methods.

[0069] The present application obtains the predicted volume value of the droplet to be measured through machine learning methods, thereby successfully capturing the spatio-temporal information between droplet images and deeply learning the change rules of droplets at different time points. This method has higher accuracy and stability than traditional image calibration.

[0070] The machine learning-based calibration method for droplet volume images is applied to pipetting devices such as ultrasonic pipetting devices, piezoelectric-driven pipettes, and pneumatic pipetting devices. It can be seen that this method is not only applicable to droplet volume calibration in ultrasonic pipetting technology, but also applicable to droplet volume calibration in other technologies, and its application scope is wide.

[0071] As Figure 2As shown, the known data of multiple training objects in this application includes video data of multiple spheres and video data of multiple droplets; processing the known data of multiple training objects and training through machine learning methods to obtain a tuned droplet volume prediction model includes: extracting images from the video data of multiple spheres and the video data of multiple droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple droplets, effectively extracting the spatial and temporal features of the images; using multiple first image sequences as first sample data and training the first sample data through machine learning methods to obtain a droplet volume prediction model; using multiple second image sequences as second sample data and training the second sample data through machine learning methods for secondary training to optimize the droplet volume prediction model to obtain a tuned droplet volume prediction model. Since this application predicts the volume of pipetted droplets, spheres and droplet data with similar droplet shapes are used for machine learning, thus successfully capturing the spatio-temporal information between droplet images, deeply learning the variation rules of droplets at different time points, and then obtaining a more accurate and stable predicted droplet volume than the traditional image calibration method through machine learning methods.

[0072] Among them, the machine learning method used for training includes the combination of a three-dimensional residual neural network and a long short-term memory network, or the machine learning method used for training includes the combination of a convolutional neural network and a fully connected network.

[0073] In one embodiment, the machine learning method used for training includes the combination of a three-dimensional residual neural network and a long short-term memory network. The three-dimensional residual neural network well solves the problem of gradient disappearance faced by deep neural networks and can well capture the spatial information of droplet images. And considering that there is temporal information during the transfer of droplets, a long short-term memory network is added to the network module in this application. The long short-term memory network can well capture the information of time series. Thus, the neural network learns the information in both spatial and temporal dimensions of droplet data. This method can bypass the cumbersome fluorescence calibration process and overcome the deficiencies in the measurement accuracy and stability of traditional image calibration, thereby more accurately calibrating the droplet volume.

[0074] In other embodiments, the machine learning method used for training includes the combination of a convolutional neural network and a fully connected network. The combination of a convolutional neural network and a fully connected network also has significant advantages in aspects such as feature extraction and image processing, as well as in processing non-image data and sequence data and strong generalization ability.

[0075] Among them, the parameter values of the sphere include the diameter of the sphere or the area value of the sphere; the parameter values of the droplet include the diameter of the droplet or the area value of the droplet. In this application, the first image sequence and the second image sequence are calibrated for diameter or area information, and the volume information is predicted by a machine learning method using a neural network.

[0076] Among them, the data volume of the first sample data is greater than that of the second sample data. Through one-time training with a larger number of the first sample data, effective training weights are obtained to enable it to learn the spatial and temporal characteristics of the sphere and better generalize on the droplet data. On this basis, through secondary training with a smaller number of the second sample data, the droplet volume prediction model is further optimized to obtain a tuned droplet volume prediction model, so that this application can obtain stable and accurate liquid volume with less data volume.

[0077] As Figure 3 shown, extracting images from the video data of multiple said spheres and the video data of multiple droplets, and obtaining a first image sequence containing the parameter values of the spheres and a second image sequence containing the parameter values of the droplets includes:

[0078] Extracting images from the video data of multiple spheres and the video data of multiple droplets, and obtaining the images to be calibrated in the video data of each sphere and the images to be calibrated in the video data of each droplet;

[0079] Performing normalization processing on all the images to be calibrated of the spheres and all the images to be calibrated of the droplets, and setting labels for the respective images to be calibrated according to the parameter values of the spheres and the parameter values of the droplets, so as to obtain a first image sequence calibrated with the parameter values of the spheres and a second image sequence calibrated with the parameter values of the droplets.

[0080] This application processes the video data of multiple spheres and the video data of multiple droplets to obtain the images to be calibrated in the video data of each sphere and the images to be calibrated in the video data of each droplet, so as to effectively extract the spatial features and temporal features of the images through image spatio-temporal data processing; by normalizing all the images to be calibrated, the features between different images can be compared and analyzed on the same scale, improving the training efficiency and prediction performance of the machine learning model, enhancing the robustness of the model, and enabling it to give stable prediction results in the face of different lighting conditions, shooting angles or image qualities; by setting labels for the images to be calibrated corresponding to the spheres and droplets according to the parameter values of the spheres and the parameter values of the droplets, a first image sequence calibrated with the parameter values of the spheres and a second image sequence calibrated with the parameter values of the droplets are obtained, so that each image sequence is calibrated with volume information, which is convenient for subsequent input into a pre-trained machine learning model for training to obtain the corresponding relationship between the parameter values and the motion trajectories.

[0081] Specifically, as Figure 4 shown, the image extraction of the video data of multiple spheres and the video data of multiple droplets, and obtaining the images to be calibrated in the video data of each sphere and the images to be calibrated in the video data of each droplet includes:

[0082] Performing image extraction on the video data of multiple spheres and the video data of multiple droplets to obtain multiple consecutive frames of images in the video data of each sphere and multiple consecutive frames of images in the video data of each droplet;

[0083] Based on the multiple consecutive frames of images of all spheres and the multiple consecutive frames of images of all droplets, selecting the regions of interest of all spheres and the regions of interest of all droplets;

[0084] Performing binarization processing on the multiple consecutive frames of images of all spheres and the multiple consecutive frames of images of all droplets, and generating the images to be calibrated of the spheres and the images to be calibrated of the droplets according to the selected regions of interest of the spheres and the regions of interest of the droplets.

[0085] This application processes the video data of multiple spheres and multiple droplets, extracts multiple consecutive frames of images in the video data of each object, then selects the regions of interest of each object, performs binarization processing on all the multiple consecutive frames of images, and generates the images to be calibrated of the spheres and the images to be calibrated of the droplets according to the regions of interest, thereby reducing the interference of irrelevant information and improving the efficiency and accuracy of subsequent processing.

[0086] Among them, the multi-frame continuous images of the sphere are the multi-frame key continuous images of the sphere, and the multi-frame continuous images of the droplet are the multi-frame key continuous images of the droplet. By selecting the most representative and informative key frames for processing, during the training process, the processed data is input into the model, achieving a higher accuracy and stability than the traditional image calibration method.

[0087] Among them, obtaining the known data of multiple training objects includes obtaining the video data of multiple spheres and obtaining the video data of multiple droplets.

[0088] Such as Figure 5 shown, obtaining the video data of multiple spheres includes:

[0089] Build a first platform with a pipeline, a shooting device, a light source, and a measuring device, push multiple spheres into the pipeline, and the pipeline is configured to allow the spheres to fall freely when opened;

[0090] Adjust the positions of the shooting device, the light source, and the measuring device so that the shooting device can capture the falling trajectory of the spheres and the measuring device;

[0091] Open the pipeline to allow multiple spheres to fall freely to obtain the video data of multiple spheres and use it as the known data of multiple spheres.

[0092] Before implementing data acquisition in this application, by building a first platform for obtaining sphere data, the standardization of the experimental environment is achieved. By adjusting the positions of the shooting device, the light source, and the measuring device, the shooting conditions are optimized, ensuring the high quality and distinctiveness of the video data for subsequent data processing.

[0093] Such as Figure 6 shown, obtaining the video data of multiple droplets includes:

[0094] Build a second platform with an ultrasonic transducer, a liquid reservoir, a target plate, a light source, a shooting device, and a measuring device. The ultrasonic transducer is configured to move the droplets in the liquid reservoir to the target plate when working;

[0095] Adjust the positions of the shooting device, the light source, and the measuring device so that the shooting device can capture the movement trajectory of the droplets and the measuring device;

[0096] Adjust the excitation parameters of the ultrasonic transducer to eject multiple droplets to obtain the video data of multiple droplets and use it as the known data of multiple droplets.

[0097] Before implementing data acquisition, this application standardizes the experimental environment through the first platform built for obtaining droplet data. By adjusting the positions of the imaging device, light source, and measurement device, the imaging conditions are optimized to ensure high-quality and distinct video data for subsequent data processing. By adjusting the excitation parameters of the ultrasonic transducer, droplets of different volumes are ejected, facilitating data acquisition of the droplets.

[0098] As Figure 7 shown, multiple first image sequences are used as the first sample data, and the first sample data is trained once through machine learning methods to obtain a droplet volume prediction model, including:

[0099] Multiple first image sequences are used as the first sample data and divided into a first training set, a first validation set, and a first test set according to a first preset ratio;

[0100] The first training set is trained through machine learning methods, and the training process is verified based on the first validation set to obtain a first model to be tested; the first test set is input into the first model to be tested for testing, and the first model to be tested is adjusted based on the test results to obtain a droplet volume prediction model.

[0101] In this application, multiple first image sequences are divided into a first training set, a first validation set, and a first test set according to a first preset ratio. The first training set is input into a pre-trained machine learning model for training, the training process of the pre-trained machine learning model is verified based on the first validation set, a first model to be tested is obtained, the first test set is input into the first model to be tested for testing, and the first model to be tested is adjusted based on the test results, so that a more accurate and reliable droplet volume prediction model can be obtained.

[0102] Furthermore, multiple second image sequences are used as the second sample data, and the second sample data is trained twice through machine learning methods to optimize the droplet volume prediction model and obtain a tuned droplet volume prediction model, including:

[0103] Multiple second image sequences are used as the second sample data and divided into a second training set, a second validation set, and a second test set according to a second preset ratio;

[0104] The second training set is trained through machine learning methods, and the training process is verified based on the second validation set to optimize the droplet volume prediction model and obtain a second model to be tested; the second test set is input into the second model to be tested for testing, and the second model to be tested is adjusted based on the test results to obtain a tuned droplet volume prediction model.

[0105] In this application, multiple second image sequences are divided into a second training set, a second validation set, and a second test set according to a second preset ratio. The second training set is input into the droplet volume prediction model for training. The training process of the pre-trained machine learning model is verified based on the second validation set to obtain a second model to be tested. The second test set is input into the second model to be tested for testing, and the second model to be tested is adjusted based on the test results, so as to obtain a more accurate and reliable optimized droplet volume prediction model.

[0106] Among them, processing the known data of the droplet to be measured to obtain at least one processed image of the droplet to be measured; inputting at least one processed image of the droplet to be measured into the optimized droplet volume prediction model for prediction, so as to obtain the predicted volume value of the droplet to be measured through machine learning includes:

[0107] Processing the original video data or original image data in the known data of the droplet to be measured to obtain at least one processed image of the droplet to be measured; inputting at least one processed image of the droplet to be measured into the optimized droplet volume prediction model for prediction, so as to obtain the predicted volume value of the droplet to be measured through machine learning. The predicted volume value obtained by machine learning has higher accuracy and higher stability compared with traditional methods.

[0108] Furthermore, based on the method for calibrating droplet volume images based on machine learning, this application also proposes a system for calibrating droplet volume images based on machine learning, which is used to implement the above method. Specifically:

[0109] As Figure 8 shown, a system for calibrating droplet volume images based on machine learning includes:

[0110] An acquisition module 1, configured to acquire the known data of multiple training objects;

[0111] A training module 2, configured to process the known data of multiple training objects and perform training through machine learning to obtain an optimized droplet volume prediction model;

[0112] A prediction module 3, configured to input the known data of the droplet to be measured into the optimized droplet volume prediction model for prediction, so as to obtain the predicted volume value of the droplet to be measured through machine learning.

[0113] Through the mutual cooperation among the acquisition module 1, the training module 2, and the prediction module 3 in this application, the spatial features and temporal features of the image are effectively extracted, the spatio-temporal information between droplet images is successfully captured, and the change law of the droplet at different time points is deeply learned, so that the predicted droplet volume has higher accuracy and higher stability than the droplet volume calibrated by the traditional image calibration method.

[0114] A machine learning-based calibration system for droplet volume images is applied to pipetting devices such as ultrasonic pipetting devices, piezoelectric-driven pipettes, and pneumatic pipetting devices. It can be seen that this system is not only applicable to the calibration of droplet volume in ultrasonic pipetting technology, but also applicable to the calibration of droplet volume in other technologies, with a wide range of applications.

[0115] Among them, the known data of multiple training objects includes video data of multiple spheres and video data of multiple droplets;

[0116] The training module 2 includes:

[0117] An extraction module for extracting the video data of multiple spheres and the video data of multiple droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple droplets;

[0118] A learning module for using multiple first image sequences as first sample data and performing a first training on the first sample data through machine learning to obtain a droplet volume prediction model; using multiple second image sequences as second sample data and performing a second training on the second sample data through machine learning to optimize the droplet volume prediction model and obtain a tuned droplet volume prediction model.

[0119] Through the mutual cooperation between the extraction module and the learning module in this application, the spatial features and temporal features of the images are effectively extracted, the spatio-temporal information between the droplet images is successfully captured, and the change rules of the droplets at different time points are deeply learned, making the predicted droplet volume have higher accuracy and higher stability than the droplet volume calibrated by the traditional image calibration method.

[0120] The extraction module includes:

[0121] A first processing module for extracting the video data of multiple spheres and the video data of multiple droplets to obtain the to-be-calibrated images in the video data of each sphere and the to-be-calibrated images in the video data of each droplet;

[0122] A second processing module for normalizing the to-be-calibrated images of all spheres and the to-be-calibrated images of all droplets, and setting labels for the respective to-be-calibrated images according to the parameter values of the spheres and the parameter values of the droplets to obtain a first image sequence calibrated with the parameter values of the spheres and a second image sequence calibrated with the parameter values of the droplets.

[0123] Through the cooperation of the first processing module and the second processing module, the present application enables the features between different images to be compared and analyzed on the same scale, improves the training efficiency and prediction performance of the machine learning model, enhances the robustness of the model, and facilitates subsequent input into the pre-trained machine learning model for training to obtain the corresponding relationship between the parameter values and the motion trajectories.

[0124] The first processing module includes:

[0125] An image extraction module, configured to extract images from the video data of multiple spheres and the video data of multiple droplets, and obtain multiple consecutive images in the video data of each sphere and multiple consecutive images in the video data of each droplet;

[0126] A region of interest selection module, configured to select the regions of interest of all spheres and the regions of interest of all droplets based on the multiple consecutive images of all spheres and the multiple consecutive images of all droplets;

[0127] An image generation module, configured to perform binarization processing on the multiple consecutive images of all spheres and the multiple consecutive images of all droplets, and generate the to-be-calibrated images of the spheres and the to-be-calibrated images of the droplets according to the selected regions of interest of the spheres and the regions of interest of the droplets.

[0128] Through the cooperation among the image extraction module, the region of interest selection module, and the image generation module, the present application reduces the interference of irrelevant information in the images and improves the efficiency and accuracy of subsequent processing.

[0129] Among them, the acquisition module includes a first acquisition module and a second acquisition module.

[0130] The first acquisition module includes:

[0131] A first construction module, configured to construct a first platform with a pipeline, a shooting device, a light source, and a measuring device, push multiple spheres into the pipeline, and the pipeline is configured to allow the spheres to freely fall when opened;

[0132] A first adjustment module, configured to adjust the positions of the shooting device, the light source, and the measuring device so that the shooting device can capture the falling trajectory of the spheres and the measuring device;

[0133] A first working module, configured to open the pipeline to allow multiple spheres to freely fall to obtain the video data of multiple spheres;

[0134] The second acquisition module includes:

[0135] A second construction module, configured to construct a second platform with an ultrasonic transducer, a liquid reservoir, a target plate, a light source, a shooting device, and a measuring device, and the ultrasonic transducer is configured to move the droplets in the liquid reservoir to the target plate when working;

[0136] A second adjustment module, configured to adjust the positions of the photographing device, the light source, and the measuring device so that the photographing device can photograph the movement trajectory of the droplet and the measuring device;

[0137] A second working module, configured to adjust the excitation parameters of the ultrasonic transducer to eject a plurality of droplets to obtain video data of the plurality of droplets.

[0138] The training module includes:

[0139] A first training module, configured to use a plurality of first image sequences as first sample data and perform a first training on the first sample data by means of machine learning to obtain a droplet volume prediction model;

[0140] A second training module, configured to use a plurality of second image sequences as second sample data and perform a second training on the second sample data by means of machine learning to optimize the droplet volume prediction model to obtain a tuned droplet volume prediction model.

[0141] Specifically, the first training module includes:

[0142] A first partitioning module, configured to use a plurality of first image sequences as first sample data and partition them into a first training set, a first validation set, and a first test set according to a first preset ratio;

[0143] A first learning module, configured to train the first training set by means of machine learning and verify the training process based on the first validation set to obtain a first model to be tested; input the first test set into the first model to be tested for testing, and adjust the first model to be tested based on the test results to obtain a droplet volume prediction model.

[0144] Through the cooperation of the first partitioning module and the first learning module of the present application, a more accurate and reliable droplet volume prediction model is obtained.

[0145] Further, the second training module includes:

[0146] A second partitioning module, configured to use a plurality of second image sequences as second sample data and partition them into a second training set, a second validation set, and a second test set according to a second preset ratio;

[0147] A second learning module, configured to train the second training set by means of machine learning and verify the training process based on the second validation set to optimize the droplet volume prediction model to obtain a second model to be tested; input the second test set into the second model to be tested for testing, and adjust the second model to be tested based on the test results to obtain a tuned droplet volume prediction model.

[0148] Through the cooperation of the second partitioning module and the second learning module, this application obtains a more accurate and reliable droplet volume prediction model.

[0149] Example 1

[0150] As Figure 9 、 10 shown, this application provides an example where the sphere is a steel ball, and experiments are conducted by collecting 200 groups of video data of steel balls and 78 groups of video data of droplets.

[0151] The collection process of steel ball data is as follows: In the experiment, a fine needle is used to push the steel ball into the pipeline, and the positions of the light source, the shooting device, and the measuring device are adjusted to ensure that the shooting device can clearly capture the movement trajectory of the steel ball and the measuring device. To a certain extent, the light source brightness may affect the accuracy of traditional image calibration. Therefore, it is necessary to collect the video data of the steel ball under the condition of uniform illumination within the field of view. In addition, a measuring device (such as a vernier caliper) is used to measure the diameter of the steel ball. For each steel ball, its diameter is measured three times, and the average value of all measured values is taken as the final diameter.

[0152] In this experiment, the volume of the steel ball is calculated through the final diameter, and steel ball data with volumes of 0.036mm 3 、0.061mm 3 、0.065mm 3 and 0.108mm 3 are used. Among them, for the steel balls with volumes of 0.036mm 3 and 0.061mm 3 , 30 groups of videos are collected for each volume; for the steel balls with volumes of 0.065mm 3 and 0.108mm 3 , 70 groups of videos are collected for each volume.

[0153] The collection process of ultrasonic pipetting droplet data is as follows: Adjust the positions of the light source, the shooting device, and the measuring device to ensure that the shooting device can clearly capture the movement trajectory of the steel ball and the measuring device; transfer the droplet from the liquid storage to the target plate through the second platform, and control the droplet volume by adjusting the excitation parameters.

[0154] In this experiment, the volume of the droplet is calibrated by fluorescence method, and droplet data with volumes of 2.93 nl, 2.95 nl, 3.06 nl, and 3.08 nl are used.

[0155] After obtaining the video data of the steel ball and the droplet, it is also necessary to process the video data of the steel ball and the droplet. The processing steps of the two types of video data are basically the same. As Figure 8As shown, first, ten original images are intercepted from the video data as an image set, and the regions of interest of the steel balls and droplets are selected. Subsequently, the image sets of the steel balls and droplets are binarized and the regions of interest are extracted. At the same time, ten pixel whiteboards are created. Finally, the extracted and binarized regions of interest are placed at the center of the pixel whiteboards. In this way, the boundaries, textures, and contour information of the binarized steel balls and droplets can be presented more clearly. To ensure the stability of the neural network fitting, before training, the binarized images need to be adjusted to a size of 145×145×10, and the data of the steel balls and droplets are normalized. And labels are set for the respective binarized images according to the volumes of the steel balls and droplets to obtain multiple images calibrated with volumes.

[0156] After obtaining the steel ball images and droplet images calibrated with volumes, machine learning is required. In one training stage, 200 pairs of steel ball data need to be divided into a training set, a validation set, and a test set, with proportions of 55%, 26.5%, and 18.5% respectively. The pre-trained machine learning model learns the spatial features of steel balls with volumes of 0.036mm3, 0.061mm3, 0.065mm3, and 0.108mm3, and selects steel balls with volumes of 0.036mm3 and 0.065mm3 for testing. At the same time, to evaluate the influence of larger-volume steel balls on the calibration, steel balls with a volume of 0.108mm3 are additionally selected for testing.

[0157] Transfer learning is a machine learning technique that improves the learning efficiency and performance of the target task by leveraging pre-trained models. Furthermore, the present application also uses 78 sets of droplet data for transfer learning, and divides the data set into a training set, a validation set, and a test set according to a ratio of 23:8:8, further optimizing the model to obtain a tuned droplet volume prediction model.

[0158] Among them, as Figure 10As shown, when constructing a pre-trained machine learning model, a three-dimensional convolutional block needs to be designed, which includes a convolutional layer and a normalization layer. The convolutional kernel size is set to 3×3×3, and the stride is set to 1 to extract initial features. Next, four residual blocks need to be constructed. The output channel numbers of the four residual blocks are set to 64, 128, 256, and 512 respectively, and the strides are 1, 2, 2, and 2 respectively. Residual connections can reduce feature loss caused by the increase in the number of network layers and solve the problem of vanishing gradients. Subsequently, a three-dimensional adaptive pooling layer needs to be designed to adjust the data dimension, and a long short-term memory network is used to extract the features of the time series. At the end of the network, a fully connected linear layer needs to be designed to map the final output to the volume value of the steel ball or droplet. The pre-trained machine learning model combines a three-dimensional convolutional neural network and a recurrent neural network, in which a long short-term memory network module is used. Then, based on this pre-trained machine learning model, it is trained with steel ball data and droplet data, and the optimized droplet volume prediction model obtained also combines a three-dimensional convolutional neural network and a recurrent neural network, in which a long short-term memory network module is used, fully mining and utilizing the complex time information and spatial information feature information in the droplet shape segmentation map. It not only considers the spatial information of the video data during the pipetting process, but also uses a long short-term memory network to capture the information in the time dimension.

[0159] In addition, in addition to the network structure of the three-dimensional convolutional neural network and the recurrent neural network, in which a long short-term memory network module is used, the encoder structure in the neural network U-Net and the fully connected layer model can also be combined to realize the calibration of the droplet volume.

[0160] In this embodiment, through one-time training of the steel ball data, effective pre-trained weights are obtained, enabling the model to learn the spatial and time features of the steel ball a priori and generalize better on the droplet data. On this basis, secondary training of the droplet data is carried out to optimize the model, so as to obtain more accurate droplet volume prediction results.

[0161] Among them, during the model training process, calculations are performed using NVIDIA GeForce RTX 3090. In the training stage of the steel ball data, the number of training times is set to 350, the learning rate is set to 0.001, and the batch size is set to 10. In the fine-tuning stage, the learning rate is kept at 0.001, the number of training times is adjusted to 230, and the batch size is still 10 to obtain good evaluation indicators on the droplet data.

[0162] In various embodiments of the present application, the expression "or" or "at least one of A or / and B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A or / and B" may include A, may include B, or may include both A and B.

[0163] Expressions used in various embodiments of the present application (such as "first", "second", etc.) may modify various constituent elements in the various embodiments, but do not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only for the purpose of distinguishing one element from other elements. For example, the first user device and the second user device indicate different user devices, although both are user devices. For example, without departing from the scope of the various embodiments of the present application, the first element may be referred to as the second element, and similarly, the second element may also be referred to as the first element.

[0164] It should be noted that in the present application, unless otherwise clearly specified and defined, terms such as "installed", "connected", "fixed", etc. should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or an integral connection; it may be a mechanical connection or an electrical connection; it may be a direct connection or an indirect connection through an intermediate medium; it may be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to specific circumstances.

[0165] In the present application, those of ordinary skill in the art need to understand that the terms indicating orientation or positional relationship in the text are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation to the present application.

[0166] The terms used in the various embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the various embodiments of the present application. As used herein, the singular form is intended to also include the plural form, unless the context clearly indicates otherwise. Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the various embodiments of the present application belong. The terms (such as those defined in a commonly used dictionary) will be interpreted as having the same meaning as the contextual meaning in the relevant technical field and will not be interpreted as having an idealized meaning or an overly formal meaning, unless clearly defined in the various embodiments of the present application.

Claims

1. A calibration method for droplet volume images based on machine learning, characterized in that, Including: Obtaining the known data of multiple training objects; Processing the known data of multiple said training objects and training through machine learning methods to obtain a tuned droplet volume prediction model; Inputting the known data of the droplet to be measured into the tuned droplet volume prediction model for prediction, so as to obtain the predicted volume value of the droplet to be measured through machine learning methods.

2. The method for calibrating a droplet volume image based on machine learning according to claim 1, wherein The machine learning-based droplet volume image calibration method is applied to a pipetting device.

3. The method for calibrating a droplet volume image based on machine learning according to claim 1 or 2, characterized in that The known data of multiple said training objects includes the video data of multiple spheres and the video data of multiple droplets; The processing the known data of multiple said training objects and training through machine learning methods to obtain a tuned droplet volume prediction model includes: Performing image extraction on the video data of multiple said spheres and the video data of multiple droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple droplets; Taking multiple said first image sequences as first sample data and performing primary training on the first sample data through machine learning methods to obtain a droplet volume prediction model; taking multiple said second image sequences as second sample data and performing secondary training on the second sample data through machine learning methods to optimize the droplet volume prediction model to obtain a tuned droplet volume prediction model.

4. The droplet volume image calibration method based on machine learning according to claim 1, characterized in that The machine learning method used for training includes the combination of a three-dimensional residual neural network and a long short-term memory network, or the machine learning method used for training includes the combination of a convolutional neural network and a fully connected network.

5. The method for calibrating a droplet volume image based on machine learning according to claim 3, wherein The parameter values of the sphere include the diameter of the sphere or the area value of the sphere; the parameter values of the droplet include the diameter of the droplet or the area value of the droplet; And / or, the data volume of the first sample data is greater than the data volume of the second sample data.

6. The method for calibrating a droplet volume image based on machine learning according to claim 3, wherein The performing image extraction on the video data of multiple said spheres and the video data of multiple droplets to obtain a first image sequence containing parameter values of multiple spheres and a second image sequence containing parameter values of multiple droplets includes: Performing image extraction on the video data of multiple said spheres and the video data of multiple droplets to obtain the images to be calibrated in the video data of each sphere and the images to be calibrated in the video data of each droplet; Normalizing all the images to be calibrated of the spheres and all the images to be calibrated of the droplets, and setting labels for the respective images to be calibrated according to the parameter values of the spheres and the parameter values of the droplets, so as to obtain a first image sequence calibrated with the parameter values of the spheres and a second image sequence calibrated with the parameter values of the droplets.

7. The method for calibrating a droplet volume image based on machine learning according to claim 6, wherein The performing image extraction on the video data of multiple said spheres and the video data of multiple droplets to obtain the images to be calibrated in the video data of each sphere and the images to be calibrated in the video data of each droplet includes: Performing image extraction on the video data of multiple said spheres and the video data of multiple droplets to obtain multiple consecutive images in the video data of each sphere and multiple consecutive images in the video data of each droplet; Based on the multi-frame continuous images of all spheres and the multi-frame continuous images of all droplets, select the regions of interest of all spheres and the regions of interest of all droplets; Perform binarization processing on the multi-frame continuous images of all spheres and the multi-frame continuous images of all droplets, and generate the to-be-calibrated images of the spheres and the to-be-calibrated images of the droplets according to the selected regions of interest of the spheres and the droplets.

8. The method for calibrating a droplet volume image based on machine learning according to claim 1 or 2, characterized in that, The obtaining the known data of multiple training objects includes obtaining the video data of multiple spheres and obtaining the video data of multiple droplets; The obtaining the video data of multiple spheres includes: Build a first platform with a pipeline, a shooting device, a light source, and a measuring device, and push multiple spheres into the pipeline, and the pipeline is configured to allow the spheres to freely fall when opened; Adjust the positions of the shooting device, the light source, and the measuring device so that the shooting device can capture the falling trajectory of the spheres and the measuring device; Open the pipeline to allow multiple spheres to freely fall to obtain the video data of multiple spheres; And / or, the obtaining the video data of multiple droplets includes: Build a second platform with an ultrasonic transducer, a liquid reservoir, a target plate, a light source, a shooting device, and a measuring device, and the ultrasonic transducer is configured to move the droplets in the liquid reservoir to the target plate when working; Adjust the positions of the shooting device, the light source, and the measuring device so that the shooting device can capture the movement trajectory of the droplets and the measuring device; Adjust the excitation parameters of the ultrasonic transducer to eject multiple droplets to obtain the video data of multiple droplets.

9. The method for calibrating a droplet volume image based on machine learning according to claim 3, wherein The taking multiple of the first image sequences as the first sample data and performing a first training on the first sample data by a machine learning method to obtain a droplet volume prediction model includes: Take multiple of the first image sequences as the first sample data and divide them into a first training set, a first validation set, and a first test set according to a first preset ratio; Train the first training set by a machine learning method and verify the training process based on the first validation set to obtain a first model to be tested; input the first test set into the first model to be tested for testing, and adjust the first model to be tested based on the test results to obtain the droplet volume prediction model; The taking multiple of the second image sequences as the second sample data and performing a second training on the second sample data by a machine learning method to optimize the droplet volume prediction model to obtain an optimized droplet volume prediction model includes: Take multiple of the second image sequences as the second sample data and divide them into a second training set, a second validation set, and a second test set according to a second preset ratio; Train the second training set by a machine learning method and verify the training process based on the second validation set to optimize the droplet volume prediction model to obtain a second model to be tested; input the second test set into the second model to be tested for testing, and adjust the second model to be tested based on the test results to obtain the optimized droplet volume prediction model.

10. A droplet volume image calibration system based on machine learning, characterized in that, Includes: An acquisition module for acquiring the known data of multiple training objects; A training module, configured to process the known data of multiple said training objects and perform training through machine learning methods to obtain an optimized droplet volume prediction model; A prediction module, configured to input the known data of the droplet to be measured into the optimized droplet volume prediction model for prediction, so as to obtain the predicted volume value of the droplet to be measured through machine learning methods.

Images