Automatic delamination pipetting method and system based on visual and theoretical volume joint determination

Abstract

The application relates to an automatic layering pipetting method and system based on visual and theoretical volume joint determination in the field of laboratory automation and liquid processing technology. The method constructs the existing liquid adding amount, pipetting amount and container geometric parameters in the experiment process into a software virtual sensor, forms an independent information source parallel to the visual sensing channel; outputs the continuous quantitative indicators of visual measurement height and confidence through an uncertainty estimation method; a confidence-driven adaptive fusion strategy is used for multimodal fusion of theoretical prediction and visual measurement, visual observation weight is automatically reduced when visual confidence is low, and the lower interface height is calculated according to the theoretical liquid layer thickness when the visual confidence of the lower interface is insufficient; the fusion result is used for judging the suckability and calculating the insertion depth, and a pipetting execution mechanism is driven to complete pipetting. The application realizes complementary fusion of visual and theoretical volume calculation, compensates by using theoretical prediction when visual is disturbed, and improves the accuracy and robustness of layering pipetting.

Description

Technical Field

[0001] This invention belongs to the field of laboratory automation and liquid handling technology, and relates to an automatic layered pipetting method and system based on a combination of visual and theoretical volume determination. Background Technology

[0002] In laboratory automation processes in fields such as chemistry, biology, and medicine, layered pipetting is a common and important operation. Layered pipetting refers to the precise aspiration of one layer of liquid from a container containing multiple layers of liquid with different densities without mixing with other layers. Common applications include liquid-liquid extraction, phase separation, and sample processing after gradient centrifugation.

[0003] Traditional layered pipetting relies primarily on manual operation. Operators visually observe the liquid-liquid interface and manually control the insertion depth and volume of the pipette. This method is not only inefficient but also highly dependent on the operator's experience and condition, making it difficult to guarantee pipetting accuracy and prone to cross-contamination or incomplete aspiration.

[0004] In recent years, with the development of machine vision technology, automated layered pipetting devices based on vision recognition have emerged. These devices acquire images of liquid containers using cameras, employ image processing algorithms to identify the location of the liquid layer interfaces, and then control the pipetting mechanism to perform automated pipetting. However, existing vision recognition layered pipetting technology has the following drawbacks: (1) When the liquids are similar in color, the interface is not obvious, or there are droplets or mist on the container wall, the accuracy of visual recognition will drop significantly, leading to pipetting failure. (2) Relying solely on visual recognition cannot cope with extreme situations where vision is completely ineffective, such as when the liquid is opaque, the container is opaque, or the light is completely blocked. (3) The visual recognition results lack confidence assessment, making it impossible to judge the reliability of the recognition results and making it difficult to make decision adjustments; (4) Although there are existing technical solutions that combine liquid level sensors (such as ultrasonic and laser ranging) with vision, these solutions are simply redundant superpositions of the same type of sensors (all of which are physical measurement methods), and do not fundamentally solve the problem of automatic compensation when vision fails. Moreover, the additional sensors significantly increase the system cost and hardware complexity. (5) In chemical experiments such as liquid-liquid extraction, the presence of the emulsion layer makes the interface itself unclear, and a single visual method can hardly work reliably. Existing technologies lack targeted solutions for such scenarios. Summary of the Invention

[0005] To address the problems existing in the above-mentioned traditional methods, this invention proposes an automatic layered pipetting method and system based on joint determination of vision and theoretical volume. It can still guarantee pipetting accuracy when the visual recognition effect is poor, and achieves automatic compensation for visual failure without increasing additional hardware costs. It is applicable to various complex liquid layering scenarios, including emulsion layers.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: On the one hand, an automated layered pipetting method based on joint visual and theoretical volume determination is provided, including the following steps: Step 1: Construct a software virtual sensor from the volume data of liquid added and / or pipetting during the chemical experiment, along with the geometric model parameters of the container. Use the software virtual sensor to determine the theoretical prediction results. The software virtual sensor is used to predict the height of the liquid layer interface inside the container through theoretical calculation based on the principle of volume conservation and container geometric constraints. The process noise covariance is calculated based on the volume accuracy and liquid evaporation parameters. The process noise covariance is used to characterize the uncertainty of the theoretical prediction results, providing a confidence basis for the subsequent weighted fusion of theoretical prediction and visual observation.

[0007] Step 2: A deep learning instance segmentation network is used to perform pixel-level segmentation of the liquid layer interface in the acquired liquid container image, identify the pixel position of the liquid layer interface, and perform multiple forward inferences on the visual measurement results through a neural network uncertainty estimation method. Based on the statistical distribution of the multiple inference results, the continuous quantitative indicators of visual measurement height and visual measurement confidence are calculated.

[0008] Step 3: Employ a confidence-driven adaptive fusion strategy to perform multimodal fusion of theoretical prediction results and visual measurement results to obtain the fused liquid layer interface height. During the fusion process, when the confidence of visual measurement is lower than a first preset threshold, the weight of visual observation is dynamically adjusted. When the confidence of visual measurement of the lower interface is insufficient, the visual measurement value of the upper interface with high confidence is used as the benchmark, and the lower interface height is calculated using the theoretically predicted liquid layer thickness.

[0009] Step 4: Determine the absorbability of the target liquid layer based on the height of the interface after fusion, calculate the insertion depth of the pipette, and drive the pipetting actuator to complete the pipetting.

[0010] On the other hand, an automated layered pipetting system that combines visual and theoretical volume determination is also provided, comprising: The image acquisition module, including an industrial camera and a light source system, is used to acquire side or front images of a liquid container.

[0011] The theoretical prediction module is used to construct a software virtual sensor from the volume data of liquid added and / or liquid transferred during the chemical experiment, as well as the geometric model parameters of the container. The software virtual sensor is used to determine the theoretical prediction results. The software virtual sensor is used to predict the height of the liquid layer interface in the container through theoretical calculation based on the principle of volume conservation and container geometric constraints, and to calculate the process noise covariance based on the volume accuracy parameters and liquid evaporation parameters.

[0012] The visual recognition module is used to perform pixel-level segmentation of the liquid layer interface in container images acquired by industrial cameras using a deep learning instance segmentation network, identify the pixel position of the liquid layer interface, and perform multiple forward inferences on the visual measurement results using a neural network uncertainty estimation method. Based on the statistical distribution of the multiple inference results, it calculates continuous quantitative indicators of visual measurement height and visual measurement confidence.

[0013] The multimodal fusion module is used to perform multimodal fusion of theoretical prediction results and visual measurement results output by the software virtual sensor using a confidence-driven adaptive fusion strategy to obtain the fused liquid layer interface height. During the fusion process, when the confidence of the visual measurement is lower than a first preset threshold, the weight of the visual observation is dynamically adjusted. When the confidence of the visual measurement of the lower interface is insufficient, the visual measurement value of the upper interface with high confidence is used as the benchmark, and the lower interface height is estimated by using the theoretically predicted liquid layer thickness.

[0014] The pipetting control module is used to determine the absorbability of the target liquid layer based on the height of the interface of the fused liquid layer, calculate the insertion depth of the pipetting needle, and drive the pipetting actuator to complete the pipetting. The pipetting actuator includes a three-dimensional motion platform, a pipetting needle, and a syringe pump or peristaltic pump, which is used to perform specific pipetting operations.

[0015] One of the above technical solutions has the following advantages and beneficial effects: The aforementioned automated layered pipetting method and system based on joint visual and theoretical volume determination constructs a software virtual sensor from existing experimental data on liquid addition, pipetting volume, and container geometry, forming an independent information source parallel to the visual sensing channel. It outputs continuous quantitative indicators of the visual measurement height and its confidence level using a neural network uncertainty estimation method. A confidence-driven adaptive fusion strategy performs multimodal fusion of theoretical prediction and visual measurement, automatically reducing the weight of visual observations when visual confidence is low, and calculating the lower interface height based on the theoretical liquid layer thickness when visual confidence at the lower interface is insufficient. Based on the fusion results, it determines absorbability and calculates the insertion depth, driving the pipetting actuator to complete the pipetting. This invention, through the complementary fusion of visual and theoretical volume calculations, utilizes theoretical prediction compensation when visual interference occurs, improving the accuracy and robustness of layered pipetting, and is applicable to scenarios such as liquid-liquid extraction and phase separation.

[0016] This method utilizes a multimodal fusion of theoretical volume calculation and visual recognition, leveraging the complementary advantages of both approaches. Its expected accuracy surpasses that of either single visual recognition or single theoretical calculation. When visual recognition is ineffective (e.g., unclear interfaces or interference), theoretical prediction can correct for errors, ensuring the reliability of pipetting operations. A neural network uncertainty estimation method quantifies the confidence level of visual recognition, enabling the system to perceive the reliability of the recognition results and make adaptive decisions. The theoretical volume calculation is entirely based on existing volume data and known container geometry during the liquid addition / pipette process, without increasing system hardware costs; effective compensation for visual failures can be achieved solely through software algorithms. Uncertainty estimation quantifies visual uncertainty, which is directly input into the measurement noise parameters of the fusion algorithm, achieving visual confidence-driven adaptive fusion without the need for manually setting fixed thresholds or switching recognition modes. This method is adaptable to various bottle types and liquid stratification scenarios, including transparent / semi-transparent liquids, regular / irregular containers, and liquid-liquid extraction systems with emulsion layers. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an automated layered pipetting method based on a combination of visual and theoretical volume determination in one embodiment. Figure 2 This is a flowchart of an automated layered pipetting method based on a combination of visual and theoretical volume determination in one embodiment; Figure 3 This is a schematic diagram illustrating the principle of a multimodal fusion algorithm in one embodiment; Figure 4 This is a schematic diagram illustrating the algorithm identifier for the sampling location in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0021] It should be noted that, in this document, the reference to "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments. The term "and / or" as used herein refers to any combination of one or more of the associated listed items, and all possible combinations, including such combinations.

[0022] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0023] In one embodiment, such as Figure 1 As shown, an automated layered pipetting method based on joint visual and theoretical volume determination is provided, which may include the following processing steps 1 to 4: Step 1: Construct a software virtual sensor from the volume data of liquid added and / or pipetting during the chemical experiment, along with the geometric model parameters of the container. Use the software virtual sensor to determine the theoretical prediction results. The software virtual sensor is used to predict the height of the liquid layer interface inside the container through theoretical calculation based on the principle of volume conservation and container geometric constraints. The process noise covariance is calculated based on the volume accuracy and liquid evaporation parameters. The process noise covariance is used to characterize the uncertainty of the theoretical prediction results, providing a confidence basis for the subsequent weighted fusion of theoretical prediction and visual observation.

[0024] Specifically, in addition, Reference 1 (Publication No.: CN121490427A) proposes an intelligent control method for a fully automated liquid-liquid extraction instrument. This method uses an RGB-D depth camera combined with a deep learning algorithm to identify the liquid layer interface and performs discrete grading based on the degree of emulsification (e.g., mild, moderate, severe emulsification). Then, a preset processing strategy is selected based on the grading results. However, in this scheme, visual recognition only outputs deterministic results and does not continuously quantify the uncertainty of the recognition; it uses discrete grading and a fixed processing strategy based on preset rules for the emulsion layer, which cannot adaptively adjust according to continuous changes in visual confidence; furthermore, this scheme relies entirely on visual information and lacks a heterogeneous complementary mechanism, making it unable to guarantee reliable system operation when visual perception is severely degraded. Therefore, this application proposes theoretical liquid layer dynamic prediction, which integrates theoretical volume calculation and visual recognition in a multimodal manner, comprehensively utilizing the complementary advantages of both methods, and is expected to achieve higher accuracy than single visual recognition or single theoretical calculation.

[0025] In existing technologies, liquid level estimation relies on visual sensors or additional hardware sensors (such as ultrasonic or capacitive sensors). The former may fail under complex lighting conditions, similar refractive indices, or emulsion layer interference, while the latter increases system cost and complexity. This method utilizes naturally occurring liquid addition / pipette volume control quantities, mass conservation relationships, and container geometric constraints during the experiment to construct an independent liquid level state estimation channel without adding any hardware. Specifically, the existing liquid addition volume, pipette volume, and container geometric parameters are used to construct a software virtual sensor. Unlike physical sensors, this software virtual sensor does not rely on additional hardware but utilizes naturally occurring control quantities (liquid addition volume, pipette volume), mass conservation relationships, and container geometric constraints to construct a real-time liquid level state estimate. Therefore, it can serve as an independent information source alongside the visual sensing channel.

[0026] The physical basis for software-based virtual sensors lies in the following: During the experiment, the volumes of liquid added to and removed from the container are known control variables; according to the law of conservation of mass, the total mass of all liquid phases in the container (and the total volume at a known density) is fixed; and the cross-sectional area-height function of the container shape is a known geometric parameter. By combining these control variables, conservation relationships, and geometric constraints, the theoretical height of each liquid layer can be calculated in real time without relying on any additional physical sensors. This calculation process has an information source independent of vision, thus forming a virtual sensing channel parallel to visual sensors.

[0027] Based on volume data from liquid addition and / or transfer, and the geometric model of the container, the liquid interface height is predicted through theoretical calculation, and the process noise covariance is calculated according to volume accuracy and liquid evaporation factors. Before starting the theoretical calculation, a layered prediction is made through the experimental process template and the knowledge base of miscibility. A three-level volume calculation model with progressive accuracy (ideal immiscible model, miscible equilibrium material balance model, and empirical distribution coefficient method) is automatically selected according to the completeness of the known information of the system to calculate the actual volume of the two phases. Dynamic volume loss compensation is achieved by combining a feedforward evaporation rate model with visual feedback.

[0028] The geometric model of the container bottle includes, but is not limited to, parametric descriptions of cylindrical, conical, spherical, and irregular bottle shapes. Based on the cross-sectional area function of different bottle shapes, the correspondence between volume and height is calculated by integration.

[0029] Process noise covariance is an inherent, unavoidable random error in a single liquid addition / pipette operation, caused by factors such as the precision of the pump used for pipetting and solvent evaporation.

[0030] The core function of process noise covariance is to characterize the degree of uncertainty (error level) of theoretical prediction results, providing a mathematical basis for model confidence for subsequent Kalman filter fusion.

[0031] In this method, a software virtual sensor theoretically calculates the liquid layer interface height based on the principle of volume conservation and container geometric constraints. However, this calculation process has two inherent sources of uncertainty: (1) Limitation of volume measurement accuracy: During the liquid addition / pipette operation, there is an error between the actual liquid volume and the nominal measurement value (determined by the accuracy of the pipetting equipment). (2) Effect of liquid evaporation: Chemical reagents naturally evaporate over time during the experiment, causing the actual liquid volume to gradually deviate from the theoretical value.

[0032] The two factors mentioned above together cause the theoretical prediction to be not a deterministic true value, but an estimate with random error. This step uses the process noise covariance matrix Q to quantitatively characterize this uncertainty: the higher the volumetric accuracy and the lower the evaporation rate, the smaller the Q matrix elements (indicating that the model prediction is more reliable); conversely, the larger Q is (indicating that the model prediction is less reliable).

[0033] In subsequent steps, the Kalman filter dynamically weights and fuses the process noise covariance Q with the observation noise covariance R of the visual sensor: when Q is small (high theoretical model reliability), the fusion result tends to rely on theoretical predictions; when Q is large (high theoretical model uncertainty), the fusion result tends to rely on visual observations. This achieves optimal joint determination of visual information and theoretical volume information, significantly improving the estimation accuracy and robustness of the liquid layer interface height.

[0034] The process noise covariance is dynamically adjusted based on the volumetric accuracy of liquid addition / transfer and liquid evaporation factors.

[0035] Step 2: A deep learning instance segmentation network is used to perform pixel-level segmentation of the liquid layer interface in the acquired liquid container image, identify the pixel position of the liquid layer interface, and perform multiple forward inferences on the visual measurement results through a neural network uncertainty estimation method. Based on the statistical distribution of the multiple inference results, the continuous quantification index of visual measurement height and visual measurement confidence is calculated; wherein the continuous quantification index of confidence is the statistical variance of the interface height value in the multiple inference results.

[0036] Specifically, the images of the liquid containers were captured using industrial cameras.

[0037] Existing technologies typically classify visual confidence into discrete levels (e.g., "high / medium / low") and switch fusion strategies according to preset rules, failing to respond to intermediate states such as blurred boundaries and gradual interference. Unlike existing technologies that rely on discrete classification based on preset rules, this method outputs a continuous quantitative index of visual measurement confidence through a neural network uncertainty estimation method. This index is then used in real-time to construct the measurement noise parameter in the fusion algorithm, enabling the fusion weights of theoretical prediction and visual measurement to adaptively adjust with the continuous changes in visual uncertainty.

[0038] The value of the confidence level continuous quantization index changes continuously with the changes in image area, lighting conditions and interface clarity; this continuous index can reflect the fine-grained confidence level differences of visual recognition under different working conditions, and directly drive the continuous adjustment of the fusion weight in step 3.

[0039] Step 3: Employ a confidence-driven adaptive fusion strategy to perform multimodal fusion of theoretical prediction results and visual measurement results to obtain the fused liquid layer interface height. During the fusion process, when the confidence of visual measurement is lower than a first preset threshold, the weight of visual observation is dynamically adjusted. When the confidence of visual measurement of the lower interface is insufficient, the visual measurement value of the upper interface with high confidence is used as the benchmark, and the lower interface height is calculated using the theoretically predicted liquid layer thickness.

[0040] Specifically, existing technologies lack systematic compensation strategies when vision fails completely or partially, typically resulting in direct error reporting or regression to purely theoretical estimations. This method utilizes the physical constraints of the liquid layer system (the sum of liquid layer thicknesses is determined by volume-geometric relationships). When visual recognition confidence is insufficient in complex scenarios such as liquid interfaces (i.e., visual interface recognition fails), the high-dimensional fusion problem is decomposed into low-dimensional stable sub-problems by leveraging the physical constraints of the liquid layer system, where the sum of the thicknesses of each liquid layer is determined by volume-geometric relationships. The location of the visual failure interface is calculated using theoretical liquid layer thicknesses, enabling automatic compensation when the visual channel fails. This decomposition of the high-dimensional fusion problem into low-dimensional stable sub-problems ensures the system continues to operate even in scenarios of visual degradation.

[0041] The theoretical prediction results and visual measurement results are fused in a multimodal manner. The fusion weight is dynamically adjusted according to the continuous quantitative index of the visual measurement confidence. When the visual confidence is lower than the preset condition, the fusion weight of the visual observation is automatically reduced. When the visual confidence in the lower interface recognition is insufficient, a differentiated fusion strategy is adopted, using the visual upper interface with high confidence as the benchmark and the theoretical liquid layer thickness to estimate the height of the lower interface. As a simplified solution, a conditional weighting strategy can also be adopted.

[0042] Step 4: Determine the absorbability of the target liquid layer based on the height of the interface after fusion, calculate the insertion depth of the pipette, and drive the pipetting actuator to complete the pipetting.

[0043] Specifically, the absorbability determination includes liquid layer thickness determination, liquid surface stability determination, and pipette insertion safety determination.

[0044] The aforementioned automated stratified pipetting method, which combines visual and theoretical volume determination, constructs a software virtual sensor using existing experimental data on liquid volume, pipetting volume, and container geometry, forming an independent information source parallel to the visual sensing channel. It outputs continuous quantitative indicators of the visual measurement height and its confidence level through a neural network uncertainty estimation method. A confidence-driven adaptive fusion strategy performs multimodal fusion of theoretical prediction and visual measurement, automatically reducing the weight of visual observations when visual confidence is low, and calculating the lower interface height based on the theoretical liquid layer thickness when visual confidence at the lower interface is insufficient. Based on the fusion results, it determines absorbability and calculates the insertion depth, driving the pipetting actuator to complete the pipetting. This invention, through the complementary fusion of visual and theoretical volume calculations, utilizes theoretical prediction compensation when visual interference occurs, improving the accuracy and robustness of stratified pipetting, and is applicable to scenarios such as liquid-liquid extraction and phase separation.

[0045] In one embodiment, the process of using software virtual sensors to calculate theoretical prediction results includes: constructing a three-level volume calculation model with progressive accuracy; wherein the three-level volume calculation model includes: the first level is an ideal immiscible model, the second level is a miscible equilibrium material balance model, and the third level is an empirical distribution coefficient method; the ideal immiscible model is used to directly calculate the liquid layer thickness based on the added volume of each phase; the miscible equilibrium material balance model is used to solve for the actual volume of the two phases by combining solubility data and the mass conservation of each phase; the empirical distribution coefficient method is used to determine the volume distribution coefficient based on historical calibration data, and subsequent runs directly call the volume distribution coefficient for volume calculation; the corresponding calculation level is automatically selected according to the completeness of the known information of the system, and the theoretical prediction results are calculated; specifically, when the interoperability of the system is lower than a preset value, the first level is used to calculate the theoretical prediction results; when the system has clear miscibility data, the second level is used to calculate the theoretical prediction results; when the system is a complex reaction liquid or extraction system, the third level is used to calculate the theoretical prediction results.

[0046] Specifically, the software virtual sensor adopts a multi-layer volumetric calculation model with progressive precision, automatically selecting the appropriate calculation level based on the completeness of the known information of the system.

[0047] In one embodiment, the software virtual sensor further includes a dynamic loss compensation mechanism, employing a closed-loop strategy that combines a feedforward evaporation model with visual feedback; wherein the feedforward evaporation model estimates the amount of evaporation based on liquid volatility parameters and exposure time; and the visual feedback detects the deviation between the actual liquid level and the theoretical prediction, and corrects the evaporation rate parameter or the total volume directly based on the deviation in real time.

[0048] In one embodiment, step 1 further includes a layering prediction mechanism: the system has a built-in experimental process template and a miscibility knowledge base. The experimental process template defines the operation type of each step. When the step is marked as extraction or layering, the theoretical liquid layer calculation module is automatically activated. The miscibility knowledge base stores data on the miscibility relationships between common solvent pairs. Based on the currently added reagent combination, it automatically determines whether the layering conditions are met, forming a judgment mechanism that combines theoretical prediction with subsequent visual detection.

[0049] In one embodiment, the neural network uncertainty estimation method in step 2 employs Monte Carlo Dropout, deep ensemble, or Bayesian neural networks.

[0050] Specifically, neural network uncertainty estimation methods employ Monte Carlo Dropout, deep ensemble, or Bayesian neural networks. When using Monte Carlo Dropout, the Dropout layer is kept active during the inference phase, and N≥10 random forward propagations are performed. The mean and variance of the interface height are statistically calculated, with the mean used as the visual measurement and the variance used as the confidence index.

[0051] In one embodiment, the multimodal fusion in step 3 is implemented using a state estimation algorithm; the state estimation algorithm includes extended Kalman filtering, unscented Kalman filtering, or particle filtering; the dynamic adjustment of the weights of visual observations is achieved by constructing the continuous quantization index of visual measurement confidence as the measurement noise covariance matrix in the state estimation algorithm in real time.

[0052] In one embodiment, step 3 further includes using a conditional weighting strategy as an alternative for multimodal fusion, wherein the fusion result is a weighted sum of theoretical predictions and visual measurements, and the weights are automatically calculated based on the variance of theoretical predictions and the variance of visual measurement confidence; the weight calculation formula is:

[0053]

[0054] in, Variance of visual confidence To theoretically predict variance, , These are the weights of the theoretical prediction and the measured data, respectively.

[0055] Specifically, theoretical prediction variance: the overall uncertainty of the theoretical model's estimate of the liquid level height after multiple liquid addition / transfer operations. That is, calculating the final theoretical error after multiple accumulations of mass.

[0056] Visual confidence variance: The uncertainty of the deep learning network's output on the interface location (which changes continuously with image quality, lighting, and interface sharpness), i.e., how blurry the interface location is as captured by the camera.

[0057] Theoretical prediction weights and visual measurement weights: The design logic is "whoever has greater uncertainty has a smaller weight," meaning that if hardware accuracy is poor (pump malfunction, severe evaporation), the system will automatically rely more on vision; conversely, if vision is poor (…), the system will rely more on visual measurement. (For large ones), it is based on theoretical calculations (when the lower interface is not clear, the upper interface height minus the theoretical liquid layer thickness is used to estimate).

[0058] In one embodiment, step 4, the absorbability determination, includes liquid layer thickness determination, liquid surface stability determination, and pipette insertion safety determination. Specifically, the liquid layer thickness determination uses the height difference between two adjacent interfaces as the target liquid layer thickness. If the target liquid layer thickness is less than a preset minimum absorbable thickness, it is determined to be unabsorbable, and a warning is issued. The liquid surface stability determination calculates the liquid surface fluctuation amplitude using the fusion results of multiple consecutive frames. If the fluctuation amplitude exceeds a preset threshold, the liquid surface is determined to be unstable, and the pipetting operation is performed only after the liquid surface stabilizes. The liquid surface fluctuation amplitude is obtained by calculating the variance or range of the fusion results of multiple consecutive frames. The insertion depth planning determines the optimal insertion depth of the pipette based on the position of the target liquid layer, the preset safety distance, and the height of the aspirated liquid column.

[0059] in, To achieve the optimal insertion depth, The location of the target liquid layer. To determine the height of the liquid column being drawn, To set a safe distance.

[0060] In one embodiment, the method further includes: establishing a liquid layer history database to record the volume of liquid added / transferred, the height of the fused liquid layer interface, and the visual confidence level in each pipetting operation, for periodically optimizing the bottle geometry parameters in the theoretical prediction model and the model weights of the visual recognition network.

[0061] Specifically, by establishing a historical database of liquid layers, data is continuously accumulated to optimize theoretical models and visual recognition algorithms, thereby continuously improving system performance.

[0062] In a specific embodiment, such as Figure 2 As shown, an automated layered pipetting method based on joint visual and theoretical volume determination is provided, comprising the following steps: S101, Theoretical Liquid Layer Dynamic Prediction Based on the liquid addition / transfer volume data, chemical system properties, and bottle geometry model, the liquid layer interface height and process noise covariance are predicted through theoretical calculations. This step constructs a software virtual sensor using the existing liquid addition volume, transfer volume, and container geometry parameters from the experiment. It then utilizes naturally occurring control variables, mass conservation relationships, and container geometric constraints to construct a real-time liquid layer state estimate. This step specifically includes: (1) Stratification Prediction: The system does not determine whether liquid-liquid stratification occurs through online chemical analysis, but rather makes a priori judgment based on the experimental procedure preset and the physical property database. The system has built-in experimental procedure templates and a miscibility knowledge base. The experimental procedure templates define the operation type of each step (addition sequence, reagent type, stirring, settling, etc.). When the step is marked as "extraction" or "stratification", the stratification calculation module is automatically activated. The miscibility knowledge base stores the miscibility relationship data of common solvent pairs (such as ethyl acetate-water, dichloromethane-water, n-butanol-water, etc.), covering completely miscible, partially miscible and immiscible systems. When the added reagent combination meets the "immiscible / partially miscible" condition, and after sufficient stirring and settling, the system theoretically predicts that two phases will inevitably form. This prediction, together with subsequent visual detection, forms a judgment mechanism that combines theoretical prediction and visual detection: if the visual system does not recognize a clear interface, an abnormal alarm is triggered.

[0063] (2) Geometric modeling of liquid container bottles: Parametric modeling of commonly used liquid containers, including cylindrical, conical, spherical and irregular bottle shapes, and establishing the functional relationship between cross-sectional area and height based on the geometric parameters of various bottle shapes.

[0064] (3) Calculation of the volume of the two phases after stratification: The actual volume of each phase after stratification cannot be simply equated to the sum of the added volumes. It is necessary to comprehensively consider the miscibility, solute distribution, and volume non-additivity. The system provides three progressively more accurate calculation models, which are automatically selected based on the completeness of the known information about the system: The first level is the ideal immiscible model: applicable to systems with extremely low or negligible miscibility (such as ethyl acetate-water, dichloromethane-water). In this case, the volumes of the two phases are approximately equal to their respective amounts added. Even neglecting trace amounts of miscibility, this model can provide sufficiently robust theoretical depth in subsequent vision-theory fusion.

[0065] The second level is the miscibility equilibrium material balance: For systems with clearly defined miscibility data (such as n-butanol-water), a material balance is required. The mass of water added is known. Organic solvent quality Query the equilibrium solubility (mass fraction of organic matter in the aqueous phase) at this temperature. water mass fraction in organic phase Let the mass of the aqueous phase after equilibrium be... The organic phase mass is The water mass conservation equation is:

[0066] The mass conservation equation for organic matter is:

[0067] The masses of the two phases are obtained by solving the equations simultaneously, and then the volumes are calculated based on the densities of each phase. If data on the density variation with composition is lacking, the density of the pure component can be used as an approximation.

[0068] The third level is the empirical partition coefficient method: For reaction solutions or extraction systems with complex compositions (such as those containing multiple solutes or pH-dependent partitioning), theoretical calculations are too complex. The system supports empirical coefficients based on historical calibration data. During the method development phase, the critical volume partition coefficient is determined through one or more standard experiments:

[0069] in, Key volume distribution coefficient; This refers to the actual volume of the upper or lower organic phase after liquid-liquid extraction. This represents the actual volume of the aqueous phase after liquid-liquid extraction and separation.

[0070] This coefficient, along with other parameters, is stored in the template of this experimental method and can be directly called upon in subsequent automatic runs, resulting in high repeatability. Key volume distribution coefficient. This is an empirical ratio obtained by calibration for a specific reagent system and fixed experimental conditions (temperature, pH, solute composition, stirring intensity, settling time, etc.). It characterizes the volume ratio of the organic phase to the aqueous phase after the system reaches extraction equilibrium and achieves stable layering. This coefficient is only applicable to the corresponding system and experimental conditions. After calibration, it is stored in the corresponding experimental method template, and can be directly called up in subsequent automated experiments under the same conditions without repeated calculation.

[0071] (4) Volume-height conversion: Based on the calculated volume data of each phase, the change in liquid layer height is calculated in combination with the bottle-shaped geometric model. The height value of each liquid layer interface is obtained by dividing the volume of each phase by the cross-sectional area at the corresponding height and integrating.

[0072] (5) Dynamic Loss Compensation – Volatilization Closed-Loop: During the experiment, solvent evaporation will cause the total liquid volume to gradually decrease, resulting in a continuously high theoretical interface height, which may eventually cause the pipette to puncture the target liquid layer. This step adopts a closed-loop compensation strategy of "feedforward model + visual feedback": Feedforward aspect: For commonly used solvents (ethyl acetate, diethyl ether, dichloromethane, etc.), the evaporation rate curves are pre-determined under standard experimental conditions (bottle neck size, stirring rate, temperature). The unit is μL / min. During the experiment, the system starts timing from the moment stirring or opening, based on the current solvent type and ambient temperature, and records the evaporation rate curve. Integrating, the cumulative evaporation volume is obtained as follows:

[0073] in, To accumulate evaporation volume, For differentiation with respect to time, T The ambient temperature.

[0074] Deducted from the total liquid volume in real time Update the theoretical liquid level.

[0075] Feedback: After settling and stratification, the liquid level is accurate and reliable when visually perceived as being at a high confidence level. At this point, the system compares the observed total liquid height with the visually measured height. Total height predicted by theory (Volatilization loss has been deducted). If there is a discrepancy between the two, the difference is used to correct the evaporation rate parameter in real time or directly compensate the total volume to eliminate accumulated errors; after eliminating accumulated errors from the theoretically predicted total height, the theoretical total liquid height corrected by visual feedback is obtained as follows:

[0076] in, This is the theoretical total liquid height after visual feedback correction.

[0077] Each layer cycle automatically calibrates to ensure high accuracy even during long-term operation.

[0078] (6) Estimation of process noise covariance: Based on the volumetric accuracy of liquid addition / transfer and the liquid evaporation factor, the theoretically predicted process noise covariance is calculated as follows:

[0079] in, The theoretically predicted process noise covariance, The height of the liquid layer interface. This represents the cumulative volume of liquid added / transferred. This represents the variance corresponding to volumetric accuracy. The variance corresponding to the liquid evaporation factor. The symbol represents the partial derivative.

[0080] The process noise covariance will be dynamically adjusted according to the actual operating conditions.

[0081] S102, Visual Recognition and Uncertainty Quantification A deep learning instance segmentation algorithm combined with an uncertainty estimation method is used to perform visual recognition on the acquired images, outputting visually measured height and confidence level. In this embodiment, the uncertainty estimation method is implemented using Monte Carlo Dropout, but it can also be replaced by deep ensemble or Bayesian neural networks. This step specifically includes: (1) Image acquisition: Acquire side or front images of the liquid container using an industrial camera. The camera should be a high-resolution industrial camera with a resolution of no less than 1920×1080 and a frame rate of no less than 30fps. Use a backlight or sidelight illumination system to enhance the contrast of the liquid layer interface.

[0082] (2) Instance segmentation: A deep learning instance segmentation algorithm is used to process the image and identify the interface locations of each liquid layer. This embodiment uses the U-Net network structure, which performs well in fields such as medical image segmentation and can effectively handle small samples and blurred boundaries. The network input is the original image, and the output is the segmentation mask of each liquid layer.

[0083] (3) Monte Carlo Dropout Confidence Quantification: During the network inference phase, the Dropout layer is kept active, and N forward propagations (N=20 in this embodiment) are performed to obtain N segmentation results. The height values ​​of each liquid layer interface are statistically analyzed: the mean is the arithmetic mean of the N forward propagation results, which is used as the height value for visual measurement; the variance is the sample variance of the N results, which is used as the confidence index for visual measurement. The larger the variance, the lower the confidence.

[0084] (4) Pixel-to-height conversion: Based on the camera calibration parameters, the pixel positions of the interfaces identified in the image are converted into actual physical heights. The true height of each liquid layer is calculated by combining the pre-calibrated pixel-to-height mapping relationship with the actual size of the container.

[0085] The continuous quantitative index of visual measurement confidence is obtained through three steps: "neural network uncertainty estimation + multiple forward inferences + statistical quantification", as follows: (1) Multiple random forward inference: During the neural network inference stage, the Dropout layer (or other random perturbation mechanism) is kept active, and the same frame of input image is forwarded N times (e.g., N=20). Each forward propagation produces slightly different segmentation results due to the random discarding of different neurons by Dropout, thereby obtaining the predicted values ​​of the height of N liquid layer interfaces.

[0086] (2) Statistical distribution quantification: Statistical calculations are performed on the above N inference results, and the arithmetic mean of the N height prediction values ​​is taken as the visual measurement height; the sample variance of the N prediction values ​​is taken as the continuous quantification index of the visual measurement confidence. The smaller the variance, the more consistent the multiple inference results (i.e., the more certain the network's judgment on the image frame), and the higher the confidence; the larger the variance, the more uncertain the network's judgment, and the lower the confidence.

[0087] (3) Continuous nature: Variance is a continuous value (rather than discrete high / medium / low levels). Its value changes continuously with the changes in image area, lighting conditions and interface clarity. It can reflect the fine-grained confidence difference of visual recognition under different working conditions, thereby driving the continuous (rather than step-like) adjustment of fusion weights in subsequent fusion steps.

[0088] In this method, in addition to Monte Carlo Dropout, neural network uncertainty estimation can also be replaced by deep ensemble or Bayesian neural networks. Their core logic is the same: they all quantify uncertainty by statistical variance through multiple randomized inferences on the same input and output a continuous confidence index.

[0089] S103, Vision-Theory Multimodal Fusion Theoretical predictions and visual measurements are fused in a multimodal manner. The fusion weights are dynamically adjusted based on a continuous quantitative index of visual measurement confidence, automatically reducing the fusion weights of visual observations when visual confidence is low. For example... Figure 3 As shown, in this embodiment, multimodal fusion is implemented using extended Kalman filtering (which can also be replaced by unscented Kalman filtering or particle filtering). This step specifically includes: (1) Extended Kalman fusion: The system state vector is defined as being composed of the height of each liquid layer interface. The state prediction and covariance prediction are performed using the bottle-shaped geometric model in step S101 as the state transition function and the volume control quantity of liquid addition / transfer as the control input. The process noise covariance Q is introduced in the covariance prediction. The visual measurement value in step S102 is used as the observation value, and the Kalman gain is calculated using the bottle-shaped geometric model as the observation function. The measurement noise covariance matrix R is directly constructed frame by frame from the visual confidence variance output in step S102. The fused state estimate and covariance matrix are obtained through the prediction-update loop.

[0090] (2) Conditional Weighted Alternative: As a simplified approach, a conditional weighted strategy can also be adopted. Let the visual measurement value be z, and the theoretical prediction value be z. The variance of visual confidence is The theoretical prediction variance is The fusion result will be... for:

[0091] Among them, weight and The calculation formula is:

[0092]

[0093] When visual confidence is low ( When (large), Automatic reduction of theoretically predicted weights Increase the size to achieve adaptive fusion.

[0094] (3) Supplement to the numerical stability of interface differentiation fusion: In liquid-liquid stratification scenarios, the visual detection confidence of the upper interface (liquid-gas interface) and the lower interface (liquid-liquid interface) usually differs by orders of magnitude. The upper interface has a clear outline and high confidence, while the lower interface has significantly lower confidence due to factors such as small refractive index difference, meniscus effect, and emulsion layer interference. When the measurement noise variance of the upper and lower interfaces differs by several orders of magnitude, the measurement noise covariance matrix R of the standard EKF becomes significantly lower. k If the condition number is too large, the calculation of the Kalman gain may become numerically unstable, leading to non-physical oscillations in the state update. Therefore, this embodiment introduces a structured processing strategy within the EKF framework: when the confidence level of visual recognition of the lower interface is lower than a preset threshold, the lower interface component in the state vector and observation vector is temporarily decoupled from the EKF update step. The upper interface is still estimated through the standard EKF prediction-update loop, while the lower interface is directly calculated downwards using the fusion result of the upper interface and the theoretical liquid layer thickness calculated in step S101. This strategy essentially utilizes the physical constraints of the liquid layer system (the sum of the thicknesses of each liquid layer is not dependent on vision but determined by volume-geometric relationships) to decompose the numerically ill-conditioned problem of high-dimensional EKF into low-dimensional stable subproblems, avoiding the numerical risks of matrix inversion while ensuring accuracy.

[0095] S104, Absorbability Determination and Insertion Planning Based on the fusion results, absorbability is determined and insertion depth is calculated, such as... Figure 3 As shown. Specifically includes: (1) Determination of liquid layer thickness: The thickness of the target liquid layer is the difference in height between two adjacent interfaces. If this thickness is less than the preset minimum absorbable thickness, the liquid layer will be determined. d min If the sample size is 0.5mm, it is determined to be unabsorbable and a warning message is given.

[0096] (2) Liquid surface stability determination: The liquid surface fluctuation amplitude is calculated by the fusion results of multiple consecutive frames. If the fluctuation amplitude exceeds the preset threshold, it indicates that the liquid surface is unstable and the liquid transfer operation should be performed only after the liquid surface stabilizes. The liquid surface fluctuation amplitude is obtained by calculating the variance or range of the fusion results of multiple consecutive frames.

[0097] (3) Insertion depth planning: Calculate the optimal insertion depth of the pipette based on the location of the target liquid layer. To avoid drawing in the lower liquid layer, the insertion position should be in the upper middle part of the target liquid layer, with a certain safe distance from the interface. (e.g., 1-2mm). Optimal insertion depth calculation formula:

[0098] in, The height of the liquid column being drawn.

[0099] S105, Automatic pipetting execution The automatic pipetting mechanism is driven to complete the liquid transfer. Specifically, a PLC or motion controller drives the pipetting mechanism to insert the pipette into the liquid at a planned depth, and then aspirates the liquid using a syringe pump or peristaltic pump. The pipetting speed is optimized based on the liquid viscosity and the pipette diameter.

[0100] It should be understood that, although the above Figure 1 The steps are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated in this document, there is no strict order in which these steps are executed; they can be performed in other orders. Furthermore, the above... Figure 1 At least some of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0101] In one embodiment, an automated layered pipetting system for joint determination of visual and theoretical volume is also provided, comprising: The image acquisition module, including an industrial camera and a light source system, is used to acquire side or front images of a liquid container.

[0102] Specifically, industrial cameras can select lenses with different resolutions and focal lengths according to the size of the container and the accuracy requirements; the light source system can select backlight, front light or side light methods according to the characteristics of the liquid.

[0103] The theoretical prediction module is used to construct a software virtual sensor from the volume data of liquid added and / or liquid transferred during the chemical experiment, as well as the geometric model parameters of the container. The software virtual sensor is used to determine the theoretical prediction results. The software virtual sensor is used to predict the height of the liquid layer interface in the container through theoretical calculation based on the principle of volume conservation and container geometric constraints, and to calculate the process noise covariance based on the volume accuracy parameters and liquid evaporation parameters.

[0104] The visual recognition module is used to perform pixel-level segmentation of the liquid layer interface in container images acquired by industrial cameras using a deep learning instance segmentation network, identify the pixel position of the liquid layer interface, and perform multiple forward inferences on the visual measurement results using a neural network uncertainty estimation method. Based on the statistical distribution of the multiple inference results, it calculates continuous quantitative indicators of visual measurement height and visual measurement confidence.

[0105] Specifically, the visual recognition module incorporates a pre-trained segmentation network model, supporting online learning and model updates. Uncertainty estimation methods include Monte Carlo Dropout, deep ensembles, and Bayesian neural networks, which can be flexibly configured according to accuracy and computational cost requirements.

[0106] The multimodal fusion module is used to perform multimodal fusion of theoretical prediction results and visual measurement results output by the software virtual sensor using a confidence-driven adaptive fusion strategy to obtain the fused liquid layer interface height. During the fusion process, when the confidence of the visual measurement is lower than a first preset threshold, the weight of the visual observation is dynamically adjusted. When the confidence of the visual measurement of the lower interface is insufficient, the visual measurement value of the upper interface with high confidence is used as the benchmark, and the lower interface height is estimated by using the theoretically predicted liquid layer thickness.

[0107] Specifically, the multimodal fusion module supports differentiated fusion strategies for interfaces with low confidence. It utilizes the physical constraints of the liquid layer system to decompose high-dimensional numerical ill-conditioned problems into low-dimensional stable subproblems and supports switching between multiple fusion algorithms (extended Kalman filter, unscented Kalman filter, particle filter, conditional weighted fusion). The optimal fusion strategy can be selected according to the application scenario.

[0108] The pipetting control module is used to determine the absorbability of the target liquid layer based on the height of the interface of the fused liquid layer, calculate the insertion depth of the pipetting needle, and drive the pipetting actuator to complete the pipetting. The pipetting actuator includes a three-dimensional motion platform, a pipetting needle, and a syringe pump or peristaltic pump, which is used to perform specific pipetting operations.

[0109] Specifically, the pipetting control module supports multiple pipetting modes, including quantitative pipetting, fixed-depth pipetting, and complete aspiration.

[0110] The pipetting actuator includes a three-dimensional motion platform, a pipetting needle, and a syringe pump or peristaltic pump, used to perform specific pipetting operations. The motion platform is driven by a servo motor, achieving a positioning accuracy of ±0.01mm; the syringe pump uses a high-precision ceramic plunger pump, achieving a pipetting accuracy of ±0.1%.

[0111] In one embodiment, the theoretical prediction module further includes the following process for calculating theoretical prediction results using software virtual sensors: constructing a three-level volume calculation model with progressive accuracy; wherein the three-level volume calculation model includes: a first level of ideal immiscible model, a second level of miscible equilibrium material balance model, and a third level of empirical distribution coefficient method; the ideal immiscible model is used to directly calculate the liquid layer thickness based on the added volume of each phase; the miscible equilibrium material balance model is used to solve for the actual volume of the two phases by combining solubility data and the mass conservation of each phase; the empirical distribution coefficient method is used to determine the volume distribution coefficient based on historical calibration data, and subsequent operations directly call the volume distribution coefficient for volume calculation; automatically selecting the appropriate calculation level according to the completeness of the known information of the system, and calculating the theoretical prediction results; specifically including: when the interoperability of the system is lower than a preset value, the first level is used to calculate the theoretical prediction results; when the system has clear miscibility data, the second level is used to calculate the theoretical prediction results; when the system is a complex reaction liquid or extraction system, the third level is used to calculate the theoretical prediction results.

[0112] In one embodiment, the software virtual sensor in the theoretical prediction module also includes a dynamic loss compensation mechanism, which adopts a closed-loop strategy combining a feedforward evaporation model and visual feedback. The feedforward evaporation model estimates the amount of evaporation based on liquid volatility parameters and exposure time. The visual feedback detects the deviation between the actual liquid level and the theoretical prediction, and corrects the evaporation rate parameter or the total volume directly based on the deviation in real time.

[0113] In one embodiment, the theoretical prediction module also includes a layered prediction mechanism: the system has a built-in experimental procedure template and a miscibility knowledge base. The experimental procedure template defines the operation type of each step. When the step is marked as extraction or layering, the theoretical liquid layer calculation module is automatically activated. The miscibility knowledge base stores data on the miscibility relationships between common solvent pairs. Based on the currently added reagent combination, it automatically determines whether the layering conditions are met, forming a judgment mechanism that combines theoretical prediction with subsequent visual detection.

[0114] Specifically, the theoretical prediction module has a built-in knowledge base of miscibility properties and experimental procedure templates, supporting hierarchical prediction functions; it has a built-in three-level volume calculation model library with progressive accuracy, which can automatically select an ideal immiscible model, a miscible equilibrium material balance model, or an empirical distribution coefficient method to calculate the two-phase volume based on known information about the system; it has a built-in library of commonly used solvent evaporation rate parameters, supporting dynamic volume loss compensation by combining feedforward evaporation models with visual feedback; and it also supports the configuration of geometric model parameters for various bottle types and the dynamic adjustment of process noise parameters.

[0115] In one embodiment, the neural network uncertainty estimation method in the visual recognition module employs Monte Carlo Dropout, deep ensemble, or Bayesian neural network.

[0116] In one embodiment, the multimodal fusion module employs a state estimation algorithm for multimodal fusion; the state estimation algorithm includes extended Kalman filtering, unscented Kalman filtering, or particle filtering; the dynamic adjustment of the weights of visual observations is achieved by constructing the continuous quantization index of visual measurement confidence as the measurement noise covariance matrix in the state estimation algorithm in real time.

[0117] In one embodiment, the multimodal fusion module employs a conditional weighting strategy as an alternative, and the fusion result is a weighted sum of theoretical predictions and visual measurements. The weights are automatically calculated based on the variance of the theoretical predictions and the variance of the visual measurement confidence level. The weight calculation formula is as follows:

[0118]

[0119] in, Variance of visual confidence To theoretically predict variance, , These are the weights of the theoretical prediction and the measured data, respectively.

[0120] In one embodiment, the aspirability determination in the pipetting control module includes liquid layer thickness determination, liquid surface stability determination, and pipetting needle insertion safety determination. Specifically, the liquid layer thickness determination uses the height difference between two adjacent interfaces as the target liquid layer thickness. If the target liquid layer thickness is less than a preset minimum aspirable thickness, it is determined to be unabsorbable, and a warning is issued. The liquid surface stability determination calculates the liquid surface fluctuation amplitude using the fusion results of multiple consecutive frames. If the fluctuation amplitude exceeds a preset threshold, the liquid surface is determined to be unstable, and pipetting is only performed after the liquid surface stabilizes. The liquid surface fluctuation amplitude is obtained by calculating the variance or range of the fusion results of multiple consecutive frames. The insertion depth planning determines the optimal insertion depth of the pipetting needle based on the position of the target liquid layer, a preset safety distance, and the height of the aspirated liquid column.

[0121] in, To achieve the optimal insertion depth, The location of the target liquid layer. To determine the height of the liquid column being drawn, To set a safe distance.

[0122] In one embodiment, the device further includes: establishing a liquid layer history database to record the volume of liquid added / transferred, the height of the fused liquid layer interface, and the visual confidence level in each pipetting operation, for periodically optimizing the bottle geometry parameters in the theoretical prediction model and the model weights of the visual recognition network.

[0123] It is understood that for a detailed explanation of the automated layered pipetting device that uses both visual and theoretical volume determination, please refer to the corresponding explanations of the embodiments of the automated layered pipetting method using both visual and theoretical volume determination above, and will not be repeated here. Each module in the above-mentioned automated layered pipetting device that uses both visual and theoretical volume determination can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in hardware or independently of a device with data processing capabilities, or it can be stored in software in the memory of the aforementioned device, so that the processor can call and execute the operations corresponding to each module. The aforementioned device can be, but is not limited to, various types of data processing computer devices already existing in the art.

[0124] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and all such modifications and improvements fall within the scope of protection of this application.

Claims

1. An automated layered pipetting method based on a combination of visual and theoretical volume determination, characterized in that, Including the following steps: Step 1: Construct a software virtual sensor from the volume data of liquid added and / or liquid transferred during the chemical experiment, and the geometric model parameters of the container. Use the software virtual sensor to determine the theoretical prediction results. The software virtual sensor is used to predict the height of the liquid layer interface in the container through theoretical calculation based on the principle of volume conservation and container geometric constraints. And calculate the process noise covariance based on the volume accuracy and liquid evaporation parameters. The process noise covariance is used to characterize the degree of uncertainty of the theoretical prediction results, and provides a confidence basis for the weighted fusion of subsequent theoretical predictions and visual observations. Step 2: Use a deep learning instance segmentation network to perform pixel-level segmentation of the liquid layer interface in the acquired liquid container image, identify the pixel position of the liquid layer interface, and perform multiple forward inferences on the visual measurement results through a neural network uncertainty estimation method. Calculate the continuous quantitative indicators of visual measurement height and visual measurement confidence based on the statistical distribution of the multiple inference results. Step 3: A confidence-driven adaptive fusion strategy is used to perform multimodal fusion of theoretical prediction results and visual measurement results to obtain the fused liquid layer interface height. During the fusion process, when the confidence of visual measurement is lower than the first preset threshold, the weight of visual observation is dynamically adjusted. When the confidence of visual measurement of the lower interface is insufficient, the visual measurement value of the upper interface with high confidence is used as the benchmark, and the lower interface height is estimated by using the theoretically predicted liquid layer thickness. Step 4: Determine the absorbability of the target liquid layer based on the height of the interface after fusion, calculate the insertion depth of the pipette, and drive the pipetting actuator to complete the pipetting.

2. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, The process of calculating theoretical prediction results using the aforementioned software virtual sensor includes: A three-level volume calculation model with progressive accuracy is constructed. The three-level model includes: a first level of ideal immiscible model, a second level of miscible equilibrium material balance model, and a third level of empirical distribution coefficient method. The ideal immiscible model is used to directly calculate the liquid layer thickness based on the added volume of each phase. The miscible equilibrium material balance model is used to solve for the actual volume of the two phases by simultaneously applying solubility data and the mass conservation of each phase. The empirical distribution coefficient method is used to determine the volume distribution coefficient based on historical calibration data, and subsequent operations directly call upon the volume distribution coefficient for volume calculation. The system automatically selects the appropriate calculation level based on the completeness of the known information about the system, and calculates the theoretical prediction results. Specifically, when the interoperability of the system is lower than the preset value, the first level of calculation is used to calculate the theoretical prediction results; when the system has clear miscibility data, the second level of calculation is used to calculate the theoretical prediction results; and when the system is a complex reaction solution or extraction system, the third level of calculation is used to calculate the theoretical prediction results.

3. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, The software virtual sensor also includes a dynamic loss compensation mechanism, which adopts a closed-loop strategy combining a feedforward evaporation model and visual feedback. The feedforward evaporation model estimates the amount of evaporation based on liquid volatility parameters and exposure time. The visual feedback detects the deviation between the actual liquid level and the theoretical prediction, and corrects the evaporation rate parameter or the total volume directly based on the deviation in real time.

4. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, Step 1 also includes a layering prediction mechanism: the system has a built-in experimental process template and a miscibility knowledge base. The experimental process template defines the operation type of each step. When the step is marked as extraction or layering, the theoretical liquid layer calculation module is automatically activated. The miscibility knowledge base stores the miscibility relationship data between common solvent pairs. Based on the currently added reagent combination, it automatically judges whether the layering conditions are met, forming a judgment mechanism that combines theoretical prediction with subsequent visual detection.

5. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, The neural network uncertainty estimation method described in step 2 employs Monte Carlo Dropout, deep ensemble, or Bayesian neural networks.

6. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, The multimodal fusion described in step 3 is implemented using a state estimation algorithm; the state estimation algorithm includes extended Kalman filtering, unscented Kalman filtering, or particle filtering. The dynamic adjustment of the weights of visual observations is achieved by constructing the continuous quantization index of the confidence level of the visual measurement into the measurement noise covariance matrix in the state estimation algorithm in real time.

7. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, Step 3 also includes, Multimodal fusion employs a conditional weighting strategy as an alternative. The fusion result is a weighted sum of theoretical predictions and visual measurements. The weights are automatically calculated based on the variance of theoretical predictions and the variance of visual measurement confidence levels. The weight calculation formula is as follows: in, Variance of visual confidence To theoretically predict variance, , These are the weights of the theoretical prediction and the measured data, respectively.

8. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, Step 4, the absorbability determination, includes the determination of liquid layer thickness, liquid surface stability, and pipette insertion safety. Among them, the liquid layer thickness determination is as follows: the difference in height between two adjacent interfaces is taken as the thickness of the target liquid layer. When the thickness of the target liquid layer is less than the preset minimum absorbable thickness, it is determined to be unabsorbable and a warning is given. Liquid surface stability determination: The fluctuation amplitude of the liquid surface is calculated by the fusion results of multiple consecutive frames. If the fluctuation amplitude exceeds the preset threshold, the liquid surface is determined to be unstable, and the liquid removal operation is not performed until the liquid surface is stable. The fluctuation amplitude of the liquid surface is obtained by calculating the variance or range of the fusion results of multiple consecutive frames. Insertion depth planning: Based on the location of the target liquid layer, the preset safety distance, and the height of the aspirated liquid column, the optimal insertion depth of the pipette is determined as follows: in, To achieve the optimal insertion depth, The location of the target liquid layer. To determine the height of the liquid column being drawn, To set a safe distance.

9. The automated layered pipetting method based on combined visual and theoretical volume determination according to claim 1, characterized in that, The method further includes: establishing a liquid layer history database to record the volume of liquid added / transferred, the height of the fused liquid layer interface, and the visual confidence level in each pipetting operation, which is used to periodically optimize the bottle geometry parameters in the theoretical prediction model and the model weights of the visual recognition network.

10. An automated layered pipetting system based on a combination of visual and theoretical volume determination, characterized in that, include: The image acquisition module includes an industrial camera and a light source system, used to acquire side or front images of liquid containers; The theoretical prediction module is used to construct a software virtual sensor from the volume data of liquid added and / or liquid transferred during the chemical experiment and the geometric model parameters of the container bottle, and to determine the theoretical prediction results using the software virtual sensor; the software virtual sensor is used to predict the height of the liquid layer interface in the container through theoretical calculation based on the principle of volume conservation and container geometric constraints, and to calculate the process noise covariance based on the volume accuracy parameters and liquid evaporation parameters. The visual recognition module is used to perform pixel-level segmentation of the liquid layer interface in container images acquired by industrial cameras using a deep learning instance segmentation network, identify the pixel position of the liquid layer interface, perform multiple forward inferences on the visual measurement results using a neural network uncertainty estimation method, and calculate continuous quantitative indicators of visual measurement height and visual measurement confidence based on the statistical distribution of the multiple inference results. The multimodal fusion module is used to perform multimodal fusion of the theoretical prediction results and visual measurement results output by the software virtual sensor using a confidence-driven adaptive fusion strategy to obtain the fused liquid layer interface height. During the fusion process, when the confidence of the visual measurement is lower than a first preset threshold, the weight of the visual observation is dynamically adjusted. When the confidence of the visual measurement of the lower interface is insufficient, the visual measurement value of the upper interface with high confidence is used as the benchmark, and the lower interface height is estimated by using the theoretically predicted liquid layer thickness. The pipetting control module is used to determine the absorbability of the target liquid layer based on the height of the interface of the fused liquid layer, calculate the insertion depth of the pipetting needle, and drive the pipetting actuator to complete the pipetting. The pipetting actuator includes a three-dimensional motion platform, a pipetting needle, and a syringe pump or peristaltic pump, which is used to perform specific pipetting operations.

Images