Double-arm visual feedback precise pipetting system and method

Abstract

The embodiment of the invention provides a double-arm visual feedback precise pipetting system and method, and the system comprises a robot body which comprises a controller; the first operating mechanism comprises a first mechanical arm and a pipette, the first mechanical arm is arranged on one side of the robot body, and the pipette is arranged at the tail end of the first mechanical arm; the second operating mechanism comprises a second mechanical arm and a second visual sensor, the second mechanical arm is arranged on the other side of the robot body, and the second visual sensor is arranged on the second mechanical arm; the controller is configured to determine the observation position of the second visual sensor according to the liquid taking position of the first mechanical arm and the target liquid taking amount, and control the second mechanical arm to move so as to move the second visual sensor to the observation position to collect the actual liquid level height in the pipette; the actual liquid taking amount of the pipette is adjusted according to the actual liquid level height, liquid taking is completed, and the first mechanical arm is controlled to move to execute liquid injection operation.

Description

Technical Field

[0001] This application belongs to the field of biomedical technology, and in particular relates to a dual-arm embodied visual feedback precision pipetting system and method. Background Technology

[0002] In experimental settings such as life sciences, drug development, molecular diagnostics, and high-throughput screening, accurate pipetting is a fundamental and crucial operation. With the development of laboratory automation, robots replacing manual pipetting tasks has become a trend. Currently, most mainstream automated pipetting systems are based on motorized pipettes, using methods such as gravimetric calibration to pre-calibrate the liquid, and then controlling the aspiration or dispensing volume of the pipette based on the calibrated pipette control parameters.

[0003] However, this robotic pipetting method requires pre-calibration for different types of liquids. During application, the robot needs to know the type of liquid to be pipetted and then control the pipetting according to the pre-calibrated control parameters, making the whole pipetting process cumbersome. Summary of the Invention

[0004] This application provides a dual-arm embodied visual feedback precision pipetting system and method, which can solve the problems of gravimetric pipetting operations being unable to adapt to pipetting unknown liquids and the cumbersome operation process.

[0005] In a first aspect, embodiments of this application provide a dual-arm, bio-feedback, precision pipetting system, comprising: The robot body includes a controller; The first operating mechanism includes a first robotic arm and a pipette. The first robotic arm is located on one side of the robot body, and the pipette is located at the end of the first robotic arm. The pipette's sampling tube has a visible area. The second operating mechanism includes a second robotic arm and a second vision sensor. The second robotic arm is located on the other side of the robot body, and the second vision sensor is located on the second robotic arm. The controller is configured to: determine the observation position of the second vision sensor based on the liquid collection position of the first robotic arm and the target liquid collection volume, and control the second robotic arm to move the second vision sensor to the observation position, and collect the actual liquid level height in the pipette through the visible area; The controller is further configured to: adjust the actual liquid volume taken by the pipette according to the actual liquid level, such that the difference between the actual liquid volume and the target liquid volume is less than or equal to a preset difference, thereby completing the liquid collection; and control the first robotic arm to move the pipette to the target container to perform the liquid injection operation. Secondly, embodiments of this application provide a dual-arm hymenoplasty visual feedback precision pipetting method, applied to the dual-arm hymenoplasty visual feedback precision pipetting system described in the first aspect above, comprising: The observation position of the second vision sensor is determined based on the target liquid volume and the liquid collection position of the first robotic arm. The first robotic arm is controlled to move so that the pipette can pick up liquid at the liquid picking position. Based on the actual liquid level height of the pipette collected by the second vision sensor at the observation position, the actual liquid volume of the pipette is adjusted so that the difference between the actual liquid volume of the pipette and the target liquid volume is less than or equal to a preset difference, and the liquid picking is completed. Control the first robotic arm to move the pipette to the target container and perform the liquid injection operation.

[0006] Thirdly, embodiments of this application provide a dual-arm, body-worn visual feedback precision pipetting device, comprising: The determining unit is used to determine the observation position of the second vision sensor based on the target liquid volume and the liquid collection position of the first robotic arm; The liquid collection unit is used to control the movement of the first robotic arm so that the pipette can collect liquid at the liquid collection position, and adjust the actual liquid collection volume of the pipette according to the actual liquid level height of the pipette collected by the second vision sensor at the observation position, so that the difference between the actual liquid collection volume and the target liquid collection volume is less than or equal to a preset difference, thereby completing the liquid collection. The liquid injection unit is used to control the movement of the first robotic arm to move the pipette to the target container and perform the liquid injection operation.

[0007] Fourthly, embodiments of this application provide a robot, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the second aspect above.

[0008] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the second aspect above.

[0009] In a sixth aspect, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the method described in the second aspect above.

[0010] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0011] The beneficial effects of this application embodiment compared with the prior art are as follows: The dual-arm embodied visual feedback precision pipetting system includes a robot body, a first operating mechanism, and a second operating mechanism. The first operating mechanism includes a first robotic arm and a pipette. The first robotic arm is located on one side of the robot body, and the pipette is located at the end of the first robotic arm. The first robotic arm controls the pipette to perform operations such as liquid collection and injection to complete pipetting. The second operating mechanism includes a second robotic arm and a second vision sensor. The second robotic arm is located on the other side of the robot body, and the second vision sensor is located on the second robotic arm. The robot body includes a controller. The controller is configured to: determine the observation position of the second vision sensor based on the liquid-taking position of the first robotic arm and the target liquid-taking volume, and then control the second robotic arm to move the second vision sensor to the observation position. The second vision sensor collects the actual liquid level height inside the pipette through the visible area. The controller is also configured to: adjust the actual liquid-taking volume of the pipette based on the actual liquid level height inside the pipette, so that the difference between the actual liquid-taking volume and the target liquid-taking volume is less than or equal to a preset difference, thus completing the liquid taking, and control the first robotic arm to move the pipette to the target container to perform the dispensing operation, completing one pipetting operation. In this embodiment, the second vision sensor is used to provide feedback on the actual liquid level height inside the pipette. The controller can adjust the liquid-taking volume inside the pipette according to the actual liquid level height, that is, based on liquid volume measurement and pipetting. Compared with the gravimetric method, this embodiment does not require pre-calibration for different types of liquids, and it is not necessary to know the type of liquid to be taken during the liquid taking process, simplifying the pipetting process. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a dual-arm hymenoplasty visual feedback precision pipetting system provided in one embodiment of this application; Figure 2 This is a flowchart illustrating a dual-arm embodied visual feedback precision pipetting method according to an embodiment of this application; Figure 3 This is a flowchart illustrating a dual-arm embodied visual feedback precision pipetting method according to another embodiment of this application; Figure 4 This is a schematic diagram of liquid collection provided in one embodiment of this application; Figure 5 This is a schematic diagram of an image of a liquid collection tube acquired by a second vision sensor according to an embodiment of this application; Figure 6 This is a schematic diagram illustrating the calibration relationship between the liquid level and the liquid collection volume of the liquid collection tube provided in one embodiment of this application; Figure 7 This is a flowchart illustrating a dual-arm embodied visual feedback precision pipetting method according to another embodiment of this application; Figure 8 This is a schematic diagram of liquid collection provided in another embodiment of this application; Figure 9 This is a schematic diagram of liquid collection provided in another embodiment of this application; Figure 10 This is a schematic diagram of the structure of the dual-arm embodied visual feedback precision pipetting device provided in the embodiments of this application.

[0014] Figure Labels 10-Robot body; 20-First operating mechanism; 201-First robotic arm; 202-First vision sensor; 203-Pipette; 2031-Liquid collection tube; 30-Second operating mechanism; 301-Second robotic arm; 302-Second vision sensor; 303-Two-finger gripper. Detailed Implementation

[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0016] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0017] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0018] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0019] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0021] This application addresses the problems of cumbersome robotic pipetting processes in related technologies by proposing a dual-arm embodied visual feedback precision pipetting system and a pipetting method applied to the system. Through collaborative closed-loop control of "perception-action", the system is endowed with the ability of "self-monitoring" and real-time correction, thereby significantly improving the pipetting accuracy and adaptability to liquids of different properties.

[0022] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] Figure 1 The diagram shown is a schematic representation of a dual-arm hymenoplasty visual feedback precision pipetting system according to an embodiment of this application. This is provided as an example and not as a limitation. Figure 1 As shown, the overall architecture of the dual-arm embodied visual feedback precision pipetting system is a dual-arm humanoid robot, which includes: a robot body 10, a first operating mechanism 20, and a second operating structure 30. The robot body 10 includes a controller, which is communicatively connected to the first operating structure 20 and the second operating mechanism 30. The controller is configured to control the first operating mechanism 20 and the second operating mechanism 30 to coordinate the pipetting process.

[0024] For example, the first operating mechanism 20 includes a first robotic arm 201 and a pipette 203. The first robotic arm 201 is located on one side of the robot body 10, and the pipette 203 is located at the end of the first robotic arm 201. The first robotic arm 201 is used to move the pipette 203 to various positions for liquid collection and injection. The first robotic arm 201 includes multiple degrees of freedom (more than 6 degrees of freedom) to realize spatial movement, rotation, and other operations. Different accessories can be installed at the end of the first robotic arm 201 to achieve specific functions. For example, in this embodiment, the pipette 203 is installed at the end of the first robotic arm 201 and can be controlled by commands to aspirate, dispense, and inject liquid.

[0025] By way of example and not limitation, pipette 203 is an electrically powered pipette that uses pneumatic pressure for liquid aspiration and calibrates the pipetting volume using purified water at standard atmospheric pressure. Pipette 203 includes a body, a tip, and a collection tube 2031, wherein the collection tube 2031 is mounted on the tip, which is detachably connected to the body. A collection tube 2031 of appropriate capacity can be selected for liquid aspiration as needed. For example, a 1000-microliter collection tube 2031 is mounted at the end of the first robotic arm 201 for liquid aspiration.

[0026] In this embodiment, the liquid collection tube 2031 has a visible area, through which the liquid level inside the liquid collection tube 2031 can be observed. Optionally, the liquid collection tube 2031 of the pipette 203 is a transparent tube, that is, the entire liquid collection tube 2031 is a transparent tube and is entirely a visible area.

[0027] As an example, the first operating mechanism 20 also includes a first vision sensor 202, which is mounted on the first robotic arm 201. The first vision sensor 202 is configured to locate the liquid dispensing position. That is, the first vision sensor 202 detects the liquid dispensing bottle, determines the position of the dispensing bottle, and thus locates the liquid dispensing position.

[0028] For example, the second operating mechanism 30 includes a second robotic arm 301 and a second vision sensor 302. The second robotic arm 301 is located on the other side of the robot body 30, and the second vision sensor 302 is located on the second robotic arm 301. The second vision sensor 302 is configured to acquire the actual liquid level height inside the pipette 203 through the visible area.

[0029] Optionally, the second vision sensor 302 is located at the end of the second robotic arm 301.

[0030] As an example, the end of the second robotic arm 301 is also provided with a two-finger gripper 303 for grasping and moving the liquid bottle. Optionally, a second vision sensor 302 is located at the connection between the two-finger gripper 303 and the second robotic arm 301, and can collect the grasping operation of the two-finger gripper 303.

[0031] In this embodiment, the controller is configured to: determine the observation position of the second vision sensor 202 based on the liquid collection position of the first robotic arm 201 and the target liquid collection volume, and control the second robotic arm 201 to move the second vision sensor 202 to the observation position, collect the actual liquid level height in the pipette 203 through the visible area; and adjust the actual liquid collection volume of the pipette 203 according to the actual liquid level height, so that the difference between the actual liquid collection volume and the target liquid collection volume is less than or equal to a preset difference, complete the liquid collection, and control the first robotic arm 201 to move the pipette 203 to the target container to perform the liquid injection operation.

[0032] Understandably, both the second vision sensor 302 and the liquid dispensing gun 203 are connected to the controller.

[0033] In this embodiment, the controller controls the first robotic arm 201 and the second robotic arm 301 to perform a coordinated pipetting operation. For example, the controller controls the first robotic arm 201 to move, moving the pipetting nozzle 203 to the pipetting position for liquid collection. Simultaneously, the controller controls the second robotic arm 301 to move, moving the second vision sensor 302 to the corresponding observation position to collect the actual liquid level in the pipetting nozzle 203. The controller also determines the actual liquid volume collected by the pipetting nozzle based on the actual liquid level fed back by the second vision sensor 302, and then controls the pipetting nozzle 203 to collect liquid based on the difference between the actual liquid volume and the target liquid volume until the difference between the actual liquid volume and the target liquid volume is less than or equal to a preset difference, completing the liquid collection. Then, the controller controls the first robotic arm 201 to move the pipetting nozzle 203 to the target container to perform the liquid injection operation. Based on the actual liquid level height control of the liquid-taking gun 203, which is controlled by the dual-arm collaborative control and the real-time feedback of the second vision sensor 302, the liquid-taking gun 203 takes liquid, and completes the collaborative closed-loop control of "perception-action". It highly simulates the liquid-taking operation of the experimenter, realizes the ability of "self-monitoring" and real-time correction, and improves the accuracy of liquid transfer and the adaptability to liquids with different properties.

[0034] To avoid visual deviations in the actual liquid level height captured by the second vision sensor 302 (such as visual deviations when the shooting angle is a lower or higher angle than a preset angle), in one possible implementation, the observation position of the second vision sensor 302 varies depending on the target liquid volume. For example, at least two observation positions are set according to the liquid volume range, and the observation position of the second vision sensor 302 is determined based on the liquid volume range in which the target liquid volume is located. It is understood that when the second vision sensor 302 captures the liquid level height at a near-parallel shooting angle from the corresponding observation position, the resulting image has high accuracy.

[0035] If the target liquid volume is very small, the liquid level in the dispensing tube will be inside the reagent bottle, making it difficult for the second vision sensor 302 to acquire the liquid level. Therefore, in some embodiments, the observation position of the second vision sensor 302 is configured to be determined based on the dispensing position of the first robotic arm 201 and the target liquid volume as follows: if the target liquid volume is greater than the first capacity, the observation position of the second vision sensor 302 is determined based on the target liquid volume and the dispensing position of the pipette 203.

[0036] It is understandable that when the target liquid volume is greater than the first capacity, the liquid level in the dispensing tube 2031 is higher than the mouth of the reagent bottle. That is, in this embodiment, if the target sample volume is greater than the first capacity, and the liquid level in the dispensing tube 2031 is outside the reagent bottle, the actual liquid level can be collected by the second visual sensor 302, thereby controlling the dispensing gun 203 to dispense liquid.

[0037] And / or, the observation position of the second vision sensor 302 is determined based on the liquid-taking position of the first robotic arm 201 and the target liquid-taking volume as follows: if the target liquid-taking volume is less than or equal to the first capacity, after the pipette 203 takes liquid, the first robotic arm 201 is controlled to move the pipette 203 to the observation position, and the observation position of the second vision sensor 302 is determined based on the observation position. That is, in some embodiments, when the target liquid-taking volume is less than the first capacity, the first robotic arm 201 is controlled to move the pipette 203 to the observation position, such as controlling the first robotic arm 201 to move such that the liquid-taking tube 2031 of the pipette 203 is at a second set height from the liquid-taking position, and then the second vision sensor 302 is controlled to collect the actual liquid level height in the liquid-taking tube 2031. Move the pipette 2031 to the observation position so that the second vision sensor 302 can directly collect the actual liquid level in the liquid collection tube 2031, thus solving the problem that the second vision sensor 302 has difficulty collecting the liquid level when the liquid level in the liquid collection tube 2031 is inside the reagent bottle.

[0038] By way of example and not limitation, the first visual sensor 202 and the second visual sensor 302 employ a camera with a field of view of 60° and a resolution of 2560×1440 or higher.

[0039] As an example and not a limitation, the first robotic arm 201 is located on the right side of the robot body 10, that is, the right arm of the robot body 10, and the second robotic arm 302 is located on the left side of the robot body 10, that is, the left arm of the robot body 10.

[0040] Figure 2 A schematic flowchart of the dual-arm hymenoplasty visual feedback precision pipetting method provided in this application is shown. This is an example and not a limitation; the method can be applied to the aforementioned dual-arm hymenoplasty visual feedback precision pipetting system, including: S201, determine the observation position of the second vision sensor based on the target liquid volume and the liquid collection position of the first robotic arm.

[0041] As an example, when a dual-arm embodied visual feedback precision pipetting system receives a liquid retrieval command, it determines the target liquid volume, target retrieval bottle, and target container based on the command. For instance, the retrieval command instructs the transfer of 'a' microliters of reagent from retrieval bottle N1 to reagent bottle N2, with the target liquid volume being 'a' microliters, the target retrieval bottle being N1, and the target container being N2.

[0042] After determining the target liquid volume, the dual-arm embodied visual feedback precision liquid transfer system identifies the position of the reagent bottle based on the first visual sensor on the first robotic arm, and determines the liquid transfer position of the first robotic arm based on the position of the reagent bottle.

[0043] After determining the liquid sampling location, the observation position of the second vision sensor is determined based on the sampling location and the target liquid volume. For example, a correspondence between the sampling location and the observation position can be pre-set, and the observation position can be calculated based on the correspondence after the sampling location is determined.

[0044] In some embodiments, the sampling location corresponds to one observation location. In other embodiments, the sampling location may correspond to multiple observation locations. For example, different observation locations may correspond to different target sampling volumes at the same sampling location. Therefore, when determining multiple observation locations based on the sampling location, the final observation location is determined from the multiple observation locations by combining the target sampling volume.

[0045] In some optional embodiments, the first robotic arm is configured with a first initial position pose P0, a ready position pose P1, a liquid-taking position P2, an observed position P3, and a liquid-injection position PE. The first initial position pose P0 refers to the initial positioning position. For example, when the robot moves according to basic navigation control (assuming a navigation error of ±5cm) to the vicinity of the operating table, the first visual sensor of the first robotic arm can observe the positions of the target liquid-taking bottle, reagent bottle, liquid-taking tube, etc., on the table. The first initial position pose P0 is a pre-calibrated fixed pose. The ready position pose P1 is the position directly above the target liquid-taking bottle, without interference. The liquid-taking position P2 is the position where the liquid-taking tube is submerged below the liquid surface of the target liquid-taking bottle. The observed position P3 is the position where the liquid-taking tube of the liquid-taking gun is located at a preset height above the mouth of the target liquid-taking bottle (or the observed position is the liquid-taking position).

[0046] After the robot moves to the vicinity of the operating table, in the first initial pose P0, it controls the first vision sensor to acquire an image of the operating table and determine the location of the target liquid collection tube. This allows the first robotic arm to replace the liquid collection tube based on its location. Furthermore, based on the location of the target liquid collection bottle, a preparation pose P1 is determined, and then the liquid collection position P2 and the observed position P3 are determined according to the preparation pose P1. Finally, the injection position PE is determined based on the location of the target container.

[0047] S202, control the first robotic arm to move so that the pipette can pick up liquid at the liquid picking position, and adjust the actual liquid picking volume of the pipette according to the actual liquid level height of the pipette collected by the second vision sensor at the observation position, so that the difference between the actual liquid picking volume and the target liquid picking volume is less than or equal to the preset difference, and the liquid picking is completed.

[0048] In this embodiment, the controller controls the first and second robotic arms to work together to collect liquid. For example, the first robotic arm is controlled to move, moving the pipette to the liquid collection position. Simultaneously, based on the actual liquid level height of the pipette collected by the second vision sensor at the observation position, the actual liquid volume collected by the pipette is controlled. When the difference between the actual liquid volume collected and the target liquid volume is less than or equal to a preset difference, that is, when the reagent volume reaches the target liquid volume (within the allowable error range), the pipette is controlled to complete the liquid collection.

[0049] In some embodiments, the control method for the coordinated liquid collection by the first robotic arm and the second robotic arm includes: controlling the first robotic arm to move to the liquid collection position, controlling the second robotic arm to move to the observation position; determining the target liquid level height of the pipette based on the target liquid collection volume; controlling the pipette to collect liquid, and controlling the second visual sensor to collect the actual liquid level height of the pipette; and controlling the pipette to stop collecting liquid when the actual liquid level height reaches the target liquid level height, thus completing the liquid collection.

[0050] As an example and not a limitation, this embodiment performs liquid collection based on a method of simultaneous liquid collection and observation. After determining the observation position of the second vision sensor based on the target liquid volume and collection location, the first robotic arm is first controlled to move, moving the pipette to the collection position. Simultaneously, the second robotic arm is controlled to move, moving the second vision sensor to the observation position. The target liquid level height that the pipette tube needs to reach is determined based on the target liquid volume (the correspondence between liquid volume and liquid level height is pre-calibrated). Then, the pipette is controlled to collect liquid. During the collection process, the second vision sensor collects the actual liquid level height of the pipette in real time and feeds it back to the controller. The controller controls the pipette to continue collecting liquid based on the actual liquid level height until the actual liquid level reaches the target liquid level height, at which point the pipette stops collecting liquid, completing the collection process.

[0051] In this embodiment, the experimenter simulates the liquid collection process and controls the pipette based on the image information fed back by the second visual sensor, forming a perception-action linkage control.

[0052] In other embodiments, the control method for the coordinated liquid collection by the first robotic arm and the second robotic arm includes: after the pipette completes the initial liquid collection at the collection position, controlling the first robotic arm to move to the observation position, the observation position being determined according to the liquid collection position of the first robotic arm, and the observation position of the second vision sensor being determined according to the observation position; controlling the second robotic arm to move so that the second vision sensor moves to the observation position and collects the actual liquid level height of the pipette; adjusting the actual liquid collection volume of the pipette according to the actual liquid level height until the difference between the actual liquid collection volume and the target liquid collection volume is less than or equal to a preset difference, thus completing the liquid collection.

[0053] As an example and not a limitation, this embodiment performs liquid collection using a liquid collection-observation-replenishment method. For instance, the observed position of the pipette is set, with a second predetermined height from the liquid collection position. After determining the liquid collection position, the observed position of the pipette is determined based on the liquid collection position and the second predetermined height. The observation position of the second vision sensor is then determined based on the observed position (the observation position corresponds to the observed position). First, the pipette completes initial liquid collection at the liquid collection position. Then, the pipette is moved to the observed position, and the second robotic arm is moved to move the second vision sensor to the observation position. The second vision sensor collects the actual liquid level height in the collection tube and feeds it back to the controller. The controller adjusts the actual liquid volume collected by the pipette based on the actual liquid level height (for example, if the actual liquid volume is less than the target liquid volume, the pipette is refilled; if the actual liquid volume is greater than the target liquid volume, the pipette is ejected), until the difference between the actual liquid volume and the target liquid volume is less than or equal to a preset difference, thus completing the liquid collection.

[0054] In this embodiment, the pipette is moved to the observation position, which makes it easier for the second vision sensor to collect the actual liquid level in the liquid collection tube. Moreover, the second vision sensor collects the actual liquid level in a static state, which is more accurate and can avoid information delay that would reduce the accuracy of liquid collection.

[0055] S203, control the first robotic arm to move so that the liquid gun moves to the target container and performs the liquid injection operation.

[0056] After the first robotic arm completes the liquid retrieval, the first robotic arm is controlled to move to the target container, and the liquid in the retrieval tube is injected into the target container to complete the pipetting operation.

[0057] In this embodiment, mimicking the observation and control method of a lab technician's liquid-taking operation, a second robotic arm carrying a second vision sensor is used to switch to the optimal position in real time, observing the liquid level in the pipette held by the first robotic arm. The pipetting action is adjusted based on the actual liquid level feedback from the second vision sensor, forming a precise pipetting operation under closed-loop control. That is, by using the second vision sensor to report the actual liquid level in the pipette, the controller can adjust the volume of liquid taken from the pipette according to the actual liquid level. This is based on liquid volume measurement and pipetting. Compared to the gravimetric method, this embodiment does not require pre-calibration for different types of liquids, and it does not require knowledge of the type of liquid to be taken during the liquid-taking process, simplifying the pipetting process. It is suitable for scientific exploration and other applications requiring frequent configuration and switching of different reagents in automated liquid chromatography experiments.

[0058] Figure 3 A schematic flowchart of the dual-arm hymenoplasty visual feedback precision pipetting method provided in this application is shown. This is an example and not a limitation; the method can be applied to the aforementioned dual-arm hymenoplasty visual feedback precision pipetting system, including: 301, Obtain the liquid retrieval command, and determine the target liquid retrieval volume based on the liquid retrieval command.

[0059] 302. If the target liquid volume is greater than the first capacity, the observation position of the second vision sensor is determined based on the liquid collection position of the first robotic arm.

[0060] For example, for a sampling tube with a volume of d microliters, the first capacity is d1, where d1 is less than d. When the target sampling volume is greater than d1, the liquid level in the sampling tube is higher than the mouth of the sampling bottle, making observation easier; when the target sampling volume is less than d1, the liquid level in the sampling tube is lower than the mouth of the sampling bottle, making accurate observation of the actual liquid level impossible. Therefore, in this embodiment, when the target sampling volume is greater than d1 microliters, sampling can be performed using the sampling method of this embodiment (an online sampling method that combines sampling and observation). When the target sampling volume is less than d1 microliters, other methods are used, such as a pre-calibrated switch sampling method or a gravimetric method. As an example and not a limitation, d is 1000 microliters, and d1 is 200 microliters.

[0061] If the target liquid volume is greater than the first capacity, liquid collection and observation can be performed simultaneously. Therefore, the observation position of the second vision sensor is set to correspond to the liquid collection position of the first robotic arm, so that while the liquid gun is collecting liquid, the second vision sensor can collect the actual liquid level height change in the liquid gun at the observation position.

[0062] In some embodiments, if the target liquid volume is greater than the first capacity, the liquid collection position is transformed according to a preset dual-arm relative transformation matrix to obtain the observation position of the second visual sensor.

[0063] In other embodiments, to further improve the accuracy of the actual liquid level height captured by the second visual sensor, multiple observation positions are set according to different target liquid volumes, so that the second visual sensor captures the actual liquid level height in the liquid collection tube at the optimal position. These observation positions are the same in the X and Y axes, but differ in their vertical height. As an example, and not a limitation, it is assumed that there are two observation positions, a first position and a second position, with the second position located above the first position and differing from the first position by a first predetermined height.

[0064] For example, such as Figure 4 As shown, the total liquid volume of the sampling tube is 1000 microliters. If the target liquid volume is greater than the first capacity (e.g., 200 microliters), and the target liquid volume is less than or equal to the second capacity (e.g., the target liquid volume is between 200 and 600 microliters), then the first position C1 is determined based on the sampling position and the relative transformation matrix of the two arms, and the first position C1 is the observation position. If the target liquid volume is greater than the second capacity (e.g., the target liquid volume is between 600 and 1000 microliters), then the second position C2 is determined based on the sampling position, the relative transformation matrix of the two arms, and the first set height, and the second position C2 is the observation position.

[0065] It should be noted that the relative transformation matrix of the two arms is a pre-calibrated pose relationship of the two end effectors in the robot's base coordinate system. This calibration is performed to ensure high precision and robustness of the coordinated operation of the two arms. The calibration process of the relative transformation matrix of the two arms is described below: Beforehand, based on the manual calibration of the first robotic arm in the liquid-collecting posture, the optimal relative observation position of the second robotic arm is determined. Then, the relationship between the liquid-collecting position and the observation position of the first robotic arm is calculated to obtain the relative transformation matrix of the two arms.

[0066] This process is accomplished through guided calibration. The operator guides both arms to a validated optimal cooperative observation posture, controlling the first robotic arm to move the pipette to the retrieval position and controlling the second robotic arm to move the second vision sensor to face the retrieval tube for optimal imaging. At this point, the controller locks and records the first position of the end effector (pipette) of the first robotic arm in the robot's base coordinate system. The second pose of the end effector (second vision sensor) of the second robotic arm in the robot base coordinate system Based on the first and second poses, a constant relative transformation matrix between the two arms is calculated using the following kinematic relationships: ; The relative transformation matrix of the two arms As the ideal geometric constraint for dual-arm collaborative observation, and as the core parameter of the system, the corresponding observation position can be calculated based on the actual liquid collection position and the relative transformation matrix of the two arms during the liquid collection operation, providing a benchmark for all subsequent collaborative motion planning.

[0067] Understandably, the relative transformation matrix of the two arms is obtained through offline calibration. In practical applications, the observation position of the second vision sensor needs to be calculated based on the actual liquid retrieval location. The following details the process of determining the observation position of the second vision sensor: First, it should be noted that the system includes the robot base coordinate system B, the first robotic arm base coordinate system BR, the second robotic arm base coordinate system BL, the first robotic arm end effector coordinate system ER, the second robotic arm end effector coordinate system EL, the first vision sensor coordinate system CR, the second vision sensor coordinate system CL, and so on. Therefore, it is necessary to pre-calibrate the relationships between these coordinate systems in order to clarify the relationships between the data collected from different parts.

[0068] For example, hand-eye calibration is performed in advance. Hand-eye calibration refers to determining the relative pose relationship between the coordinate system of the first vision sensor and the coordinate system of the first robotic arm end effector, i.e., the homogeneous transformation matrix. In this embodiment of the application, the following is adopted: The separate solution method calibrates the relative pose relationship between the coordinate system of the first vision sensor and the coordinate system of the first robotic arm's end effector. For example, by having the first robotic arm carry the first vision sensor to observe a fixed calibration object (e.g., a checkerboard calibration board) from different poses, the constraint relationship between the motion of the first vision sensor and the motion of the first robotic arm's end effector is established, thereby solving for the homogeneous transformation matrix. Similarly, the above method is used to determine the relative pose relationship between the second vision sensor and the coordinate system of the second robotic arm end effector.

[0069] Furthermore, in practical applications, it is necessary to convert the target object positions perceived by the first and second vision sensors into the robot's base coordinate system B so that the controller can understand and execute the commands. Therefore, it is necessary to calibrate the relationship between the first / second vision sensors and the robot's base coordinate system. The visual detection calibration methods for the first and second vision sensors are the same; therefore, the following explanation does not distinguish between the visual calibration of the first and second vision sensors, but uses the calibration process between the camera and the robot's base coordinate system B as an example: First, the camera acquires images and uses an object detection algorithm (such as YOLO) to obtain the two-dimensional pixel coordinates of the target in the image. Combined with the depth values ​​provided by the depth camera Using the known camera intrinsic parameter matrix, two-dimensional pixels can be back-projected into three-dimensional coordinates in the camera coordinate system C. The transformation relationship is as follows: ; in, and For camera focal length, and These are the coordinates of the principal point of the image.

[0070] After obtaining the target's coordinates in the camera coordinate system, they need to be transformed to the robot's base coordinate system B using the homogeneous transformation matrix described above. This process is fundamental to achieving vision-guided grasping or manipulation, and its transformation chain is as follows: ; in: P B These are the coordinates of the target transformed into the robot's base coordinate system B; These are the homogeneous coordinates of the target in the camera coordinate system C; It is the homogeneous transformation matrix obtained from the camera coordinate system C to the end effector coordinate system E, obtained through hand-eye calibration; It is the transformation matrix from the end effector coordinate system E to the robot base coordinate system B (this matrix is ​​calculated in real time by reading the robot's current joint angles and using the positive kinematics model).

[0071] In this way, the system can accurately convert the target "seen" by the visual sensor into the target's three-dimensional spatial pose that the robot "can reach".

[0072] Based on the aforementioned calibration, when the robot moves near the operating table and needs to perform a liquid retrieval operation, the observation pose is determined as follows: The first robotic arm first moves to the first initial position pose P0. At this position, according to the homogeneous transformation matrix... The robot performs visual detection and localization of target liquid bottles and containers, and calculates the precise pose of the target liquid bottle in the robot's base coordinate system. .

[0073] Adopt the target liquid collection bottle pose To construct the dynamic anchor point for this task—the ready position pose P1. For example, the calculation process is as follows: Determine the position of the target liquid collection bottle The horizontal coordinates (X, Y) are used to add a preset safety offset to the Z-axis of the coordinate system to obtain the ready position pose P1. That is, the pose of P1 is set to a predefined, fixed value that ensures the pipette axis is in the negative direction of the Z-axis of the world coordinate system (i.e., vertically downwards). By combining the calculated position with the preset pose, the system generates a complete and correctly oriented ready position pose P1.

[0074] Based on the ready position P1, the device moves along the Z-axis of its coordinate system to determine the liquid retrieval position P2, maintaining the same posture. It is ensured that the projections of the ready position P1 and the liquid retrieval position P2 on the horizontal (XY) plane are completely coincident, achieving strict vertical lifting and lowering. That is, the liquid retrieval position P2 is calculated by moving the ready position P1 vertically downwards along its Z-axis by a third predetermined height. This third predetermined height is determined based on the liquid level in the target liquid retrieval bottle, ensuring that the retrieval tube is submerged in the liquid level of the target liquid retrieval bottle at the retrieval position.

[0075] Once the liquid sampling position P2 is determined, then the pre-calibrated relative transformation matrix of the two arms is used. The observation position C corresponding to the liquid sampling position P2 is calculated.

[0076] In some embodiments, if different observation positions are set according to different target liquid volumes, then after determining the liquid collection position P2, the observation position is determined by a two-arm relative transformation matrix. The first position C1 is calculated; then, the second position C2 is calculated by moving the first position C1 vertically upward along its Z-axis by a first set height. If there are more than three observation positions, the calculation method for the other positions is the same as that for the second position, and will not be repeated here.

[0077] S303, control the first robotic arm to move to the liquid collection position, and control the second robotic arm to move to the observation position; S304, determine the target liquid level height of the pipette based on the target liquid volume.

[0078] In this embodiment, before controlling the pipette to take liquid, the pipette is first moved to the liquid taking position, the second vision sensor is moved to the observation position, and the target liquid level height corresponding to the pipette when the target liquid taking volume is reached is determined before the liquid taking operation is performed. This allows for synchronous feedback of changes in the actual liquid level height during the liquid taking process, and then adjustment of the actual liquid taking volume of the pipette.

[0079] As an example, determining the target liquid level height of the pipette based on the target liquid volume includes: obtaining the calibration relationship between liquid level height and liquid volume; and determining the target liquid level height corresponding to the target liquid volume based on the calibration relationship. That is, by pre-calibrating the correspondence between the liquid level height and liquid volume of the pipette, the target liquid volume can be converted into the corresponding liquid level height through the calibration system. This facilitates observation by the second vision sensor and allows the controller to quickly determine whether the target liquid volume has been reached, thereby controlling the pipette to stop dispensing liquid.

[0080] As an example and for limitation, the following illustrates one method for calibrating liquid level and volume: The calibration process for liquid level height and liquid volume includes vision-based liquid level detection and calibration, which mainly includes the following steps: after the second vision sensor acquires the image, the image is enhanced to obtain an enhanced image; the liquid level height is detected based on the enhanced image; calibration is performed based on the liquid level height and liquid volume, and a calibration relationship comparison table of liquid level height and liquid volume is generated by calibrating different liquid level heights and liquid volumes multiple times.

[0081] For example, in this embodiment, in order to enhance the grayscale contrast between the liquid surface, the liquid extraction tube area and the background, improve edge saliency, and thus provide a stable and clear image input for subsequent liquid surface extraction, a variety of image enhancement algorithms are used to process the image, including gamma transform, Gaussian filtering and morphological operations.

[0082] First, a gamma transform is performed on the image to adjust the overall brightness distribution and improve the visibility of the liquid surface in the sampling tube. The gamma transform is a non-linear grayscale mapping, and its mathematical expression is: ; in, I is the normalized value of the input image; c is a scaling constant, which is 1 in this embodiment; γ The gamma coefficient is optional. γ The value is 3.

[0083] After gamma transform enhancement, Gaussian filtering is applied to smooth the enhanced image. By weighting the pixel neighborhood with Gaussian weights, random noise and isolated bright spots are effectively suppressed, while preserving the main contours of the liquid collection tube edge and liquid surface area. Subsequently, morphological opening operations based on rectangular structuring elements are used to further optimize the filtering results. This process removes small isolated bright spots and reflective false edges, preserves the vertical boundary structure of the liquid collection tube, and improves the grayscale consistency of the liquid surface area.

[0084] Since the observation position C and the liquid collection position P2 are relatively fixed, in order to eliminate interference from irrelevant backgrounds, this embodiment cropps the image acquired by the second visual sensor at a fixed ratio around the center of the liquid collection tube (e.g., ...). Figure 5 ).

[0085] The liquid level calculation aims to accurately obtain the change in liquid level within the sampling tube. In this embodiment, the sampling volume (i.e., the actual amount of liquid sampled) can be calculated using the pixel difference between the liquid level position and the reference position. It is understood that, as... Figure 5 As shown, the pixel difference between the pipette tip and the liquid collection tube is relatively large and there is a clear boundary. Therefore, the connection between the liquid collection tube and the pipette is used as the reference position. To ensure the stability of the liquid level measurement, the reference position is a pre-calibrated fixed height, which serves as the starting point for liquid level detection, thereby determining the actual liquid level height in the liquid collection tube.

[0086] For example, the enhanced image is processed to identify reference locations. Specifically, the Otsu thresholding method is used to calculate the separation threshold. The Otsu method is an adaptive thresholding segmentation method based on the image grayscale distribution. By calculating the inter-class variance between the foreground and background in the image, the grayscale value that maximizes the inter-class variance is selected as the optimal threshold for binarization, thereby achieving effective separation of the foreground and background.

[0087] To achieve image binarization, a threshold T needs to be selected to divide the image pixels into foreground and background parts, with grayscale values ​​ranging from [0, T] to [T+1, 255], respectively. Let the grayscale value of the foreground region be... μ 0, the expected grayscale value of the background area is 0. μ 1. The pixel ratios of the foreground and background are respectively ω 0 and ω 1. The average grayscale value of the entire image is μ Then we can define it as follows: ; ; ; Where Pi is the number of pixels with a grayscale value of i.

[0088] Based on the above definition, the inter-class variance σ can be calculated. 2 : ; Otsu's thresholding method searches for a threshold value that minimizes the inter-class variance σ by iterating through all possible grayscale thresholds. 2 The maximum threshold is selected as the binarization standard for the liquid surface image.

[0089] Furthermore, the input image is segmented using a binary threshold, and the calculation process is as follows: ; in, T ostu The optimal threshold obtained based on Otsu's method is... img binary This is the binary image obtained by segmenting using the optimal threshold.

[0090] After segmentation using the optimal threshold, the outer contours of the segmented image are extracted and all extracted contours are filled. Finally, area filtering is used to remove noise contours with values ​​less than 20 pixels, resulting in a binary image with the liquid collection tube as the foreground region. ; in, mask ( x, y () is a binary image with noise contours greater than 20 pixels.

[0091] Furthermore, to determine the position of the calibration point in the image, each row of the binary image is scanned from top to bottom, starting from the y-coordinate 0 point. The number of non-zero pixels is counted. When the number of consecutive non-zero pixels exceeds the empirical threshold of 20 set in this embodiment, the y-coordinate of that row is recorded as the reference position coordinate. top To obtain a reference position.

[0092] After determining the coordinates of the reference position, the liquid level is detected to calculate the height difference and then fit the actual liquid volume. As an example and not a limitation, this application embodiment identifies the liquid level in the sampling tube in the following manner: Since liquid level detection only needs to focus on the liquid-absorbing area of ​​the sampling tube, to avoid interference and improve detection accuracy, the region of interest (ROI) is further defined based on the determined reference position. Specifically, an offset is added to the vertical coordinate y of the reference position; for example, the ROI range is limited to [y+350:y+900]. This operation crops the image to only the effective area containing the liquid-absorbing area of ​​the sampling tube, ensuring that the input to the subsequent liquid level detection algorithm is more focused and accurate.

[0093] After obtaining the cropped effective region image, to accurately extract the liquid surface location, a detection method based on vertical gradient changes is used. The Sobel operator is applied to the cropped effective region image to enhance the edge features in the vertical direction. The significant gradient change region where the liquid surface is located is extracted using the first derivative response of the Sobel operator in the vertical direction. The calculation formula is as follows: ; in, For the input image, Let be the gradient matrix in the vertical direction.

[0094] The absolute value of the calculated gradient matrix is ​​taken and then summed along the horizontal direction to obtain the vertical gradient intensity distribution S(y) of each row. The calculation method is as follows: ; After obtaining the gradient intensity distribution S(y) Then, select the vertical index corresponding to its maximum value. y(max) The location where the liquid surface gradient changes most significantly is taken as the liquid surface position.

[0095] After identifying the liquid level position inside the sampling tube using the above method, the difference between the liquid level position and the reference position is calculated to obtain the final actual liquid level height. For example, the actual liquid level height is calculated using the following formula: ; Where h is the actual liquid level height in the sampling tube. y top For reference position, y max This indicates the liquid level.

[0096] In some embodiments, to obtain accurate liquid surface pixel height, multiple observation positions are set to adjust the viewing angle of the second visual sensor relative to the liquid surface, thereby reducing detection errors caused by the liquid surface shooting angle. For example, the liquid volume is divided into three parts: 0-200ul, 200-600ul, and 600-1000ul. When the liquid volume falls within the corresponding range, the second visual sensor is moved to the preset corresponding positions C3, C1, and C2 respectively, and the corresponding image is captured.

[0097] Optionally, within each liquid volume range, different volumes of purified water are sequentially drawn in increments of 50 μL. The vertical coordinate pixel height h of the liquid surface and the reference position, as well as the corresponding liquid volume Vi, are recorded. After recording multiple sets of data, a fitting formula between the height h and the corresponding liquid volume is established.

[0098] For example, for a liquid volume of 0-200 μL, the second vision sensor is fixed at the third position C3: First, determine the height ht of the conical section of the pipette tube and the volume of liquid to be dispensed. V t For a fixed sampling tube, these two should be constants, and V t ≤200ul. Alternatively, in other embodiments, the liquid can be manually aspirated to the 200ul threshold to obtain ht, and the volume of liquid taken can be obtained by measurement. V t .

[0099] When the liquid volume is 0 < V i <= V t At this time, the liquid level is located in the conical region of the liquid collection tube, and the height h is related to the liquid collection volume. V iThe relationship exhibits a nonlinear correspondence, which is fitted using a cubic nonlinear function. The corresponding formula is: ; Where H is the length of the liquid collection tube, and a is a constant that is related to the height and diameter of the conical region of the liquid collection tube.

[0100] When the liquid volume V t < V i When the liquid level is less than or equal to 200 μL, the liquid level is located in the cylindrical region of the sampling tube, and the height h is related to the volume of liquid sampled. V i They exhibit a linear correspondence, and the corresponding formula is: ; in, V t The volume of liquid taken corresponding to the cone section V t , b It is a constant.

[0101] When the aspirated volume is greater than 200 μL, the second visual sensor is located at positions C1 and C2: At this point, the liquid level is mainly located in the cylindrical area of ​​the liquid collection tube, and the liquid level height h is related to the liquid collection volume. V i They exhibit a linear correspondence, and a linear function is used for fitting; the corresponding formula is: ; like Figure 6 As shown, by fitting the collected data with the corresponding fitting function, the fitting function can be obtained. The liquid level height h and the volume of liquid sampled can then be obtained through the corresponding fitting function. V i By establishing the correspondence between the liquid volume and the liquid level, and calibrating the relationship between the liquid volume and the liquid level, the liquid volume corresponding to any liquid level h can be obtained. V It is also possible to obtain any volume of liquid taken. V The corresponding target liquid level height is also indicated in a table, showing the calibration relationship between the liquid level height and the volume of liquid taken.

[0102] S305 controls the pipette to pick up liquid and controls the second vision sensor to collect the actual liquid level height of the pipette; S306: When the actual liquid level reaches the target liquid level, control the pipette to stop taking liquid and complete the liquid taking.

[0103] In this embodiment, during the process of controlling the pipette to pick up liquid, the second vision sensor is controlled to collect the actual liquid level height of the pipette and compare the actual liquid level height with the target liquid level height. If the difference between the actual liquid level height and the target liquid level height is less than a preset difference, it is determined that the actual liquid level height has reached the target liquid level height, and the pipette is controlled to stop picking up liquid, thus completing the liquid picking process.

[0104] In some embodiments, when the actual liquid level reaches the target liquid level, the pipette is controlled to stop dispensing liquid. Completing the dispensing process includes: after dispensing liquid at a set dispensing speed, if the actual liquid level reaches the target liquid level (e.g., actual liquid level h1 = target liquid level h), the pipette is controlled to stop dispensing liquid. After a preset time interval, a second vision sensor is controlled to collect the actual liquid level h' of the pipette for dispensing verification. For example, if |h'-h|>th, an error is considered to exist in the dispensing, and the final accuracy can be improved by slow dispensing or aspirating.

[0105] As an example, and not a limitation, to improve liquid sampling accuracy while maintaining efficiency, multiple sampling speed settings can be adopted. For example, the sampling speed can be determined based on the difference between the actual liquid level and the target liquid level. The sampling speed controls the liquid dispensing gun. If the difference between the target liquid level and the current actual liquid level is large, a first sampling speed can be set. If the difference is small, a second sampling speed can be set. The first sampling speed is greater than the second sampling speed, meaning that liquid is sampled quickly in the initial sampling stage to increase the sampling speed, while the sampling speed is slowed down when approaching the target volume to avoid the difference in liquid volume affecting sampling accuracy. For example, the first sampling speed can be 50 microliters / second, and the second sampling speed can be 5 microliters / second.

[0106] S307 controls the movement of the first robotic arm to move the liquid gun to the target container and perform the liquid injection operation.

[0107] After accurately collecting the liquid, the first robotic arm is moved to the injection position PE. The liquid gun is positioned directly above the target container at the PE position to perform the injection operation.

[0108] In one embodiment, while controlling the first robotic arm to perform the liquid injection operation, the second robotic arm is simultaneously controlled to move, moving the second vision sensor to the observation position corresponding to the injection position. The pipette is then controlled to dispense all the liquid, and the second vision sensor simultaneously observes the liquid level remaining in the pipette tip after dispensing. Normally, there should be no residue. If residue remains, the dispensing command is repeatedly sent to the pipette to force emptying the pipette. If residue still remains after retrying, the liquid level height h3 is detected and recorded, and the residual liquid volume b3 is retrieved. This volume is then recorded in the experimental record or used for additional pipetting, depending on the specific circumstances.

[0109] In this embodiment, when the target liquid volume exceeds the first capacity, a simultaneous liquid collection and observation method is adopted to simplify the liquid collection process and improve collection efficiency. Furthermore, in some embodiments, different observation positions are used to observe the liquid collection tube for different liquid level heights, improving the accuracy of liquid level detection. Based on different observation positions and liquid level heights, piecewise functions are fitted and constructed to achieve a calibration mapping relationship between visual pixel height and actual liquid volume.

[0110] Figure 7 This illustration shows a schematic flowchart of the dual-arm embodied visual feedback precision pipetting method provided in this application. This embodiment is a further embodiment based on the foregoing embodiments, and is intended as an example rather than a limitation, including: S701, Obtain liquid dispensing command, and determine the target liquid dispensing volume based on the liquid dispensing command.

[0111] S702, if the target liquid volume is less than or equal to the first capacity, the observed position of the pipette is determined according to the liquid collection position of the first robotic arm.

[0112] S703, the third position is determined based on the observed position and the relative transformation matrix of the two arms, and the third position is the observation position.

[0113] The difference between this embodiment and the previous embodiment is that when the target liquid volume is less than a1, the liquid level in the sampling tube is lower than the mouth of the sampling bottle, making it impossible to accurately observe the actual liquid level. Therefore, in this embodiment, when the target liquid volume is less than or equal to a1 microliters, the first robotic arm is controlled to move, moving the sampling gun to the observation position (so that the liquid level is outside the sampling bottle), and then the actual liquid level in the sampling tube is collected by the second vision sensor. That is, this embodiment separates the liquid sampling and observation operations, sampling first and then observing, which is an offline observation liquid sampling method.

[0114] like Figure 8 As shown, in this embodiment, the observed position of the first robotic arm is different from the liquid retrieval position. For example, the observed position P3 is obtained by moving the liquid retrieval position P2 vertically upward by a set distance along its Z-axis.

[0115] Correspondingly, in some embodiments, such as Figure 9 As shown, the first robotic arm is configured with three observation positions: position C1, position C2, and position C3. When the pipette is located at the observed position, the second vision sensor observes from position C3. That is, the observed position P3 is associated with a third position C3, serving as the observation position for the second vision sensor. When the pipette can be observed from the liquid dispensing position, the observation position of the second vision sensor is either position C1 or position C2 (as in the aforementioned embodiment).

[0116] It is understood that in this embodiment, the relative transformation matrix of the two arms is the same as that of the relative transformation matrix of the two arms in the previous embodiment. The calibration process of the relative transformation matrix of the two arms can be referred to the previous embodiment, and will not be repeated here.

[0117] S704, after the pipette completes the initial liquid collection at the liquid collection position, control the first robotic arm to move the pipette to the observation position; S705 controls the movement of the second robotic arm to move the second vision sensor to the observation position and collect the actual liquid level height of the pipette. S706: Adjust the actual liquid volume of the pipette according to the actual liquid level until the difference between the actual liquid volume and the target liquid volume is less than or equal to the preset difference, and the liquid dispensing is completed.

[0118] Since liquid collection and image acquisition cannot be performed simultaneously in this embodiment, after determining the observed position and the observation position, the pipette is first controlled to complete the initial liquid collection at the liquid collection position. Then, the first robotic arm is controlled to move and move the pipette to the observed position. At the same time, the second robotic arm is controlled to move and move the second vision sensor to the third position to collect the actual liquid level height of the pipette's collection tube.

[0119] In this embodiment, the method of initial liquid collection with a pipette is not limited.

[0120] As an example, adjusting the actual liquid volume of the pipette according to the actual liquid level until the difference between the actual liquid volume and the target liquid volume is less than or equal to a preset difference, the specific process of completing the liquid dispensing can include the following steps: A1: Obtain the calibration relationship between liquid level and liquid volume. The calibration relationship between liquid level and liquid volume is obtained through the calibration process described in the preceding embodiments, and will not be repeated here.

[0121] A2: Determine the actual liquid volume corresponding to the actual liquid level height based on the calibration relationship.

[0122] Based on the calibration relationship between liquid level and liquid volume, the corresponding liquid volume can be determined after obtaining the actual liquid level in the liquid collection tube.

[0123] It is understood that in this embodiment, the determination of the actual liquid level in the liquid sampling tube is the same as the calculation method of the liquid level in the previous embodiment. Specifically, the calculation method described in the above embodiment can be used, which will not be repeated here.

[0124] A3: If the actual liquid volume is less than the target liquid volume, the target replenishment volume is determined based on the ratio of the actual liquid volume to the target liquid volume and the preset correction deviation.

[0125] If, after the initial liquid extraction is completed, it is determined based on the actual liquid level height fed back by the second vision sensor that the actual liquid extraction volume is less than the target liquid extraction volume, it indicates that the current liquid extraction volume of the liquid extraction gun does not meet the requirements. At this time, it is necessary to control the liquid extraction gun to continue extracting liquid to supplement the liquid extraction volume.

[0126] In a possible implementation, the target supplementary liquid volume is calculated in the following manner: Taking the actual liquid extraction volume as b1 and the target liquid extraction volume as a, if b1 < a, it is necessary to consider that supplementary liquid is needed, and calculate the target supplementary liquid volume: ; Among them, cmd is the target supplementary liquid volume, add is the actual volume of the liquid that needs to be supplemented add = a-b1 ; a / b1 is the correction coefficient, indicating the ratio of the liquid absorption coefficient of the liquid (type) based on the previous liquid extraction situation to the standard liquid; dt is the optional correction deviation amount, generally taking 0 μL - 5 μL, to ensure that after executing the supplementary liquid instruction, the total volume of the sucked liquid is slightly more or equal to the initial target a.

[0127] A4: Control the pipette gun to continue extracting liquid according to the target supplementary liquid volume until the difference between the actual liquid extraction volume and the target liquid extraction volume is less than or equal to the preset difference, and the liquid extraction is completed.

[0128] After determining the target supplementary liquid volume, control the first robotic arm to move to the liquid extraction position, the pipette gun enters the liquid surface, perform the second liquid extraction operation according to the target supplementary liquid volume, and then repeat the above steps A1 - A4 until the difference between the actual liquid extraction volume and the target liquid extraction volume is less than or equal to the preset difference, and the liquid extraction is completed.

[0129] A5: If the actual liquid extraction volume is greater than the target liquid extraction volume, control the pipette gun to eject liquid until the difference between the actual liquid extraction volume and the target liquid extraction volume is less than or equal to the preset difference, and the liquid extraction is completed.

[0130] Taking the actual liquid extraction volume as b2 as an example, if the actual liquid extraction volume b2 is greater than the target liquid extraction volume a, execute the slow liquid ejection instruction, and at the same time control the second vision sensor to collect the actual liquid level height, and control the pipette gun to stop ejecting liquid according to the actual liquid level height collected by the second vision sensor, and the liquid extraction is completed.

[0131] S707, control the first robotic arm to move so that the liquid gun moves to the target container and perform the liquid injection operation.

[0132] After the first robotic arm completes the liquid extraction, the first robotic arm performs the liquid injection operation to complete one liquid transfer.

[0133] In this embodiment of the application, for liquid sampling scenarios with small liquid volumes, the pipette is moved to the observed position to provide a better observation position for the second vision sensor, thereby improving the accuracy of liquid sampling.

[0134] In conjunction with the aforementioned embodiments, different pipetting schemes are adopted for different target liquid volumes, so that the system can adapt to situations where the liquid level corresponding to the liquid volume is inside the liquid bottle and cannot be observed, while ensuring efficient liquid collection.

[0135] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] Corresponding to the dual-arm omnidirectional visual feedback precise pipetting method in the above embodiment, Figure 5 The diagram shows a structural block diagram of the dual-arm embodied visual feedback precision pipetting device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0137] Reference Figure 10 The dual-arm, body-worn visual feedback precision pipetting device 100 includes: The determining unit 1001 is used to determine the observation position of the second vision sensor based on the target liquid volume and the liquid collection position of the first robotic arm. The liquid collection unit 1002 is used to control the movement of the first robotic arm so that the pipette can collect liquid at the liquid collection position, and adjust the actual liquid collection volume of the pipette according to the actual liquid level height of the pipette collected by the second vision sensor at the observation position, so that the difference between the actual liquid collection volume and the target liquid collection volume is less than or equal to a preset difference, and the liquid collection is completed. The liquid injection unit 1003 is used to control the movement of the first robotic arm so that the liquid gun moves to the target container to perform the liquid injection operation.

[0138] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0140] This application also provides a robot, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0141] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0142] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0143] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A dual-arm body vision feedback precision pipetting system, characterized in that, The robot body comprises a controller; The first operation mechanism comprises a first mechanical arm and a pipette, the first mechanical arm is arranged on one side of the robot body, and the pipette is arranged at the end of the first mechanical arm, and the liquid taking pipe of the pipette has a visible area; The second operation mechanism comprises a second mechanical arm and a second visual sensor, the second mechanical arm is arranged on the other side of the robot body, and the second visual sensor is arranged on the second mechanical arm; The controller is configured to determine the observation position of the second visual sensor according to the liquid taking position of the first mechanical arm and the target liquid taking amount, control the second mechanical arm to move, and move the second visual sensor to the observation position, and collect the actual liquid level in the pipette through the visible area; The controller is further configured to adjust the actual liquid taking amount of the pipette according to the actual liquid level, so that the difference between the actual liquid taking amount and the target liquid taking amount is less than or equal to a preset difference, complete liquid taking, and control the first mechanical arm to move, so as to move the pipette to a target container and execute liquid injection operation.

2. The double-arm body visual feedback precise pipetting system according to claim 1, wherein If the target liquid taking amount is greater than a first capacity, the observation position of the second visual sensor is determined according to the target liquid taking amount and the liquid taking position of the pipette; If the target liquid taking amount is less than or equal to the first capacity, the first mechanical arm is controlled to move the pipette to an observed position after the pipette takes liquid, and the observed position of the second visual sensor is determined according to the observed position. The liquid taking pipe of the pipette is a transparent pipe.

3. The dual arm body-vest visual feedback precision pipetting system of claim 2, wherein, The first operation mechanism further comprises a first visual sensor arranged on the first mechanical arm, and the first visual sensor is configured to locate the liquid taking position.

4. The dual arm body-vest visual feedback precision pipetting system of claim 1, wherein, The method is applied to the double-arm body visual feedback precise pipetting system of any one of claims 1 to 4, and the method comprises:

5. A two-armed body vision feedback precise pipetting method, characterized in that, determining the observation position of the second visual sensor according to the target liquid taking amount and the liquid taking position of the first mechanical arm; controlling the first mechanical arm to move, so that the pipette takes liquid at the liquid taking position, and adjusting the actual liquid taking amount of the pipette according to the actual liquid level of the pipette collected by the second visual sensor at the observation position, so that the difference between the actual liquid taking amount of the pipette and the target liquid taking amount is less than or equal to a preset difference, and completing liquid taking; controlling the first mechanical arm to move, so as to move the pipette to a target container and execute liquid injection operation. ​ 6. The dual arm body vision feedback precision pipetting method according to claim 5, wherein, The first mechanical arm is controlled to move so that the pipette takes liquid at the liquid-taking position, and according to the actual liquid level of the pipette collected by the second visual sensor at the observation position, the actual liquid-taking amount of the pipette is adjusted so that the difference between the actual liquid-taking amount of the pipette and the target liquid-taking amount is less than or equal to a preset difference, and liquid taking is completed, comprising: The first mechanical arm is controlled to move so that the pipette moves to the liquid-taking position, and the second mechanical arm is controlled to move so that the second visual sensor moves to the observation position; A target liquid level of the pipette is determined according to the target liquid-taking amount; The pipette is controlled to take liquid, and the second visual sensor is controlled to collect the actual liquid level of the pipette; When the actual liquid level reaches the target liquid level, the pipette is controlled to stop taking liquid, and liquid taking is completed.

7. The dual arm body vision feedback precision pipetting method according to claim 6, wherein, The target liquid level of the pipette is determined according to the target liquid-taking amount, comprising: A calibration relationship between liquid level and liquid-taking amount is obtained; A target liquid level corresponding to the target liquid-taking amount is determined according to the calibration relationship.

8. The dual arm body vision feedback precision pipetting method of claim 6, wherein, The observation position of the second visual sensor is determined according to the target liquid-taking amount and the liquid-taking position of the first mechanical arm, comprising: If the target liquid-taking amount is greater than a first capacity, the observation position of the second visual sensor is determined according to the liquid-taking position of the first mechanical arm.

9. The dual arm body-attaching visual feedback precision pipetting method according to claim 8, wherein, The observation position of the second visual sensor is determined according to the liquid-taking position of the first mechanical arm, comprising: If the target liquid-taking amount is greater than the first capacity and less than or equal to a second capacity, a first position is determined according to the liquid-taking position and a double-arm relative transformation matrix, the first position being the observation position; If the target liquid-taking amount is greater than the second capacity, a second position is determined according to the liquid-taking position, the double-arm relative transformation matrix and a first set height, the second position being the observation position, the second position being above the first position, and the second position being different from the first position by the first set height.

10. The dual arm body-attaching visual feedback precision pipetting method according to claim 5, wherein, The first mechanical arm is controlled to move so that the pipette takes liquid at the liquid-taking position, and according to the actual liquid level of the pipette collected by the second visual sensor at the observation position, the actual liquid-taking amount of the pipette is adjusted so that the difference between the actual liquid-taking amount of the pipette and the target liquid-taking amount is less than or equal to a preset difference, and liquid taking is completed, comprising: After the pipette completes preliminary liquid taking at the liquid-taking position, the first mechanical arm is controlled to move so that the pipette moves to an observed position, the observed position being determined according to the liquid-taking position of the first mechanical arm, and the observation position of the second visual sensor being determined according to the observed position; The second mechanical arm is controlled to move so that the second visual sensor moves to the observation position to collect the actual liquid level of the pipette; The actual liquid-taking amount of the pipette is adjusted according to the actual liquid level until the difference between the actual liquid-taking amount and the target liquid-taking amount is less than or equal to a preset difference, and liquid taking is completed.

11. The dual arm body-attaching visual feedback precision pipetting method according to claim 10, wherein, The actual liquid taking amount of the pipette is adjusted according to the actual liquid level until the difference between the actual liquid taking amount and the target liquid taking amount is less than or equal to a preset difference value, and liquid taking is completed, comprising: obtaining a calibration relationship between liquid level and liquid taking amount; determining the actual liquid taking amount corresponding to the actual liquid level according to the calibration relationship; if the actual liquid taking amount is less than the target liquid taking amount, determining a target liquid supplementing amount according to the ratio of the actual liquid taking amount and the target liquid taking amount and a preset correction deviation amount; controlling the pipette to continue taking liquid according to the target liquid supplementing amount until the difference between the actual liquid taking amount and the target liquid taking amount is less than or equal to the preset difference value, and liquid taking is completed.

12. The dual arm body-attuned visual feedback precision pipetting method of claim 11, wherein, Further comprising: if the actual liquid taking amount is greater than the target liquid taking amount, controlling the pipette to spit liquid until the difference between the actual liquid taking amount and the target liquid taking amount is less than or equal to the preset difference value, and liquid taking is completed.

13. The dual arm body-attaching visual feedback precision pipetting method of claim 10, wherein, The observation position of the second visual sensor is determined according to the target liquid taking amount and the liquid taking position of the first mechanical arm, comprising: if the target liquid taking amount is less than or equal to a first capacity, determining the observed position of the pipette according to the liquid taking position of the first mechanical arm; determining a third position according to the observed position and a double-arm relative transformation matrix, the third position being the observation position.

14. A dual arm body vision feedback precision pipetting device, characterized in that, Applied to the double-arm body visual feedback precise liquid taking system as claimed in any one of claims 1 to 4, comprising: a determination unit configured to determine the observation position of the second visual sensor according to the target liquid taking amount and the liquid taking position of the first mechanical arm; a liquid taking unit configured to control the first mechanical arm to move so that the pipette takes liquid at the liquid taking position, and to adjust the actual liquid taking amount of the pipette according to the actual liquid level of the pipette collected by the second visual sensor at the observation position, so that the difference between the actual liquid taking amount of the pipette and the target liquid taking amount is less than or equal to a preset difference value, and liquid taking is completed; a liquid injecting unit configured to control the first mechanical arm to move so that the pipette is moved to a target container to perform liquid injecting operation.

15. A robot, characterized in that comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method as claimed in any one of claims 5 to 12 when executing the computer program.

16. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the method as claimed in any one of claims 5 to 12.

Images