Pipetting workstation experiment control program generation method and device and electronic equipment

Abstract

The invention discloses a pipetting workstation experiment control program generation method and device and electronic equipment, and relates to the technical field of pipetting workstations and the technical field of visual recognizing.The method comprises the steps that an operation table image obtained by shooting an operation table of a pipetting workstation is obtained; performing experiment accessory identification on the operation table image to obtain accessory types of all experiment accessories placed on the operation table and positions of all the experiment accessories on the operation table; on the basis of the identified accessory types and the positions of the experimental accessories, according to the experimental process needing to be executed, an experimental control program matched with the experimental process is generated, and the experimental control program is used for controlling a pipetting arm of the pipetting workstation to operate the experimental accessories on the operation table so as to complete the experimental process. By adopting the scheme, the use convenience and efficiency of the pipetting workstation are improved.

Description

Technical Field

[0001] This application relates to the fields of pipetting workstation technology and visual recognition technology, and in particular to a method, apparatus and electronic device for generating experimental control programs for a pipetting workstation. Background Technology

[0002] A pipetting workstation is an analytical instrument used in the field of biology. With the rapid development of fields such as biopharmaceuticals, the number of liquid samples that need to be processed in a short period of time is constantly increasing, promoting the rapid development of pipetting workstations. A pipetting workstation is a highly flexible and expandable device that can replace traditional pipetting tools in sample analysis, automatically completing high-precision liquid handling tasks such as gradient dilution, pipetting, and mixing.

[0003] The pipetting workstation has an operating table and a pipetting arm. Various experimental accessories can be placed on the operating table. During use, the experimental control program can be used to control the pipetting arm to transfer the experimental liquid between the various experimental accessories, thereby completing a specific experimental procedure.

[0004] For example, it can perform functions such as continuous gradient dilution, liquid distribution, addition, combination and continuous gradient dilution between microplates, and is widely used in genomics research such as nucleic acid extraction, construction of various biological laboratory systems, large-scale sequencing, drug screening, protein analysis and other large-scale pipetting processes.

[0005] As a liquid handling device, the pipetting workstation features high automation, high precision, high compatibility, high cost-effectiveness, and high throughput.

[0006] In practical applications of pipetting workstations, some experimental procedures are quite complex and often require the use of various experimental accessories. Operators need to pre-place these accessories accurately in their designated positions on the worktable so that the matching experimental control program can be used to control the pipetting arm to complete the corresponding experimental procedure. However, due to the large variety of experimental accessories, and the similar appearance of some different types, operators may inadvertently place them in the wrong position, leading to errors in the subsequent experimental procedure, ultimately resulting in experimental failure and significant losses. Summary of the Invention

[0007] This application provides a method, apparatus, and electronic device for generating experimental control programs for a pipetting workstation, in order to solve the problems of inconvenience and low efficiency of pipetting workstations in the prior art.

[0008] This application provides a method for generating a pipetting workstation experimental control program, including:

[0009] Acquire images of the control panel of the pipetting workstation by taking photographs;

[0010] The experimental attachments are identified by performing experimental attachment recognition on the operation table image to obtain the attachment type of each experimental attachment placed on the operation table, as well as the position of each experimental attachment on the operation table;

[0011] Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the experimental procedure is generated according to the required experimental procedure. The experimental control program is used to control the pipetting arm of the pipetting workstation to operate each experimental accessory on the operating table to complete the experimental procedure.

[0012] Furthermore, the area on the operating table used for placing experimental accessories is divided into multiple sub-areas;

[0013] The acquisition of the image of the operating table obtained by photographing the operating table of the pipetting workstation includes:

[0014] Multiple images of the operating table of the pipetting workstation are acquired by taking pictures of the operating table. The multiple operating table images correspond one-to-one with the multiple sub-regions, and each operating table image is an image of the corresponding sub-region.

[0015] The step of identifying experimental attachments in the image of the workbench to obtain the attachment types of each experimental attachment placed on the workbench and the position of each experimental attachment on the workbench can be performed using any of the following methods:

[0016] Experimental attachments are identified in the multiple workbench images to obtain the attachment types of experimental attachments placed on each sub-region of the workbench. The position of each sub-region on the workbench is taken as the position of the experimental attachments placed on that sub-region on the workbench.

[0017] Alternatively, experimental attachment recognition can be performed on the multiple workbench images to obtain the attachment type of the experimental attachment placed in each sub-region and the location of the experimental attachment placed in each sub-region on that sub-region. Combined with the preset position of that sub-region on the workbench, the location of the experimental attachment placed in that sub-region on the workbench can be determined.

[0018] Furthermore, the step of identifying experimental attachments in the image of the operating table to obtain the attachment types of each experimental attachment placed on the operating table and the position of each experimental attachment on the operating table includes:

[0019] The experimental attachments are identified in the image of the operating table to obtain the attachment type of each experimental attachment placed on the operating table, the position of each experimental attachment on the operating table, the size of each experimental attachment, and the position of each component on each experimental attachment.

[0020] Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the required experimental procedure is generated, including:

[0021] Based on the identified types and locations of the experimental attachments, the dimensions of the experimental attachments, and the locations of the components on each experimental attachment, an experimental control program matching the experimental procedure to be executed is generated.

[0022] Furthermore, based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the required experimental procedure is generated, including:

[0023] Obtain the current experimental control program corresponding to the experimental procedure to be executed;

[0024] Based on the identified accessory types and locations of each experimental accessory, the position parameters representing each experimental accessory in the current experimental control program are changed to represent the locations of each experimental accessory, thereby obtaining an experimental control program that matches the experimental process.

[0025] Furthermore, before generating an experimental control program matching the experimental procedure according to the required experimental procedure based on the identified accessory types and locations of each experimental accessory, the method further includes:

[0026] Determine the experimental procedures to be performed based on the user's selection of the displayed human-computer interaction interface.

[0027] Furthermore, after recognizing experimental attachments in the control panel image, and before generating an experimental control program matching the experimental procedure according to the required experimental procedure based on the type and location of the recognized experimental attachments, the method further includes:

[0028] The types and locations of the identified experimental accessories are displayed on the human-computer interaction interface.

[0029] After receiving a user's input indicating that the recognition result is correct, the steps are as follows: based on the recognition of the types and locations of the various experimental attachments, generate an experimental control program that matches the experimental process according to the required experimental process.

[0030] After receiving a user's input indicating an error in the recognition result, the system displays a recognition result correction option on the human-computer interaction interface, which is used to receive the corrected recognition result from the user.

[0031] Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the required experimental procedure is generated, including:

[0032] Based on the corrected identification results, which indicate the attachment types and locations of each experimental attachment, an experimental control program matching the experimental procedure to be executed is generated.

[0033] Furthermore, the experimental attachment recognition of the operating table image includes:

[0034] A pre-trained machine learning model is used to identify experimental attachments in the workbench image. The machine learning model is trained based on known types of experimental attachment samples and the features of each attachment in the experimental attachment samples.

[0035] The features of each accessory include at least one of the following features:

[0036] Shape, color, size, number of holes, hole spacing, and label.

[0037] This application embodiment also provides a device for generating a pipetting workstation experimental control program, including:

[0038] The image acquisition module is used to acquire images of the operating table of the pipetting workstation obtained by taking pictures of the operating table;

[0039] An image recognition module is used to identify experimental attachments in the image of the operating table, and to obtain the attachment type of each experimental attachment placed on the operating table, as well as the position of each experimental attachment on the operating table.

[0040] The program generation module is used to generate an experimental control program that matches the experimental procedure according to the identified accessory types and locations of each experimental accessory. The experimental control program is used to control the pipetting arm of the pipetting workstation to operate each experimental accessory on the operating table to complete the experimental procedure.

[0041] Furthermore, the area on the operating table used for placing experimental accessories is divided into multiple sub-areas;

[0042] The image acquisition module is specifically used to acquire multiple images of the operating table of the pipetting workstation by taking pictures of the operating table. The multiple operating table images correspond one-to-one with the multiple sub-regions, and each operating table image is an image of the corresponding sub-region.

[0043] The image recognition module is specifically used to identify experimental attachments in the multiple workbench images respectively, to obtain the attachment type of the experimental attachments placed on each sub-area of ​​the workbench, and the position of each sub-area on the workbench is used as the position of the experimental attachments placed on that sub-area on the workbench, or the position of the experimental attachments placed on each sub-area is used as the position of the experimental attachments placed on that sub-area on the workbench.

[0044] Furthermore, the image recognition module is specifically used to identify experimental attachments in the image of the operating table, and to obtain the attachment type of each experimental attachment placed on the operating table, the position of each experimental attachment on the operating table, the size of each experimental attachment, and the position of each component on each experimental attachment.

[0045] The program generation module is specifically used to generate an experimental control program that matches the experimental process according to the identified types and locations of the experimental accessories, the dimensions of the experimental accessories, and the locations of the components on the experimental accessories.

[0046] Furthermore, the program generation module is specifically used to obtain the current experimental control program corresponding to the experimental process to be executed; according to the identified attachment types and locations of each experimental attachment, the position parameters representing each experimental attachment in the current experimental control program are changed to represent the locations of each experimental attachment, thereby obtaining an experimental control program that matches the experimental process.

[0047] Furthermore, the program generation module is also used to determine the experimental process to be executed selected by the user based on the displayed human-computer interaction interface before generating an experimental control program that matches the experimental process according to the experimental process to be executed based on the identified attachment types and locations of the experimental attachments.

[0048] Furthermore, the program generation module is also configured to: after the image recognition module performs experimental accessory recognition on the operating table image, display the accessory type and location of each identified experimental accessory on the human-computer interaction interface; after receiving a user input indicating that the recognition result is correct, execute the step of generating an experimental control program matching the experimental process based on the identified accessory type and location of each experimental accessory and according to the required experimental process; and after receiving a user input indicating that the recognition result is incorrect, display a recognition result correction option on the human-computer interaction interface to receive the corrected recognition result input by the user.

[0049] The program generation module is specifically used to generate an experimental control program that matches the experimental process according to the experimental process to be executed, based on the attachment type and location of each experimental attachment represented by the corrected identification result.

[0050] Furthermore, the image recognition module is specifically used to identify experimental attachments in the workbench image using a pre-trained machine learning model. The machine learning model is trained based on known types of experimental attachment samples and the attachment features of each of the experimental attachment samples.

[0051] The features of each accessory include at least one of the following features:

[0052] Shape, color, size, number of holes, hole spacing, and label.

[0053] This application also provides an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, and the processor is prompted by the machine-executable instructions to implement any of the above-described methods for generating experimental control programs for pipetting workstations.

[0054] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for generating experimental control programs for pipetting workstations.

[0055] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the above-described pipetting workstation experimental control program generation methods.

[0056] The beneficial effects of this application include:

[0057] The method provided in this application involves acquiring an image of the operating table of a pipetting workstation by taking a photograph. Experimental attachments are identified from the image to determine the attachment type and location of each attachment on the table. Based on the identified attachment types and locations, an experimental control program matching the required experimental procedure is generated. This program controls the pipetting arm of the workstation to manipulate the attachments on the table to complete the experimental procedure. This method eliminates the need for operators to place attachments in specific locations; they only need to place them on the operating table. This reduces the operational requirements for operators, making the workstation easier to use and improving their operational efficiency, thus increasing the overall efficiency of the pipetting workstation.

[0058] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0059] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. In the drawings:

[0060] Figure 1 A flowchart illustrating the method for generating an experimental control program for a pipetting workstation provided in this application embodiment;

[0061] Figure 2 A flowchart illustrating a method for generating a pipetting workstation experimental control program according to another embodiment of this application;

[0062] Figure 3 This is a schematic diagram of the structure of the pipetting workstation experimental control program generation device provided in the embodiments of this application;

[0063] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0064] To provide a solution for improving the convenience and efficiency of pipetting workstations, this application provides a method, apparatus, and electronic device for generating experimental control programs for pipetting workstations. The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit this application. Furthermore, the embodiments and features described in this application can be combined with each other unless otherwise specified.

[0065] This application provides a method for generating an experimental control program for a pipetting workstation, such as... Figure 1 As shown, it includes:

[0066] Step 11: Obtain an image of the control panel of the pipetting workstation by taking a picture of it;

[0067] Step 12: Perform experimental accessory recognition on the workbench image to obtain the accessory type of each experimental accessory placed on the workbench and the location of each experimental accessory on the workbench.

[0068] Step 13: Based on the identified types and locations of each experimental accessory, generate an experimental control program that matches the experimental procedure to be executed. This experimental control program is used to control the pipetting arm of the pipetting workstation to operate on each experimental accessory on the operating table to complete the experimental procedure.

[0069] The method for generating experimental control programs for pipetting workstations provided in this application eliminates the need for experimenters to place each experimental accessory in a specific location; they only need to place them on the operating table. This reduces the operational requirements for experimenters, making it easier for them to use the pipetting workstation and improving their operational efficiency, thus increasing the overall efficiency of the pipetting workstation.

[0070] The method provided in this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0071] This application provides a method for generating an experimental control program for a pipetting workstation, such as... Figure 2 As shown, the specific steps include the following:

[0072] Step 21: Obtain an image of the operating table of the pipetting workstation by taking a picture.

[0073] In this embodiment of the application, a camera can be installed on the pipetting workstation to capture images of the workstation's operating table, including taking photos and recording videos.

[0074] The camera's installation position can be flexibly set according to the actual application needs, as long as it can capture the operating table. It can be installed on the side of the pipetting workstation's casing, or preferably on the top of the casing, to capture the operating table downwards. This allows for a more accurate display of the location of each experimental accessory from the image, facilitating subsequent accurate identification. Alternatively, it can be installed on the pipetting arm, allowing the camera to be moved by controlling the movement of the pipetting arm, thus enabling a more comprehensive capture of the operating table area.

[0075] In this embodiment of the application, the area on the operating table used for placing experimental accessories can be divided into multiple sub-areas in advance, and images can be taken for each sub-area to obtain an operating table image corresponding to that sub-area, that is, multiple operating table images corresponding one-to-one with the multiple sub-areas.

[0076] Furthermore, the location of each sub-area on the control panel can be predetermined and set in advance.

[0077] When taking images of each sub-region, a corresponding camera can be set up for each sub-region to take pictures, and the resulting console image is used as the console image of that sub-region.

[0078] Alternatively, a movable camera can be installed on the pipetting workstation. When taking images of each sub-area, the camera first moves to the corresponding position of the sub-area and takes the picture. The resulting image of the worktable is then used as the worktable image of that sub-area.

[0079] Step 22: Perform experimental accessory recognition on the acquired workbench image.

[0080] In this embodiment of the application, the types of experimental accessories placed on the operating table and the positions of each experimental accessory on the operating table can be identified.

[0081] Furthermore, the dimensions of each experimental accessory can be identified, as well as the position of each component on each experimental accessory within its respective experimental accessory.

[0082] In this embodiment of the application, the identified position information can be represented using coordinates of a coordinate system established based on the operating table. For example, for a square experimental accessory, the position can be represented based on the coordinates of the four vertices of the experimental accessory; for a circular experimental accessory, the position can be represented based on the coordinates of the center of the experimental accessory; and for an irregularly shaped experimental accessory, the position can be represented based on the coordinates of a specific component on the experimental accessory.

[0083] Since the shape and size of certain types of experimental attachments are determined and known, their positions can be represented solely based on the coordinates of specific points on the experimental attachment, such as the coordinates of a vertex of a square experimental attachment or the coordinates of the center of a circular experimental attachment.

[0084] In this step, when the acquired workbench image includes multiple workbench images that correspond one-to-one with multiple sub-regions, experimental attachment identification can be performed on each of the multiple workbench images. Furthermore, the position of each sub-region on the workbench can be used as the position of the experimental attachment placed on that sub-region on the workbench. For example, the shape of each sub-region is defined, such as square or circle, and the position of a specific point on each sub-region represents the position of that sub-region. The specific point can be the vertex of a square sub-region or the center of a circular sub-region, etc. When a square experimental attachment is placed in a square sub-region, and a vertex of the square experimental attachment is aligned with the vertex of the square sub-region as the specific point, the position of that specific point in the square sub-region can be directly used as the position of the square experimental attachment on the workbench.

[0085] The acquired workbench images include multiple workbench images that correspond one-to-one with multiple sub-regions. Experimental attachment recognition can be performed on each of these multiple workbench images to obtain the position of the experimental attachment placed in each sub-region within that sub-region. This position is then used as the position of the experimental attachment placed in that sub-region on the workbench. For example, a coordinate system can be established for each sub-region, and the coordinates of the experimental attachment placed in that sub-region can be identified within that coordinate system. The sub-region number and the coordinates can then be used to represent the position of the experimental attachment placed in that sub-region on the workbench.

[0086] In this embodiment, the pipetting workstation can use various known specifications of operating tables, such as 96-well plates and 384-well plates. Different operating tables have different area distributions. In this step, targeted position identification can be performed based on the known area distribution of the operating tables in order to improve the accuracy of position identification.

[0087] In this step, the identification method for the types of accessories used in identifying experimental accessories can include the following two methods:

[0088] First identification method:

[0089] A correspondence is pre-created between the types of known experimental attachments and their corresponding attachment features. When it is necessary to identify experimental attachments from the workbench image, firstly, the image area covered by each placed experimental attachment is extracted from the workbench image. Then, features are extracted from the covered image area. Finally, the attachment type corresponding to the extracted attachment features is found from the pre-created correspondence and used as the identified attachment type of the experimental attachment.

[0090] In the first identification method, each attachment feature may include at least one of the following features:

[0091] Shape, color, size, number of holes, hole spacing, and label. Among these, shape, number of holes, and hole spacing can all indicate the specifications of the experimental accessories, and the label can be a barcode or an RFID (Radio Frequency Identification) label.

[0092] Depending on the needs of the experimental procedure, various experimental attachments may be used. Based on the structure of these attachments, as long as their characteristics can represent the type of experimental attachment, they can all be used in practical applications.

[0093] The second identification method:

[0094] A pre-trained machine learning model is used to identify experimental attachments in the workbench image. This machine learning model is trained based on known types of experimental attachment samples and the features of each attachment in the experimental attachment samples.

[0095] In the second identification method, each attachment feature may include at least one of the following features:

[0096] Shape, color, size, number of holes, hole spacing, and label. Among these, shape, number of holes, and hole spacing can all indicate the specifications of the experimental accessories, and the label can be a barcode or an RFID (Radio Frequency Identification) label.

[0097] In the second identification method, the machine learning model used can be any known type of machine learning model, which will not be described here with examples.

[0098] In the second identification method, the machine learning model used can also be used to identify the location of experimental attachments. It only requires training the model on experimental attachment samples with known locations.

[0099] Furthermore, the system first processes the workbench images using an integrated, multi-task-trained machine learning model. The model's training dataset contains image samples of various experimental attachments (such as 96-well plates, 384-well plates, various reagent bottles, magnetic racks, etc.). Each sample is not only labeled with the attachment type and overall bounding box, but also with fine annotations, including: the center coordinates of each well in the well plate, the center coordinates of the reagent bottle opening, the outline of the liquid surface inside the container, and key dimensions of the attachment (such as the well spacing).

[0100] Based on this model, the recognition process specifically includes:

[0101] Precise structure identification of multi-well plate attachments: For microwell plates, deep-well plates, etc., this multi-task model can output attachment type, overall bounding box, and initial region proposal for each well in parallel. The system then uses the initial region proposal to call a lightweight image processing unit (such as an optimized Hough circle transform algorithm) to perform sub-pixel-level precise localization of the center coordinates of each well, thereby obtaining its precise two-dimensional coordinates (X,Y) in the global coordinate system of the operating table. This hybrid architecture of "model proposal + algorithm refinement" ensures robustness of recognition while achieving higher coordinate localization accuracy, meeting the needs of pipetting operations. At the same time, by analyzing the texture, color, and brightness features of each well region image, the model can determine the current "state" of the well, such as: "empty", "containing liquid", or "covered by sealing film". For wells containing liquid, the liquid level height or relative volume (e.g., 25%, 50%, full, etc.) can be further estimated based on the proportion of the liquid imaging area.

[0102] The system identifies the status of container accessories: for reagent bottles, centrifuge tubes, etc., it identifies the precise center position of the bottle / tube opening as the target point for pipette insertion. More importantly, for transparent or semi-transparent containers, the system analyzes the specific light and dark contours formed by light refraction on the liquid surface in the image and matches them with a pre-set 3D geometric model of the container to estimate the absolute height or remaining volume of the liquid level inside the container. This process can be modeled and calculated using optical principles such as Snell's law.

[0103] Identifying accessory-specific parameters and functional module interfaces: The system identifies key dimensional parameters of accessories, such as the pitch of a multi-well plate, which is directly used to calculate the step size of the pipette arm within the plate. For functional accessories such as magnetic racks and temperature control modules, the system identifies the coordinates of their effective working area (such as the boundary of the magnet array).

[0104] Compared to traditional methods that only identify the overall type and border of accessories, this identification process directly outputs the precise coordinates of operable parts such as the orifice and bottle opening, as well as real-time physical information such as liquid level and whether there is liquid in the orifice. This lays an indispensable data foundation for subsequent high-precision and high-reliability automated operations. It significantly improves the automation limit and reliability of subsequent processes. The accurate component coordinates enable precise pipetting depth calculations, orifice spacing-based step calculations, and optimal path planning. Simultaneously, the perception of liquid level and orifice status allows the system to anticipate potential problems (such as insufficient reagent) before execution, fundamentally avoiding runtime failures caused by discrepancies in physical conditions.

[0105] In this embodiment of the application, the types of experimental accessories placed on the operating table can be various experimental accessories that may be used in the experimental process, including: functional modules of the pipetting workstation, experimental consumables, experimental reagents, etc.

[0106] Furthermore, functional modules may include: magnetic racks, temperature controllers, oscillators, temperature-controlled oscillators, PCR instruments, fluorescence quantitative modules, and other detection modules, as well as reagent racks of different specifications;

[0107] Experimental consumables may include: pipette tip boxes of different sizes, pipette tips, deep well plates, PCR plates, ELISA plates, cell culture plates, consumable caps (e.g., caps for cell culture plates), reagent troughs, reagent bottles, etc.

[0108] Experimental reagents may include reagents of different colors, such as black or brown magnetic beads.

[0109] In this embodiment of the application, the identified location information may also include the liquid surface position information of the liquid contained in the experimental accessory, that is, it can indicate the sample volume of the contained liquid, and accordingly, it can be used to generate the corresponding experimental control program.

[0110] Step 23: Display the recognition results on the human-computer interaction interface.

[0111] In this step, the displayed identification results may include the type and location of each identified experimental accessory, as well as the dimensions of each accessory and the location of each component within that accessory. The display interface not only presents the accessory types and locations in a list format but also overlays them graphically onto the workbench image. For example, different colored dots mark each identified orifice and its status (empty, liquid present), lines mark the liquid level of the reagent bottle, and the measured key dimensional parameters are displayed.

[0112] Step 23 is optional. It allows for human intervention to further ensure the accuracy of the subsequently generated experimental control procedures.

[0113] Step 24: Determine whether the user input indicates that the recognition result is correct or incorrect. If the recognition result is correct, proceed to step 26; if the recognition result is incorrect, proceed to step 25.

[0114] Step 25: After receiving the user's input indicating that the recognition result is incorrect, display the recognition result correction option on the human-computer interaction interface, and receive the user's input of the corrected recognition result.

[0115] In this step, users can use the human-computer interaction interface to correct the incorrectly identified content and input the corrected identification result, including the type, location, and size of the experimental attachment, as well as the location of the components on the experimental attachment within the corresponding experimental attachment.

[0116] Step 26: Determine the experimental procedure to be executed based on the human-computer interaction interface displayed by the user.

[0117] In this step, the human-computer interaction interface can be shown to the user, which includes various experimental procedures that the pipetting workstation can perform, for the user to choose from.

[0118] After the user makes a selection, the selected experimental procedure is determined and used as the experimental procedure for generating the experimental control program.

[0119] Step 27: Based on the current identification results of each experimental attachment, generate an experimental control program that matches the experimental process according to the required experimental process.

[0120] If the recognition result has not been corrected by the user, the current recognition result is the recognition result obtained in step 22 above. If it has been corrected by the user, the current recognition result is the recognition result corrected by the user in step 25 above.

[0121] In this step, based on the identification items included in the identification results, an experimental control program can be generated accordingly. For example, based on the type and location of each identified experimental accessory, the size of each experimental accessory, and the location of each component on each experimental accessory, an experimental control program matching the experimental process can be generated according to the experimental process to be executed.

[0122] In this step, during the process of generating the experimental control program, the current experimental control program corresponding to the experimental procedure to be executed can be obtained first.

[0123] Then, according to the attachment type and location of each experimental attachment in the current identification results, the position parameters representing each experimental attachment in the current experimental control program are changed to represent the location of each experimental attachment, thus obtaining an experimental control program that matches the experimental process.

[0124] If the identification results also include the dimensions of each experimental accessory and the location of each component on each experimental accessory, the parameters representing the dimensions of each experimental accessory and the location of each component on each experimental accessory in the current experimental control program will be changed accordingly.

[0125] In this step, based on the identification results, relevant parameters in the current experimental control program can be changed according to the needs of the experimental procedure. For example, based on the type and size of the experimental accessories, the specifications of the pipette tips used in some specific pipetting operations in the experimental procedure, as well as parameters such as the volume of each pipetting, can be adjusted.

[0126] Alternatively, the above embodiments of this application can be implemented by generating an experimental control program that matches the experimental procedure through the following process:

[0127] a. Pre-verification of experimental procedures and resource mapping.

[0128] The system maintains an "experimental procedure rule base," with each procedure corresponding to a set of logical rules. For example, the rules for the "nucleic acid extraction" procedure might be defined as follows: "Sample source plate (type: 96-well deep plate)", "Magnetic bead reagent (container type: reagent bottle, minimum liquid volume requirement: 5mL)", "Washing buffer (container type: reagent bottle)", etc.

[0129] The rule base stores the hardware configuration templates for each experimental procedure in a structured data format (such as JSON or XML). For example, the template for the "Nucleic Acid Extraction" procedure might be defined as: Required Resource List: [{Name: "Sample Source Plate", Type: "96-Well Deep Plate", Minimum Quantity: 1},

[0130] {Name: "Magnetic Bead Reagent", Type: "Reagent Bottle", Minimum Volume Requirement: "5mL"}...].

[0131] Before generating the program, the system automatically compares and verifies the depth recognition results from step 22 with the rules of the selected process.

[0132] The verification content includes: 1) the existence and type matching of attachments; 2) the compliance of key status (such as whether the liquid level in the reagent bottle meets the minimum requirement); 3) the mapping of resource locations (such as mapping the "sample source plate" in the process logic to a specific 96-well plate that has been identified).

[0133] If the verification fails, the system will generate a highlighted error report on the interactive interface (e.g., "Error: The estimated volume of the magnetic bead reagent bottle (located at position A5) is 3 mL, which is lower than the required minimum of 5 mL"), and pause the program to guide the user to replenish the reagent or adjust the layout.

[0134] b. Fine-grained generation of program instructions and dynamic parameter adaptation. The system only begins assembling the final executable control instruction sequence after compliance verification has passed.

[0135] Coordinate and motion parameter precision: The source and target coordinates in all liquid transfer commands are directly derived from the precise coordinates identified in step 22 (such as the center coordinates of the source reagent bottle opening and the center coordinates of the target orifice). The movement trajectory (XYZ coordinates) of the pipette is thus directly determined.

[0136] Dynamic calculation of operating parameters: Based on the identified liquid level in the source container, the system dynamically calculates the Z-axis depth of the pipette tip during the aspiration operation, ensuring that the tip is submerged within a safe distance (e.g., 2mm) of the liquid surface. It may also adaptively adjust the aspiration speed according to the liquid level (reducing the speed at low liquid levels). Based on the identified multi-well plate type (e.g., a 384-well plate), the system automatically adopts pipette arm stepping motion parameters that match the well spacing of the plate.

[0137] c. Global optimization of pipetting path: For complex processes involving multiple source and destination locations, after generating the basic instruction sequence, the system calls a path optimization algorithm (e.g., an approximate solution algorithm based on the Traveling Salesman Problem (TSP)). This algorithm takes the coordinates of all identified locations to be visited as input, aims to minimize the total idle travel time of the pipetting arm, and reorders or merges the operation sequence to generate a time-optimal or suboptimal execution sequence.

[0138] d. Program simulation and output: The generated final control program can first be graphically simulated in the software's virtual environment. After the user confirms that there are no errors, it can be sent to the pipetting workstation for execution.

[0139] By introducing an experimental procedure rule base for compliance verification and performing dynamic parameter adaptation and path optimization based on deep learning results, pre-verification of the experimental procedure was achieved, greatly improving the success rate and effectively avoiding the waste of precious samples and reagents. The generated program is no longer a script with fixed parameters. "Dynamic adaptation" of the control program was realized, enhancing system robustness. Path optimization algorithms intelligently sort multiple liquid transfer operations, significantly reducing the idle travel distance and time of the pipette arm during execution.

[0140] Based on the same inventive concept, and according to the pipetting workstation experimental control program generation method provided in the above embodiments of this application, another embodiment of this application also provides a pipetting workstation experimental control program generation device, the structural schematic diagram of which is shown below. Figure 3 As shown, it specifically includes:

[0141] Image acquisition module 31 is used to acquire images of the operating table of the pipetting workstation obtained by taking pictures of the operating table;

[0142] Image recognition module 32 is used to identify experimental accessories in the image of the operating table, and to obtain the accessory type of each experimental accessory placed on the operating table, as well as the position of each experimental accessory on the operating table.

[0143] The program generation module 33 is used to generate an experimental control program that matches the experimental procedure according to the identified accessory types and locations of each experimental accessory. The experimental control program is used to control the pipetting arm of the pipetting workstation to operate each experimental accessory on the operating table to complete the experimental procedure.

[0144] Furthermore, the area on the operating table used for placing experimental accessories is divided into multiple sub-areas;

[0145] The image acquisition module 31 is specifically used to acquire multiple operating table images obtained by taking pictures of the operating table of the pipetting workstation. The multiple operating table images correspond one-to-one with the multiple sub-regions, and each operating table image is an image of the corresponding sub-region.

[0146] The image recognition module 32 is specifically used to identify experimental attachments in the multiple workbench images respectively, to obtain the attachment type of the experimental attachments placed on each sub-area of ​​the workbench, and the position of each sub-area on the workbench is used as the position of the experimental attachments placed on that sub-area on the workbench, or the position of the experimental attachments placed on each sub-area is used as the position of the experimental attachments placed on that sub-area on the workbench.

[0147] Furthermore, the image recognition module 32 is specifically used to identify experimental attachments in the image of the operating table, and to obtain the attachment type of each experimental attachment placed on the operating table, the position of each experimental attachment on the operating table, the size of each experimental attachment, and the position of each component on each experimental attachment.

[0148] The program generation module 33 is specifically used to generate an experimental control program that matches the experimental process according to the experimental process to be executed, based on the identified types and locations of the experimental accessories, the dimensions of the experimental accessories, and the locations of the components on the experimental accessories.

[0149] Furthermore, the program generation module 33 is specifically used to obtain the current experimental control program corresponding to the experimental process to be executed; according to the identified attachment types and locations of each experimental attachment, the position parameters representing each experimental attachment in the current experimental control program are changed to represent the locations of each experimental attachment, thereby obtaining an experimental control program that matches the experimental process.

[0150] Furthermore, the program generation module 33 is also used to determine the experimental process to be executed selected by the user based on the human-computer interaction interface before generating an experimental control program that matches the experimental process according to the experimental process to be executed based on the identified types and locations of the experimental attachments.

[0151] Furthermore, the program generation module 33 is also used to display the type and location of each identified experimental accessory on the human-computer interaction interface after the image recognition module 32 performs experimental accessory recognition on the operating table image; after receiving a user input indicating that the recognition result is correct, it executes the step of generating an experimental control program matching the experimental process according to the required experimental process based on the identified accessory type and location of each experimental accessory; after receiving a user input indicating that the recognition result is incorrect, it displays a recognition result correction option on the human-computer interaction interface to receive the corrected recognition result input by the user.

[0152] The program generation module 33 is specifically used to generate an experimental control program that matches the experimental process according to the experimental process to be executed, based on the attachment type and location of each experimental attachment represented by the corrected identification result.

[0153] Furthermore, the image recognition module 32 is specifically used to identify experimental attachments in the operation table image using a pre-trained machine learning model. The machine learning model is trained based on known types of experimental attachment samples and the attachment features of each of the experimental attachment samples.

[0154] The features of each accessory include at least one of the following features:

[0155] Shape, color, size, number of holes, hole spacing, and label.

[0156] The functions of the above modules can be corresponding to Figure 1 or Figure 2 The corresponding processing steps in the process shown will not be repeated here.

[0157] The pipetting workstation experimental control program generation device provided in the embodiments of this application can be implemented by a computer program. Those skilled in the art should understand that the above-described module division method is only one of many module division methods. Whether it is divided into other modules or not divided into modules, as long as the pipetting workstation experimental control program generation device has the above-described functions, it should be within the protection scope of this application.

[0158] This application also provides an electronic device, such as... Figure 4As shown, it includes a processor 41 and a machine-readable storage medium 42, the machine-readable storage medium 42 storing machine-executable instructions that can be executed by the processor 41, the processor 41 being prompted by the machine-executable instructions to implement any of the above-described methods for generating experimental control programs for pipetting workstations.

[0159] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-described methods for generating experimental control programs for pipetting workstations.

[0160] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the above-described pipetting workstation experimental control program generation methods.

[0161] The machine-readable storage medium in the aforementioned electronic device may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0162] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0163] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of devices, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0164] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0165] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating an experimental control program for a pipetting workstation, characterized in that, include: Acquire images of the control panel of the pipetting workstation by taking photographs; The experimental attachments are identified by performing experimental attachment recognition on the operation table image to obtain the attachment type of each experimental attachment placed on the operation table, as well as the position of each experimental attachment on the operation table; Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the experimental procedure is generated according to the required experimental procedure. The experimental control program is used to control the pipetting arm of the pipetting workstation to operate each experimental accessory on the operating table to complete the experimental procedure.

2. The method as described in claim 1, characterized in that, The area on the operating table used for placing experimental accessories is divided into multiple sub-areas; The acquisition of the image of the operating table obtained by photographing the operating table of the pipetting workstation includes: Multiple images of the operating table of the pipetting workstation are acquired by taking pictures of the operating table. The multiple operating table images correspond one-to-one with the multiple sub-regions, and each operating table image is an image of the corresponding sub-region. The step of identifying experimental attachments in the image of the workbench to obtain the attachment types of each experimental attachment placed on the workbench and the position of each experimental attachment on the workbench can be performed using any of the following methods: Experimental attachments are identified in the multiple workbench images to obtain the attachment types of experimental attachments placed on each sub-region of the workbench. The position of each sub-region on the workbench is taken as the position of the experimental attachments placed on that sub-region on the workbench. Alternatively, experimental attachment recognition can be performed on the multiple workbench images to obtain the attachment type of the experimental attachment placed in each sub-region and the location of the experimental attachment placed in each sub-region on that sub-region. Combined with the preset position of that sub-region on the workbench, the location of the experimental attachment placed in that sub-region on the workbench can be determined.

3. The method as described in claim 1, characterized in that, The step of identifying experimental attachments in the image of the workbench to obtain the attachment types of each experimental attachment placed on the workbench and the position of each experimental attachment on the workbench includes: The experimental attachments are identified in the image of the operating table to obtain the attachment type of each experimental attachment placed on the operating table, the position of each experimental attachment on the operating table, the size of each experimental attachment, and the position of each component on each experimental attachment. Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the required experimental procedure is generated, including: Based on the identified types and locations of the experimental attachments, the dimensions of the experimental attachments, and the locations of the components on each experimental attachment, an experimental control program matching the experimental procedure to be executed is generated.

4. The method as described in claim 1, characterized in that, Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the required experimental procedure is generated, including: Obtain the current experimental control program corresponding to the experimental procedure to be executed; Based on the identified accessory types and locations of each experimental accessory, the position parameters representing each experimental accessory in the current experimental control program are changed to represent the locations of each experimental accessory, thereby obtaining an experimental control program that matches the experimental process.

5. The method as described in claim 1, characterized in that, Before generating an experimental control program that matches the experimental procedure according to the required experimental procedure based on the identified accessory types and locations of each experimental accessory, the method further includes: Determine the experimental procedures to be performed based on the user's selection of the displayed human-computer interaction interface.

6. The method as described in claim 1, characterized in that, After identifying experimental attachments in the control panel image, and before generating an experimental control program matching the required experimental procedure based on the identified attachment types and locations, the method further includes: The types and locations of the identified experimental accessories are displayed on the human-computer interaction interface. After receiving a user's input indicating that the recognition result is correct, the steps are as follows: based on the recognition of the types and locations of the various experimental attachments, generate an experimental control program that matches the experimental process according to the required experimental process. After receiving a user's input indicating an error in the recognition result, the system displays a recognition result correction option on the human-computer interaction interface, which is used to receive the corrected recognition result from the user. Based on the identified accessory types and locations of each experimental accessory, an experimental control program matching the required experimental procedure is generated, including: Based on the corrected identification results, which indicate the attachment types and locations of each experimental attachment, an experimental control program matching the experimental procedure to be executed is generated.

7. The method as described in claim 1, characterized in that, The experimental attachment recognition of the operation table image includes: A pre-trained machine learning model is used to identify experimental attachments in the workbench image. The machine learning model is trained based on known types of experimental attachment samples and the features of each attachment in the experimental attachment samples. The features of each accessory include at least one of the following features: Shape, color, size, number of holes, hole spacing, and label.

8. A device for generating experimental control programs for a pipetting workstation, characterized in that, include: The image acquisition module is used to acquire images of the operating table of the pipetting workstation obtained by taking pictures of the operating table; An image recognition module is used to identify experimental attachments in the image of the operating table, and to obtain the attachment type of each experimental attachment placed on the operating table, as well as the position of each experimental attachment on the operating table. The program generation module is used to generate an experimental control program that matches the experimental procedure according to the identified accessory types and locations of each experimental accessory. The experimental control program is used to control the pipetting arm of the pipetting workstation to operate each experimental accessory on the operating table to complete the experimental procedure.

9. An electronic device, characterized in that, The method includes a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor, the processor being prompted by the machine-executable instructions to perform the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images