Method and device for controlling experiment operation based on experiment process

Abstract

The invention discloses a method and device for controlling experiment operation based on an experiment process, and the method comprises the steps: receiving an experiment process control instruction which comprises a plurality of operation instructions formed according to operation steps; executing the current operation instruction: determining a working area for executing the current operation instruction, and splitting the current operation instruction into executable action sequences according to the spatial layout and the operation sequence of the working area; controlling the operation unit to sequentially execute each action in the current action sequence; and after the current operation instruction is completed, executing the next operation instruction until all the operation instructions are completed. All operations in the complete experiment process can be automatically completed, manual intervention is not needed in the whole process, and automation of chemical experiments is comprehensively achieved from the action level. The investment of labor cost can be effectively saved, the number of reactions which can be completed in a single laboratory can be increased, and meanwhile, potential dangers caused by some chemical reactions to the health of workers can be avoided.

Description

Technical Field

[0001] The present invention relates to the field of automatic operation of chemical experiments, and in particular to a method, device, robot and computer-readable medium for controlling experimental operations based on experimental procedures. Background Art

[0002] At present, most traditional laboratories require experimenters to perform experiments for a long time, and most experiments are highly repetitive. Manual experiments are inefficient and costly. With the development of science and technology, the application of robots in chemical laboratories has become a hot direction in the field of robot applications. The automation of chemical laboratories has become increasingly important in scientific research and industrial applications. At present, the application of robots in chemical experiments is limited to a single experimental link or a single operation step (such as weighing, filtering, etc.). For a complete experimental process, partial participation of staff is still required. Therefore, it is urgent to design an experimental operation method that can control the complete experimental process, so as to automatically complete each operation in the complete experimental process without manual intervention and fully realize experimental automation. Summary of the invention

[0003] In view of this, the present invention provides a method, device, robot and computer-readable medium for controlling experimental operations based on experimental processes, in order to at least partially solve at least one of the above-mentioned technical problems.

[0004] In order to solve the above technical problems, the first aspect of the present invention proposes a method for controlling experimental operations based on an experimental process, the method comprising: Receiving an experimental process control instruction, wherein the experimental process control instruction includes: a plurality of operation instructions composed of operation steps; the operation instruction at least includes an operation action; Executing the current operation instruction, including: determining a work area for executing the current operation instruction, and dividing the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence; controlling the operation unit to sequentially execute each action in the current action sequence; After completing the current operation instruction, execute the next operation instruction until all operation instructions are completed.

[0005] According to a preferred embodiment of the present invention, the spatial layout of the workspace includes: objects in the workspace, relative positions between objects, and postures of objects.

[0006] According to a preferred embodiment of the present invention, determining a work area for executing the current operation instruction, and splitting the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence includes: Identify a work area for executing a current operation instruction and control the operation unit to go to the work area; The control operation unit acquires the image information of the work area, and identifies the spatial layout in the work area according to the image information; The current operation instruction is split into executable action sequences according to the spatial layout and operation sequence in the workspace.

[0007] According to a preferred embodiment of the present invention, the operation sequence includes at least one of: dependency between actions, execution time, and execution sequence. Before splitting the current operation instruction into executable action sequences according to the spatial layout in the workspace and the operation sequence, the method further includes: The current operation instruction is input into the splitting model, and at least one of the dependency relationship, execution time, and execution sequence between actions in the current operation instruction is output.

[0008] According to a preferred embodiment of the present invention, the current operation instruction is split into executable action sequences according to the spatial layout, object features and operation sequence in the workspace.

[0009] According to a preferred embodiment of the present invention, the control operation unit sequentially executes each action in the current action sequence including: Controlling the operation unit to execute the current action in the current action sequence includes: determining the execution object, execution posture and execution path of the current action in the current action sequence; controlling the operation unit to reach the execution object according to the execution path and adjust to the execution posture before executing the current action; After the current action is executed, the control operation unit executes the next action in the current action sequence until all actions in the current action sequence are completed.

[0010] According to a preferred embodiment of the present invention, the method further comprises: Collect RGB images and depth images of the work area in real time; Identify the real-time spatial layout of the workspace according to the RGB image and the depth image; Determine whether to adjust the execution posture and / or execution path according to the real-time spatial layout, and adjust the execution posture and / or execution path according to the determination result.

[0011] In order to solve the above technical problems, the second aspect of the present invention provides a device for controlling experimental operations based on an experimental process, characterized in that the device comprises: A receiving module, used for receiving an experimental process control instruction, wherein the experimental process control instruction includes: a plurality of operation instructions composed of operation steps; the operation instruction at least includes an operation action; The first execution module is used to execute the current operation instruction, including: determining the work area for executing the current operation instruction, and splitting the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence; and controlling the operation unit to sequentially execute each action in the current action sequence; The second execution module is used to execute the next operation instruction after completing the current operation instruction until all operation instructions are completed.

[0012] In order to solve the above technical problems, the third aspect of the present invention provides an experimental operation robot based on experimental process control, comprising: Processor; and A memory storing computer executable instructions, which when executed cause the processor to perform the above method.

[0013] In order to solve the above technical problem, the fourth aspect of the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the above method is implemented.

[0014] The present invention can receive pre-generated experimental process control instructions, and the experimental process control instructions include: a plurality of operating instructions composed of operating steps; by determining the work area for executing the current operating instruction, and splitting the current operating instruction into an executable action sequence according to the spatial layout of the work area and the operation sequence; controlling the operation unit to execute each action in the current action sequence in sequence; automatically executing the current operating instruction, and after completing the current operating instruction, executing the next operating instruction according to the operating steps until all operating instructions are completed. Thus, each operation in the complete experimental process is automatically completed, and the whole process does not require manual intervention, and the automation of chemical experiments is fully realized from the action level. Compared with the prior art, it can effectively save the investment of labor costs, increase the number of reactions that can be completed in a single laboratory, and also avoid the potential dangers of some chemical reactions to the health of workers. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to make the technical problems solved by the present invention, the technical means adopted and the technical effects achieved more clearly, the specific embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, it should be noted that the drawings described below are only drawings of exemplary embodiments of the present invention, and those skilled in the art can obtain drawings of other embodiments based on these drawings without creative work.

[0016] Figure 1 It is a flow chart of a method for controlling experimental operation based on an experimental process of the present invention; Figure 2Schematic diagram of an RGB image of a weighing area captured by an embodiment of the present invention; Figure 3 It is a schematic diagram of the structural framework of an experimental operation device based on experimental process control; Figure 4 is a structural block diagram of an exemplary embodiment of an experimental operation robot controlled based on an experimental process according to the present invention; Figure 5 is a schematic diagram of an embodiment of a computer readable medium of the present invention. DETAILED DESCRIPTION

[0017] The exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. Although each exemplary embodiment can be implemented in a variety of specific ways, it should not be understood that the present invention is limited to the embodiments set forth herein. On the contrary, these exemplary embodiments are provided to make the content of the present invention more complete and to facilitate the comprehensive communication of the inventive concept to those skilled in the art.

[0018] Under the premise of conforming to the technical concept of the present invention, the structure, performance, effect or other characteristics described in a specific embodiment may be combined with one or more other embodiments in any appropriate manner.

[0019] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0020] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to actual conditions without departing from the main purpose of the present invention.

[0021] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0022] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0023] Figure 1 Schematic diagram of a method for controlling experimental operations based on an experimental process provided by the present invention; Figure 1 As shown, the method includes: S1, receiving experimental process control instructions; In this embodiment, the experimental process control instruction refers to the process instruction corresponding to completing a complete experiment from beginning to end. It may include: a plurality of operation instructions composed of operation steps, and these operation instructions constitute the process instruction corresponding to a complete experiment; the operation instruction at least includes an operation action. For example, in the process of setting up a reaction device, a complete setting process can be completed by multiple operation actions. Further, the operation instruction may also include: raw material information and / or reaction conditions. Among them: raw material information may include: raw material name, raw material weight, etc., and reaction conditions may include: pressure value, temperature value, pH value, etc. For example: in a complete experimental process control instruction, in the chemical reaction preparation stage, it includes an operation instruction of "weighing 5ml of water in the weighing area".

[0024] In this embodiment, the experimental process control instruction can be manually edited and input, or it can be an experimental process control instruction generated after making decisions and judgments at the chemical experimental scheme level based on the target chemical molecular formula input by the user. For example, in one example, the experimental process control instruction can be generated after making decisions and judgments at the chemical experimental scheme level based on the target chemical molecular formula input by the user. In this step, the experimental process control instruction is received, and according to the present invention, the experimental process control instruction is disassembled into executable actions to form a corresponding action sequence, and the control operation unit (corresponding hardware) sequentially completes the individual actions in the action sequence, thereby automatically completing the chemical experiment operation.

[0025] S2, executing the current operation instruction, including: determining a work area for executing the current operation instruction, and splitting the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence; and controlling the operation unit to sequentially execute each action in the current action sequence; Considering that the execution order of actions in experimental operations is affected by the spatial layout of objects (experimental instruments, experimental raw materials, etc.) and the order of operations; for example: in two actions with the same order of operations, the position of the experimental instrument used in action 1 is closer to the operation unit than the position of the experimental instrument used in action 2, then action 1 can be executed first. This embodiment splits the current operation instruction into an executable action sequence according to the spatial layout of the workspace and the order of operations, wherein: the spatial layout of the workspace includes: objects in the workspace, relative positions between objects, object postures, etc. Then the workspace for determining the execution of the current operation instruction and splitting the current operation instruction into executable action sequences according to the spatial layout of the workspace and the order of operations may include: S21, identifying a work area where a current operation instruction is executed and controlling the operation unit to go to the work area; For example, the operation instructions and the corresponding workspaces can be collected in advance to train the workspace recognition model, and then the current operation instruction can be input into the trained workspace recognition model to output the workspace that executes the current operation instruction. The workspace recognition model can specifically adopt a neural network model, etc. Among them: the operation unit can be a hardware structure such as a robot or a mechanical arm that performs a specific action.

[0026] Taking the current operation instruction of "weigh 5 ml of water in the weighing area" as an example, in this step, "weigh 5 ml of water in the weighing area" can be input into the workspace recognition model, and the output workspace is "weighing area", then the robot can be started to go to the "weighing area".

[0027] S22, controlling the operating unit to obtain image information of the work area, and identifying the spatial layout in the work area according to the image information; In this embodiment, the image information may include: RGB image and depth image, and the operation unit may be provided with a shooting device capable of simultaneously acquiring RGB image and depth image, such as: depth camera, binocular camera, etc. Then, this step may control the camera to shoot multiple RGB images and depth images of the work area, and input them into the trained Visual Language Models (VLM) to identify objects in the work area, relative positions between objects and object postures, thereby identifying the spatial layout in the work area.

[0028] Taking the current operation instruction of "weighing 5 ml of water in the weighing area" as an example, in this step, the robot's external camera can be used to take a picture of the weighing area desktop to obtain the RGB screen image of the weighing area (such as Figure 2) and the depth image of the weighing area. The VLM identifies 7 liquid raw material bottles, sample tubes, syringes, funnels and other objects within the camera range of the weighing area, and locates the position of each object. Since the liquid raw material bottles have similar appearances, they can be further identified by the QR code on the liquid raw material bottles. The camera can further identify the QR code information on the liquid raw material bottles to determine the relative positions of the liquid raw material bottles. For example, the camera can identify the QR code information on each raw material bottle and know that the "water raw material bottle" is located at the second right position in the raw material bottle area.

[0029] S23. Split the current operation instruction into executable action sequences according to the spatial layout and operation sequence in the workspace.

[0030] The operation sequence is used to describe the execution sequence between actions, and may include: at least one of the dependency relationship between actions, execution time, and execution sequence. Before this step, the operation sequence may be determined based on the current operation instruction. Exemplarily, the operation instruction and the dependency relationship, execution time, and execution sequence between the actions contained in the operation instruction may be collected in advance to train the split model, and the current operation instruction may be input into the trained split model to output at least one of the dependency relationship, execution time, and execution sequence between the actions in the current operation instruction.

[0031] Furthermore, the spatial layout, operation sequence, and corresponding action sequence in the workspace can be collected in advance as training data to train the action training split model, and then the spatial layout and operation sequence in the workspace of the current operation instruction can be input into the trained action split model to output the executable current action sequence. Among them: the action sequence consists of multiple executable actions, and the executable actions may include: grabbing, placing, inserting, unplugging, pushing / pulling, etc.

[0032] For example, Figure 2 The spatial layout of the weighing area and the operation sequence input action splitting model, the output action sequence is: 4.1- The right arm grabs the "water" raw material bottle; 4.2-Transfer to the left arm; 4.3- Pull out the cork with the right arm; 4.4- Place the bottle cork on the table with your right arm; 4.5- Place the raw material bottle on the table with the left arm; 4.6-The right arm grabs the sample tube from the test tube rack and places it in a fixed position; 4.7-Grab the syringe with the right arm; 4.8-The right arm inserts the needle into the raw material bottle and fixes the position; 4.9-Use your left arm to pull the needle tube to the position corresponding to the volume of 5ml; 4.10-The right arm moves the syringe until the needle is inserted into the sample tube; 4.11-Use your left arm to push the needle to the position corresponding to the liquid being drained; 4.12-Use your right arm to place the needle on the table; 4.13-The right arm grabs the sample tube and puts it back into the test tube rack; 4.14-Grab the bottle cork with the right arm; 4.15- Grab the sample bottle with the left arm; 4.16-The right arm inserts the stopper into the sample bottle; 4.17- The left arm transfers the sample bottle to the right arm; 4.18-The right arm returns the sample bottle to its original position.

[0033] In a preferred example, after obtaining the spatial layout in the workspace, the object features in the workspace can be further obtained, such as object shape, size, etc., and the current operation instruction can be split into executable action sequences based on the pre-trained model according to the spatial layout, object features and operation sequence in the workspace. Among them, object features may include object shape, size, whether there is a QR code and QR code information, components, etc. It can be obtained through a picture taken by an external camera through a visual recognition algorithm.

[0034] In one example, the controlling operation unit sequentially executing each action in the current action sequence may include: S201, controlling the operation unit to execute the current action in the current action sequence, including: determining the execution object, execution posture and execution path of the current action in the current action sequence; controlling the operation unit to reach the execution object according to the execution path, and after adjusting to the execution posture, executing the current action; Among them: the execution object refers to the object that can be operated, which may include: a cylinder, a raw material bottle, a syringe, etc. The execution posture may further include: the execution point and the execution angle of the execution object. The execution path is the route from the current position to the execution object.

[0035] Exemplarily, executable actions and corresponding execution objects, execution postures and execution paths may be collected in advance as training data to train the execution determination model. After the current action is input into the execution determination model, the execution object, execution posture and execution path are output.

[0036] For example, in the action sequence output in step S23, the current action is the first action, namely "the right arm grabs the water raw material bottle". Then this step inputs the execution action of "the right arm grabs the water raw material bottle" into the execution determination model, and the output result is as follows: 5.1- Execution objects: water raw material bottles, 5.2-The execution location is: bottle body position; 5.3-The execution path is: moving from the location of the water raw material bottle to the unfixed location (the executable area determined by the robot); 5.4-Control the robot's right arm to move to the position of the water raw material bottle (in conjunction with the real-time image), grab the body of the water raw material bottle, and move the water raw material bottle to the executable area above the desktop through the execution path.

[0037] Furthermore, in the actual workspace, the position and / or posture of some objects may change. In order to improve the accuracy of the execution posture and execution path, the execution posture and / or execution path can be adjusted according to the real-time spatial layout of the workspace. After the execution determines the model output execution object, execution posture and execution path, it is also possible to: control the camera of the operating unit to collect the RGB image and depth image of the workspace in real time; identify the real-time spatial layout of the workspace according to the RGB image and depth image; determine whether to adjust the execution posture and / or execution path according to the real-time spatial layout, and adjust the execution posture and / or execution path according to the judgment result. Specifically, the real-time spatial layout obtained by the camera collecting images during the movement can be input into the preset model. The preset model calculates and determines whether there are obstacles on the execution path and whether there is enough space to complete the action according to the real-time spatial layout. If there are obstacles or insufficient space, the preset model returns the information and replans the execution path based on the information.

[0038] S202: After the current action is executed, the control operation unit executes the next action in the current action sequence until all actions in the current action sequence are completed.

[0039] Since the execution posture of the operating unit changes during the execution of the action, the execution posture of the execution object can be monitored in real time during the execution of the current action. When the execution posture is consistent with the preset standard execution posture, it is determined that the current action is completed.

[0040] When the current action is completed, determine whether the current action is the last executable action in the current action sequence. If so, proceed to step S3; if not, obtain the next action from the current action sequence, and execute steps S21 to S202 in sequence to control the operation unit to execute the next action in the current action sequence until all actions in the current action sequence are completed. For example, when it is determined that the robot has completed the action in 4.1, start to execute the next action 4.2. And so on until all actions in 4 are completed.

[0041] S3. After completing the current operation instruction, execute the next operation instruction until all operation instructions are completed.

[0042] For example, if the current action is determined to be the last executable action in the current action sequence, indicating that the current operation instruction is completed, step S2 is executed repeatedly until all operation instructions are completed. Thus, each operation in the complete experimental process is automatically completed, and the entire process does not require manual intervention, and the automation of chemical experiments is fully realized from the action level.

[0043] It should be noted that the model provided in each step of the present invention can be based on a pre-trained model with some general and extensive capabilities, and the pre-trained model can be further trained on its corresponding training set to adjust the pre-trained model parameters to obtain a model that can be applied to a specific field in each step, so as to better complete the scenarios and tasks facing the model. In addition, the model can be optimized and trained by continuously collecting various types of data during the experiment (such as user feedback and experimental results, etc.) to improve the accuracy of the model.

[0044] Based on the above method for controlling experimental operation based on experimental process, the present invention also provides an experimental operation device based on experimental process. Figure 3 It is a schematic diagram of the structural framework of an experimental operation device based on experimental process control of the present invention, such as Figure 3 As shown, the device comprises: A receiving module 31 is used to receive an experimental process control instruction, and the experimental process control instruction includes: a plurality of operating instructions composed of operating steps; the operating instruction includes at least an operating action; a first execution module 32 is used to execute the current operating instruction, including: determining a working area for executing the current operating instruction, and splitting the current operating instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence; controlling the operation unit to execute each action in the current action sequence in sequence; optionally, the spatial layout of the working area includes: objects in the working area, relative positions between objects, and object postures.

[0045] The second execution module 33 is used to execute the next operation instruction after completing the current operation instruction until all operation instructions are completed.

[0046] In a specific implementation, the first execution module 32 includes: A first identification module, used for identifying a working area where a current operation instruction is executed and controlling the operation unit to go to the working area; A second recognition module is used to control the operation unit to obtain the image information of the work area, and recognize the spatial layout in the work area according to the image information; The splitting module is used to split the current operation instruction into executable action sequences according to the spatial layout and operation sequence in the work area. Preferably, the splitting module splits the current operation instruction into executable action sequences according to the spatial layout, object features and operation sequence in the work area.

[0047] In one example, the operation sequence includes at least one of a dependency relationship between actions, an execution time, and an execution sequence, and the device further includes: The sub-splitting module is used to input the current operation instruction into the splitting model and output at least one of the dependency relationship, execution time, and execution sequence between actions in the current operation instruction.

[0048] Furthermore, the first execution module 32 also includes: The first sub-execution module is used to control the operation unit to execute the current action in the current action sequence, including: determining the execution object, execution posture and execution path of the current action in the current action sequence; controlling the operation unit to reach the execution object according to the execution path, and after adjusting to the execution posture, execute the current action; The second sub-executing module is used to control the operating unit to execute the next action in the current action sequence after the current action is executed, until all actions in the current action sequence are completed.

[0049] In a preferred embodiment, the device further comprises: The acquisition module is used to acquire the RGB image and depth image of the working area in real time; A real-time recognition module, used to recognize the real-time spatial layout of the workspace according to the RGB image and the depth image; The adjustment module is used to determine whether to adjust the execution posture and / or execution path according to the real-time spatial layout, and adjust the execution posture and / or execution path according to the determination result.

[0050] Those skilled in the art will appreciate that the modules in the above device embodiments may be distributed in the device as described, or may be changed accordingly and distributed in one or more devices different from the above embodiments. The modules in the above embodiments may be combined into one module, or may be further split into multiple submodules.

[0051] The following describes an embodiment of the experimental operation robot based on experimental process control of the present invention, which can be regarded as an implementation of the method and device embodiments of the present invention in a physical form. The details described in the robot embodiment of the present invention should be regarded as a supplement to the above method or device embodiments; details not disclosed in the robot embodiment of the present invention can be implemented with reference to the above method or device embodiments.

[0052] Figure 4 is a structural block diagram of an exemplary embodiment of a robot according to the present invention. Figure 4 The robot shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0053] like Figure 4 As shown, the robot 400 of this exemplary embodiment is in the form of a general data processing device. The components of the robot 400 may include, but are not limited to: at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different electronic device components (including the storage unit 420 and the processing unit 410), a display unit 440, etc.

[0054] The storage unit 420 stores a computer-readable program, which may be a source program or a code of a read-only program. The program may be executed by the processing unit 410, so that the processing unit 410 performs the steps of various embodiments of the present invention. For example, the processing unit 410 may perform the following steps: Figure 1 Steps shown.

[0055] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 4201 and / or a cache storage unit 4202, and may further include a read-only storage unit (ROM) 4203. The storage unit 420 may also include a program / utility 4204 having a set (at least one) of program modules 4205, such program modules 4205 include but are not limited to: operating electronic devices, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0056] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0057] The robot 400 may also communicate with one or more external devices 300 (e.g., keyboard, display, network device, Bluetooth device, etc.) so that a user can interact with the robot 400 via these external devices 300, and / or the robot 400 can communicate with one or more other data processing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 450, and may also be performed through a network adapter 460 with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet). The network adapter 460 may communicate with other modules of the robot 400 via the bus 430. It should be understood that although Figure 5Not shown, other hardware and / or software modules may be used in the robot 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID electronic devices, tape drives, and data backup storage electronic devices, etc.

[0058] Figure 5 Schematic diagram of a computer readable medium embodiment of the present invention. Figure 5 As shown, the computer program can be stored on one or more computer-readable media. The computer-readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electronic device, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. When the computer program is executed by one or more data processing devices, the computer-readable medium is enabled to implement the above-mentioned method of the present invention, namely: receiving an experimental process control instruction, the experimental process control instruction includes: a plurality of operation instructions composed of operation steps; the operation instruction includes at least an operation action; executing the current operation instruction, including: determining a work area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the work area and the operation sequence; controlling the operation unit to execute each action in the current action sequence in sequence; after completing the current operation instruction, executing the next operation instruction until all operation instructions are completed.

[0059] Through the description of the above implementation modes, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation mode of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a data processing device (which can be a personal computer, a server, or a network device, etc.) to execute the above method according to the present invention.

[0060] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction-executing electronic device, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0061] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0062] In summary, the present invention can be implemented by a method, apparatus, electronic device or computer-readable medium that executes a computer program. In practice, a general-purpose data processing device such as a microprocessor or a digital signal processor (DSP) can be used to implement some or all of the functions of the present invention.

[0063] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for controlling experimental operations based on an experimental process, characterized in that: The method comprises: Receiving an experimental process control instruction, wherein the experimental process control instruction includes: a plurality of operation instructions composed of operation steps; the operation instruction at least includes an operation action; Executing the current operation instruction, including: determining a work area for executing the current operation instruction, and dividing the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence; controlling the operation unit to sequentially execute each action in the current action sequence; After completing the current operation instruction, execute the next operation instruction until all operation instructions are completed.

2. The method according to claim 1, characterized in that The spatial layout of the workspace includes: objects in the workspace, relative positions between objects, and postures of objects.

3. The method according to claim 1, characterized in that Determining the work area for executing the current operation instruction, and splitting the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence includes: Identify a work area for executing a current operation instruction and control the operation unit to go to the work area; The control operation unit acquires the image information of the work area, and identifies the spatial layout in the work area according to the image information; The current operation instruction is split into executable action sequences according to the spatial layout and operation sequence in the workspace.

4. The method according to claim 1, characterized in that: The operation sequence includes at least one of: dependency between actions, execution time, and execution sequence. Before splitting the current operation instruction into executable action sequences according to the spatial layout in the workspace and the operation sequence, the method further includes: The current operation instruction is input into the splitting model, and at least one of the dependency relationship, execution time, and execution sequence between actions in the current operation instruction is output.

5. The method according to claim 1, characterized in that The current operation instruction is split into executable action sequences according to the spatial layout, object characteristics and operation sequence in the workspace.

6. The method according to claim 1, characterized in that The control operation unit sequentially executes each action in the current action sequence, including: Controlling the operation unit to execute the current action in the current action sequence includes: determining the execution object, execution posture and execution path of the current action in the current action sequence; controlling the operation unit to reach the execution object according to the execution path and adjust to the execution posture before executing the current action; After the current action is executed, the control operation unit executes the next action in the current action sequence until all actions in the current action sequence are completed.

7. The method according to claim 5, characterized in that The method further comprises: Collect RBG images and depth images of the working area in real time; Identify the real-time spatial layout of the workspace according to the RGB image and the depth image; Determine whether to adjust the execution posture and / or execution path according to the real-time spatial layout, and adjust the execution posture and / or execution path according to the determination result.

8. A device for controlling experimental operations based on an experimental process, characterized in that: The device comprises: A receiving module, used for receiving an experimental process control instruction, wherein the experimental process control instruction includes: a plurality of operation instructions composed of operation steps; the operation instruction at least includes an operation action; The first execution module is used to execute the current operation instruction, including: determining the work area for executing the current operation instruction, and splitting the current operation instruction into executable action sequences according to the spatial layout of the work area and the operation sequence; and controlling the operation unit to sequentially execute each action in the current action sequence; The second execution module is used to execute the next operation instruction after completing the current operation instruction until all operation instructions are completed.

9. An experimental operation robot based on experimental process control, characterized in that: include: processor; as well as A memory storing computer executable instructions which, when executed, cause the processor to perform the method according to any one of claims 1-6.

10. A computer-readable storage medium, wherein: The computer-readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images