Method and apparatus for controlling experimental operations on the basis of experimental process

Abstract

Disclosed in the present invention are a method and apparatus for controlling experimental operations on the basis of an experimental process. The method comprises: receiving an experimental process control instruction, wherein the experimental process control instruction comprises a plurality of operation instructions formed according to operation steps; executing the current operation instruction, involving: determining a working area for executing the current operation instruction, splitting the current operation instruction into executable action sequences on the basis of the spatial layout of the working area and an operation sequence, and controlling an operation unit to sequentially execute various actions in the current action sequence; and after the current operation instruction is completed, executing the next operation instruction until all the operation instructions are completed. The present invention can automatically complete various operations in a complete experimental process without the need for manual intervention throughout the whole process, thereby fully realizing the automation of chemical experiments at an action level. Therefore, investment of labor costs can be effectively reduced, the number of reactions that can be completed in a single laboratory can be increased, and potential dangers of some chemical reactions to the health of workers can also be avoided.

Description

Method and device for controlling experimental operation based on experimental process TECHNICAL FIELD

[0001] The present application relates to the field of automatic operation of chemical experiments, in particular to a method and device for controlling experimental operation based on experimental process, a robot and a computer readable medium. BACKGROUND

[0002] Most of the traditional laboratories now require experimental personnel to perform experimental operations for a long time, and most of the experiments have strong repeatability, and manual experimental efficiency is low and the cost is high. With the development of science and technology, the application of robots in chemical laboratories has become a popular direction in the field of robot application, and the automation of chemical laboratories has become increasingly important in scientific research and industrial applications.

[0003] At present, the application of robots in chemical experiments is limited to a single experimental link, or a single operation step (such as weighing, filtering, etc.). For a complete experimental process, the participation of workers is still required, therefore, there is an urgent need to design an experimental operation method capable of controlling a complete experimental process to automatically complete each operation in the complete experimental process without human intervention and fully realize experimental automation. TECHNICAL PROBLEM

[0004] Therefore, the present application provides a method and device for controlling experimental operation based on experimental process, a robot and a computer readable medium, in order to at least partially solve at least one of the above technical problems. TECHNICAL SOLUTION

[0005] In order to solve the above technical problems, the present application provides a method for controlling experimental operation based on experimental process, which comprises:

[0006] receiving an experimental process control instruction, the experimental process control instruction comprising: a plurality of operation instructions composed according to operation steps; the operation instruction at least comprising an operation action;

[0007] executing the current operation instruction, comprising: determining the working area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence; controlling the operation unit to execute each action in the current action sequence in turn;

[0008] After completing the current operation instruction, the next operation instruction is executed until all operation instructions are completed.

[0009] According to a preferred embodiment of the present application, the spatial layout of the working area comprises: objects in the working area, relative positions between objects, and object postures.

[0010] According to a preferred embodiment of the present application, the method further comprises:

[0011] identifying the work area in which the current operation instruction is executed and controlling the operation unit to go to the work area;

[0012] controlling the operation unit to acquire image information of the work area, and identifying the spatial layout in the work area according to the image information;

[0013] splitting the current operation instruction into an executable action sequence according to the spatial layout in the work area and the operation sequence.

[0014] According to a preferred embodiment of the present application, the operation sequence comprises at least one of a dependency relationship between actions, an execution time, and an execution sequence, and before splitting the current operation instruction into an executable action sequence according to the spatial layout in the work area and the operation sequence, the method further comprises:

[0015] inputting the current operation instruction into a splitting model to output at least one of a dependency relationship between actions, an execution time, and an execution sequence in the current operation instruction.

[0016] According to a preferred embodiment of the present application, the current operation instruction is split into an executable action sequence according to the spatial layout in the work area, the object features, and the operation sequence.

[0017] According to a preferred embodiment of the present application, the controlling the operation unit to sequentially execute each action in the current action sequence comprises:

[0018] controlling the operation unit to execute a current action in the current action sequence, comprising: determining an execution object, an execution posture, and an execution path of the current action in the current action sequence; and controlling the operation unit to reach the execution object according to the execution path and adjust to the execution posture, and then execute the current action;

[0019] after the current action is executed, controlling the operation unit to execute a next action in the current action sequence until all actions in the current action sequence are completed.

[0020] According to a preferred embodiment of the present application, the method further comprises:

[0021] collecting a real-time RGB image and a depth image of the work area;

[0022] identifying a real-time spatial layout of the work area according to the RGB image and the depth image;

[0023] According to the real-time spatial layout, whether to adjust the execution posture and / or the execution path is determined, and the execution posture and / or the execution path are adjusted according to the determination result.

[0024] To solve the above technical problems, the second aspect of the present application provides a device for controlling experimental operation based on experimental process, characterized in that the device comprises:

[0025] The receiving module is configured to receive an experimental process control instruction, wherein the experimental process control instruction comprises a plurality of operation instructions according to operation steps, and the operation instruction at least comprises an operation action.

[0026] The first execution module is configured to execute the current operation instruction, including determining a working area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence; and controlling the operation unit to execute each action in the current action sequence in sequence.

[0027] The second execution module is configured to execute the next operation instruction after completing the current operation instruction, until all operation instructions are completed.

[0028] To solve the above technical problems, the third aspect of the present application provides an experimental operation robot based on experimental process control, comprising a processor; and

[0029] The memory stores computer executable instructions, which when executed cause the processor to execute the above method.

[0030] To solve the above technical problems, the fourth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores one or more programs, and when the one or more programs are executed by a processor, the above method is implemented. Advantages

[0031] The present application can receive a pre-generated experimental process control instruction, which comprises a plurality of operation instructions according to operation steps, and by determining a working area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence, the operation unit is controlled to execute each action in the current action sequence in sequence; the current operation instruction is automatically executed, and after completing the current operation instruction, the next operation instruction is executed according to the operation steps, until all operation instructions are completed. Thus, each operation in the complete experimental process is automatically completed, the entire process does not require human intervention, and the automation of chemical experiments is fully realized from the action level. Compared with the prior art, the present application can effectively save the investment of human cost, improve the number of reactions that can be completed by a single laboratory, and avoid potential dangers to the health of workers caused by some chemical reactions. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to make the technical problems solved by the present application, the technical means adopted and the technical effects obtained more clear, specific embodiments of the present application will be described in detail below with reference to the drawings. However, it should be pointed out that the drawings described below are only the drawings of exemplary embodiments of the present application, and for those skilled in the art, other drawings of embodiments can be obtained from these drawings without creative labor.

[0033] Fig. 1 is a flowchart of a method for controlling experimental operation based on experimental procedure according to an embodiment of the present application;

[0034] Fig. 2 is a schematic diagram of an RGB picture image of a weighing area taken by an embodiment of the present application;

[0035] Fig. 3 is a schematic diagram of a structural framework of a device for controlling experimental operation based on experimental procedure according to an embodiment of the present application;

[0036] Fig. 4 is a structural block diagram of an exemplary embodiment of a robot for controlling experimental operation based on experimental procedure according to an embodiment of the present application;

[0037] Fig. 5 is a schematic diagram of an embodiment of a computer readable medium according to the present application. Best Mode for Carrying Out the Invention

[0038] An embodiment of the present application provides a method for controlling experimental operation based on experimental procedure, which comprises:

[0039] receiving an experimental procedure control instruction, wherein the experimental procedure control instruction comprises a plurality of operation instructions according to operation steps, and the operation instruction at least comprises an operation action;

[0040] executing a current operation instruction, which comprises determining a working area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence; and controlling an operation unit to execute each action in the current action sequence in sequence;

[0041] After completing the current operation instruction, a next operation instruction is executed until all operation instructions are completed. Embodiments of the present application

[0042] Exemplary embodiments of the present application will now be described more fully with reference to the accompanying drawings, in which exemplary embodiments can be embodied in various specific forms, but should not be construed as limited to the embodiments set forth herein. On the contrary, the exemplary embodiments are provided so that the present application can be more complete and so that the inventive concept can be fully conveyed to those skilled in the art.

[0043] The structures, properties, effects or other features described in a certain specific embodiment may be combined with one or more other embodiments in any suitable manner within the technical concept of the present application.

[0044] In the introduction of specific embodiments, the detailed description of structures, properties, effects or other features is for the purpose of making the embodiments fully understood by those skilled in the art. However, it does not exclude that those skilled in the art can implement the present application without the above-mentioned structures, properties, effects or other features in specific cases.

[0045] The flowchart in the drawings is only an exemplary flow demonstration, and does not represent that all the contents, operations and steps in the flowchart must be included in the scheme of the present application, nor does it represent that the execution must be performed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the inventive concept of the present application.

[0046] The block diagram in the drawings generally represents functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0047] The same reference signs in the drawings represent the same or similar elements, components or parts, so that the repeated description of the same or similar elements, components or parts can be omitted hereinafter. It should also be understood that although the first, second, third, etc. denotative adjectives may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these adjectives. That is, these adjectives are only used to distinguish one from another. For example, the first device can also be referred to as the second device without departing from the essential technical solution of the present application. In addition, the terms "and / or", "and / or" mean all combinations of the listed items.

[0048] Fig. 1 is a flowchart of a method for controlling experimental operation based on experimental process provided by the present application; as shown in Fig. 1, the method comprises:

[0049] S1, receiving an experimental process control instruction;

[0050] In this embodiment, the experimental process control instruction refers to a process instruction corresponding to a complete experiment from beginning to end. It can include: a plurality of operation instructions composed of operation steps, which constitute a process instruction corresponding to a complete experiment; the operation instruction at least includes operation action. For example, in the process of erecting a reaction device, a complete erection process can be completed by a plurality of operation actions. Further, the operation instruction can also include: raw material information and / or reaction conditions. Among them: the raw material information can include: raw material name, raw material weight, etc., and the reaction conditions can include: pressure value, temperature value, PH value, etc. For example: in a complete experimental process control instruction, in the chemical reaction preparation stage, there is an operation instruction of "weighing 5ml water in the weighing area".

[0051] In this embodiment, the experimental process control instruction can be manually edited and input, or can be generated after making decisions and judgments at the level of chemical experiment scheme according to the target chemical molecular formula input by the user. For example, in an example, the experimental process control instruction can be generated after making decisions and judgments at the level of chemical experiment scheme according to the target chemical molecular formula input by the user, then this step receives the experimental process control instruction, and according to the application, the experimental process control instruction is disassembled into executable actions to form the corresponding action sequence, and the operation unit (corresponding hardware) is controlled to sequentially complete the single actions in the action sequence, thereby automatically completing the chemical experiment operation.

[0052] S2, executing the current operation instruction, including: determining the working area for executing the current operation instruction, and disassembling the current operation instruction into an executable action sequence according to the spatial layout and operation sequence of the working area; controlling the operation unit to sequentially execute each action in the current action sequence;

[0053] Considering that the execution order of actions in the experiment operation is affected by the spatial layout of objects (experimental instruments, experimental raw materials, etc.) and the operation sequence; for example: in two actions with the same operation sequence, the position of the experimental instrument used by action 1 is closer to the operation unit than the position of the experimental instrument used by action 2, then action 1 can be executed first. In this embodiment, the current operation instruction is disassembled into an executable action sequence according to the spatial layout and operation sequence of the working area, wherein: the spatial layout of the working area includes: objects in the working area, relative positions between objects, object postures, etc. Then the determination of the working area for executing the current operation instruction and the disassembly of the current operation instruction into an executable action sequence according to the spatial layout and operation sequence of the working area can include:

[0054] S21, identifying the working area for executing the current operation instruction and controlling the operation unit to go to the working area;

[0055] For example, the operation instruction and the corresponding work area can be pre-collected to train the work area recognition model, and then the current operation instruction can be input into the trained work area recognition model to output the work area for executing the current operation instruction. The work area recognition model can be a neural network model or the like. The operation unit can be a robot, a mechanical arm or the like.

[0056] For example, the current operation instruction is "weigh 5ml water in the weighing area", and the work area recognition model can be input with "weigh 5ml water in the weighing area" to output the work area as "weighing area", and then the robot can be started to go to the "weighing area".

[0057] S22, control the operation unit to acquire image information of the work area, and recognize the spatial layout in the work area according to the image information.

[0058] In this embodiment, the image information can include an RGB image and a depth image, and the operation unit can be provided with a shooting device capable of acquiring the RGB image and the depth image at the same time, such as a depth camera, a binocular camera or the like. Then, the camera can be controlled to shoot multiple RGB images and depth images of the work area, which are input into the trained visual language model (VLM) to recognize the objects in the work area, the relative positions between the objects and the object poses, so as to recognize the spatial layout in the work area.

[0059] For example, the current operation instruction is "weigh 5ml water in the weighing area", and the work area recognition model can be input with "weigh 5ml water in the weighing area" to output the work area as "weighing area", and then the robot can be started to go to the "weighing area".

[0060] S23, according to the spatial layout in the work area and the operation sequence, the current operation instruction is divided into an executable action sequence.

[0061] The operation sequence is used to describe the execution order between actions, and can include at least one of a dependency relationship between actions, an execution time, and an execution order. Before this step, the operation sequence can be determined according to the current operation instruction. For example, the operation instruction and the dependency relationship between actions, the execution time, and the execution order included in the operation instruction can be pre-collected to train the splitting model, and then the current operation instruction can be input into the trained splitting model to output at least one of the dependency relationship between actions, the execution time, and the execution order in the current operation instruction.

[0062] Further, the spatial layout in the work area, the operation sequence, and the corresponding action sequence can be pre-collected as training data to train the action splitting model, and then the spatial layout in the work area and the operation sequence of the current operation instruction can be input into the trained action splitting model to output an executable current action sequence. The action sequence is composed of a plurality of executable actions, and the executable actions can include grasping, placing, inserting, pulling out, pushing / pulling, etc.

[0063] For example, the spatial layout and the operation sequence of the weighing area in FIG. 2 are input into the action splitting model, and the output action sequence is:

[0064] 4.1-Right arm grasps the “water” raw material bottle;

[0065] 4.2-Transfers to the left arm;

[0066] 4.3-Right arm pulls out the bottle plug;

[0067] 4.4-Right arm places the bottle plug on the table top;

[0068] 4.5-Left arm places the raw material bottle on the table top;

[0069] 4.6-Right arm grasps the sample tube from the test tube rack to a fixed position;

[0070] 4.7-Right arm grasps the needle cylinder;

[0071] 4.8-Right arm inserts the needle tube into the raw material bottle and fixes the position;

[0072] 4.9-Left arm pulls the needle tube to a position corresponding to a 5ml volume;

[0073] 4.10-Right arm moves the needle cylinder to insert the needle tube into the sample tube;

[0074] 4.11-Left arm pushes the needle tube to a position corresponding to emptying the liquid;

[0075] 4.12-Right arm places the needle tube on the table top;

[0076] 4.13-Right arm grasps the sample tube and places it back on the test tube rack;

[0077] 4.14 - Right arm grasps the bottle stopper;

[0078] 4.15 - Left arm grasps the sample bottle;

[0079] 4.16 - Insert the stopper into the sample vial with your right arm;

[0080] 4.17 - The left arm transfers the sample vial to the right arm;

[0081] 4.18 - The right arm places the sample vial back to its original position.

[0082] In a preferred example, after obtaining the spatial layout within the work area, further object features within the work area can be acquired, such as object shape and size. Then, based on a pre-trained model, the current operation instruction can be broken down into an executable action sequence according to the spatial layout, object features, and operation order within the work area. Object features may include object shape, size, presence of a QR code and its information, components, etc. These features can be obtained from images captured by an external camera using a visual recognition algorithm.

[0083] In one example, the control operation unit sequentially executing each action in the current action sequence may include:

[0084] S201. The control operation unit executes the current action in the current action sequence, including: determining the execution object, execution posture, and execution path of the current action in the current action sequence; the control operation unit reaches the execution object according to the execution path, adjusts to the execution posture, and then executes the current action;

[0085] Wherein: the execution object refers to the object that can be manipulated, which may include: cylinders, raw material bottles, syringes, etc. The execution posture may further include: the execution point and execution angle on the execution object. The execution path is the route taken from the current position to the execution object.

[0086] For example, executable actions, their corresponding execution objects, execution postures, and execution paths can be pre-collected as training data to train the execution deterministic model. Then, when the current action is input into the execution deterministic model, the execution object, execution posture, and execution path are output.

[0087] For example, in the action sequence output in step S23, the current action is the first action, namely "right arm grabs water bottle". In this step, the action of "right arm grabs water bottle" is input into the execution determination model, and the output result is as follows:

[0088] 5.1 - The target of execution is: water raw material bottles.

[0089] 5.2 - The execution point is: the bottle body location;

[0090] 5.3-Execution path is: from the position of the water raw material bottle to the position without fixed position (the executable area determined by the robot);

[0091] 5.4-Control the right arm of the robot to move to the position of the water raw material bottle (cooperate with the image shot in real time), grab the bottle body position of the water raw material bottle, and move the water raw material bottle to the executable area above the table top through the execution path.

[0092] Further, in the actual work area, the position and / or posture of some objects may change. In order to improve the accuracy of the execution posture and the execution path, the execution posture and / or the execution path can be adjusted according to the real-time spatial layout of the work area. Therefore, after determining the execution object, the execution posture and the execution path by the model, the method can further comprise: controlling the camera of the operation unit to collect the RGB image and the depth image of the work area in real time; identifying the real-time spatial layout of the work area according to the RGB image and the depth image; judging whether to adjust the execution posture and / or the execution path according to the real-time spatial layout, and adjusting the execution posture and / or the execution path according to the judgment result. Specifically, the real-time spatial layout obtained by the camera during the movement process can be input into the preset model. The preset model calculates and judges whether there is an obstacle on the execution path and whether there is enough space to complete the action according to the real-time spatial layout. If there is an obstacle or not enough space, the preset model returns this information, and replans the execution path based on the information.

[0093] S202, after the current action is executed, the next action in the current action sequence is executed by the operation unit until all actions in the current action sequence are completed.

[0094] Since the execution posture of the operation unit changes during the action execution process, the execution posture of the execution object can be monitored in real time during the current action execution process. When the execution posture is consistent with the preset standard execution posture, it is determined that the current action is completed.

[0095] When the current action is completed, it is judged whether the current action is the last executable action in the current action sequence. If yes, step S3 is entered; if not, the next action is obtained from the current action sequence, and steps S21-S202 are sequentially executed to control the operation unit to execute the next action in the current action sequence until all actions in the current action sequence are completed. For example, when it is judged that the robot has completed the action of 4.1, the next action 4.2 is started. In this way, all actions in 4 are completed.

[0096] S3, after completing the current operation instruction, the next operation instruction is executed until all operation instructions are completed.

[0097] For example, if it is determined that the current action is the last executable action in the current action sequence, indicating that the current operation instruction is completed, then step S2 is executed in a loop until all operation instructions are completed. Thus, each operation in the complete experimental process is automatically completed, and the entire process does not require human intervention, thereby fully realizing the automation of chemical experiments from the action level.

[0098] It should be noted that the model provided in each step of the present application can be based on a pre-trained model with some general and broad capabilities, and the pre-trained model is further trained on its corresponding training set to adjust the pre-trained model parameters to obtain a model applicable to a specific field in each step, thereby better completing the scenario and task faced by the model. In addition, the model can be optimized and trained by continuously collecting various data (such as user feedback and experimental results, etc.) in the experimental process to improve the accuracy of the model.

[0099] Based on the above method of controlling experimental operation based on experimental process, the present application further provides a device for controlling experimental operation based on experimental process. FIG. 3 is a structural framework diagram of a device for controlling experimental operation based on experimental process according to the present application. As shown in FIG. 3, the device comprises:

[0100] The receiving module 31 is configured to receive an experimental process control instruction, wherein the experimental process control instruction comprises a plurality of operation instructions according to operation steps, and the operation instruction comprises at least an operation action. The first execution module 32 is configured to execute a current operation instruction, including determining a working area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence.

[0101] The second execution module 33 is configured to execute a next operation instruction after completing the current operation instruction, until all operation instructions are completed.

[0102] In a specific embodiment, the first execution module 32 comprises:

[0103] The first identification module is configured to identify a working area for executing the current operation instruction and control the operation unit to go to the working area.

[0104] The second identification module is configured to control the operation unit to obtain image information of the working area, and identify the spatial layout in the working area according to the image information.

[0105] The splitting module is configured to split the current operation instruction into an executable action sequence according to the spatial layout in the work area and the operation sequence. Preferably, the splitting module is configured to split the current operation instruction into an executable action sequence according to the spatial layout in the work area, the object features, and the operation sequence.

[0106] In an example, the operation sequence includes at least one of a dependency relationship between actions, an execution time, and an execution order. The apparatus further includes:

[0107] The sub-splitting module is configured to input the current operation instruction into a splitting model and output at least one of a dependency relationship between actions, an execution time, and an execution order in the current operation instruction.

[0108] Further, the first execution module 32 further includes:

[0109] The first sub-execution module is configured to control the operation unit to execute a current action in the current action sequence, including: determining an execution object, an execution posture, and an execution path of the current action in the current action sequence; and controlling the operation unit to execute the current action after reaching the execution object and adjusting to the execution posture according to the execution path.

[0110] The second sub-execution module is configured to control the operation unit to execute a next action in the current action sequence after the current action is executed, until all actions in the current action sequence are completed.

[0111] In a preferred embodiment, the apparatus further includes:

[0112] The acquisition module is configured to acquire real-time RGB images and depth images of the work area.

[0113] The real-time recognition module is configured to recognize a real-time spatial layout of the work area according to the RGB images and the depth images.

[0114] The adjustment module is configured to determine whether to adjust the execution posture and / or the execution path according to the real-time spatial layout, and to adjust the execution posture and / or the execution path according to a determination result.

[0115] Those skilled in the art can understand that each module in the above apparatus embodiments can be distributed in the apparatus as described, or can be changed accordingly and distributed in one or more apparatuses different from the above embodiments. The modules of the above embodiments can be combined into one module, or can be further split into multiple sub-modules.

[0116] The following describes an experimental procedure control experimental operation robot embodiment of the present application, which can be considered as a physical form of implementation of the above-mentioned method and device embodiments of the present application. For the details described in the robot embodiment of the present application, it should be considered as a supplement to the above-mentioned method or device embodiments; for the details not disclosed in the robot embodiment of the present application, it can be implemented with reference to the above-mentioned method or device embodiments.

[0117] Figure 4 is a structural block diagram of an exemplary embodiment of a robot according to the present application. The robot shown in Figure 4 is only an example and should not bring any limitation to the function and use range of the embodiment of the present application.

[0118] As shown in Figure 4, the robot 400 of the exemplary embodiment is in the form of a general-purpose data processing device. The components of the robot 400 can include, but are not limited to, at least one processing unit 410, at least one storage unit 420, a bus 430 connecting different electronic device components (including the storage unit 420 and the processing unit 410), a display unit 440, etc.

[0119] Among them, the storage unit 420 stores a computer readable program, which can be a source program or a code of a read-only program. The program can be executed by the processing unit 410, so that the processing unit 410 performs the steps of various embodiments of the present application. For example, the processing unit 410 can perform the steps as shown in Figure 1.

[0120] The storage unit 420 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 4201 and / or a cache memory unit 4202, and can further include a read-only memory (ROM) 4203. The storage unit 420 can also include programs / utilities 4204 having a set of (at least one) program modules 4205, such as operating electronic devices, one or more application programs, other program modules, and program data, each of which or some combination thereof can include implementation of a network environment.

[0121] The bus 430 can be one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0122] The robot 400 can also communicate with one or more external devices 300 (such as a keyboard or a printer) to enable a user to interact with the robot 400 and / or one or more other data processing devices (such as a router or a modem) to enable the robot 400 to communicate with one or more other data processing devices. Such communication can occur via the input / output (I / O) interface 450, and also via a network adapter 460 to one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the public network, such as the Internet). The network adapter 460 can communicate with the other modules of the robot 400 via the bus 430. It should be appreciated that although not shown in FIG. 5, other hardware and / or software modules can be used in the robot 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID electronic devices, tape drives, and data backup storage electronic devices, etc.

[0123] FIG. 5 is a schematic diagram of an embodiment of the computer readable medium of the present application. As shown in FIG. 5, the computer program can be stored on one or more computer readable media. The computer readable media can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. The computer readable medium enables the above-described method of the present application when the computer program is executed by one or more data processing devices, i.e., receiving an experiment flow control instruction, the experiment flow control instruction including a plurality of operation instructions according to operation steps; the operation instructions including at least operation actions; executing a current operation instruction, including determining a working area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the working area and the operation sequence; controlling an operation unit to execute each action in the current action sequence in sequence; and after completing the current operation instruction, executing a next operation instruction until all operation instructions are completed.

[0124] Those skilled in the art can easily understand that the exemplary embodiments described in the present application can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a computer readable storage medium (which can be a CD-ROM, an U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a data processing device (which can be a personal computer, a server, or a network device, etc.) execute the above-mentioned methods according to the present application.

[0125] The computer readable storage medium can include a data signal carried in the baseband or propagated as a carrier wave in a propagated data signal, which bears computer-readable program code. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. Computer readable medium can be any medium that can be read by a computer or a data processing device, and can be any medium that can carry or store program codes in the form of any suitable medium, which can be transmitted, propagated, or transferred for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer readable medium can be transmitted in any suitable medium, including, but not limited to, wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0126] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0127] In summary, the present application can be implemented by a computer program method, device, electronic device or computer readable medium. Some or all of the functions of the present application can be implemented in practice by using a general-purpose data processing device such as a microprocessor or a digital signal processor (DSP).

[0128] The above-described specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application, and it should be understood that the present application is not inherently related to any specific computer, virtual device or electronic equipment, and various general-purpose devices can also implement the present application. The above-described is only a specific embodiment of the present application and is not used to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of controlling experimental operations based on an experimental protocol, characterized by, The method comprises: receiving experimental process control instructions, the experimental process control instructions comprising: a plurality of operation instructions composed according to operation steps; the operation instructions comprising at least operation actions; executing a current operation instruction, comprising: determining a work area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the work area and the operation sequence; and controlling an operation unit to sequentially execute each action in the current action sequence; after completing the current operation instruction, executing a next operation instruction until all operation instructions are completed.

2. The method of claim 1, wherein, The spatial layout of the work area comprises: objects in the work area, relative positions between the objects, and object postures.

3. The method of claim 1, wherein, The determining of the work area for executing the current operation instruction, and the splitting of the current operation instruction into the executable action sequence according to the spatial layout of the work area and the operation sequence comprise: identifying the work area for executing the current operation instruction and controlling the operation unit to go to the work area; controlling the operation unit to acquire image information of the work area, and identifying the spatial layout in the work area according to the image information; splitting the current operation instruction into the executable action sequence according to the spatial layout in the work area and the operation sequence.

4. The method of claim 1, wherein, The operation sequence comprises at least one of a dependency relationship between actions, an execution time, and an execution sequence, and before the splitting of the current operation instruction into the executable action sequence according to the spatial layout in the work area and the operation sequence, the method further comprises: inputting the current operation instruction into a splitting model to output at least one of the dependency relationship between actions, the execution time, and the execution sequence in the current operation instruction.

5. The method of claim 1, wherein, splitting the current operation instruction into the executable action sequence according to the spatial layout in the work area, object features, and the operation sequence.

6. The method of claim 1, wherein, The controlling of the operation unit to sequentially execute each action in the current action sequence comprises: controlling the operation unit to execute a current action in the current action sequence, comprising: determining an execution object, an execution posture, and an execution path of the current action in the current action sequence; and controlling the operation unit to reach the execution object and adjust to the execution posture according to the execution path, and then execute the current action; after the current action is executed, controlling the operation unit to execute a next action in the current action sequence until all actions in the current action sequence are completed.

7. The method of claim 5, wherein, The method further comprises: collecting real-time RGB images and depth images of the work area; identifying real-time spatial layouts of the work area according to the RGB images and the depth images; judging whether to adjust the execution posture and / or the execution path according to the real-time spatial layouts, and adjusting the execution posture and / or the execution path according to a judgment result.

8. A device for controlling experimental operations based on experimental procedures, characterized in that, The device comprises: a receiving module configured to receive experimental process control instructions, the experimental process control instructions comprising: a plurality of operation instructions composed according to operation steps; the operation instructions comprising at least operation actions; a first execution module configured to execute a current operation instruction, comprising: determining a work area for executing the current operation instruction, and splitting the current operation instruction into an executable action sequence according to the spatial layout of the work area and the operation sequence; and controlling an operation unit to sequentially execute each action in the current action sequence; A second execution module is configured to execute the next operation instruction after the current operation instruction is completed until all operation instructions are completed.

9. An experimental procedure-based control experimental operation robot, characterized by, Comprise: a processor; and a memory storing computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-6.

10. A computer readable storage medium, wherein, The computer-readable storage medium stores one or more programs, which when executed by a processor, implement the method of any of claims 1-6.

Images