Experimental task execution method and apparatus

Abstract

The present application provides an experimental task execution method and device, the method comprises: based on a visual language model, the task information of a target experimental task is divided to obtain a subtask sequence, the subtask sequence is arranged from front to back according to execution order by multiple subtasks; starting from the first subtask in the subtask sequence, the visual language model is used to process the text information corresponding to the current subtask and the experimental image before the execution of the current subtask to obtain a visual prompt image, and through a visual language action model, based on the text information, the experimental image and the visual prompt image processing, the robot is guided to execute the current subtask, until the last subtask in the subtask sequence is completed, and the target experimental task is determined to be completed. The present application effectively improves the success rate, operation safety and procedure compliance of the experimental task, and has high universality and safety.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for performing experimental tasks. Background Technology

[0002] While existing robotic chemical experiment techniques have made some progress, in practical applications, chemical experiments often involve complex operations with multiple steps. Existing robotic systems, such as the Advanced Chunk Transformer (ACT), Robotics Diffusion Transformer (RDT), and π0, struggle to achieve efficient execution of the entire process when handling these long-running tasks, and are prone to failures or errors at the transitions between steps. Therefore, an effective solution is urgently needed to address these issues. Summary of the Invention

[0003] To address the above problems, the present invention provides a method and apparatus for performing experimental tasks.

[0004] This invention provides a method for performing experimental tasks, comprising:

[0005] Based on the visual language model, the task information of the target experimental task is divided to obtain a sub-task sequence, which is formed by arranging multiple sub-tasks in the execution order from front to back.

[0006] Starting from the first subtask in the subtask sequence, the visual language model processes the text information corresponding to the current subtask and the experimental image before the execution of the current subtask to obtain a visual cue image. Based on the text information, the experimental image, and the visual cue image, the visual language action model guides the robot to execute the current subtask until the last subtask in the subtask sequence is completed, thus confirming the completion of the target experimental task.

[0007] According to an experimental task execution method provided by the present invention, the step of processing the text information corresponding to the current subtask and the experimental image before the execution of the current subtask through the visual language model to obtain a visual cue image includes:

[0008] Extract the text information corresponding to the current subtask;

[0009] The text information is semantically analyzed using the visual language model to obtain semantic analysis results.

[0010] The visual cues image is obtained by annotating the experimental image using the visual language model and based on the semantic analysis results.

[0011] According to an experimental task execution method provided by the present invention, the experimental image is a multi-view experimental image;

[0012] The step of annotating the experimental image based on the semantic analysis results using the visual language model to obtain the visual cue image includes:

[0013] Using the visual language model, the front view in the multi-view experimental image is labeled based on the semantic analysis results to obtain the visual cue image.

[0014] According to an experimental task execution method provided by the present invention, after guiding the robot to execute the current sub-task based on the text information, the experimental image, and the visual cue image processing, the method further includes:

[0015] The visual language model is used to acquire and verify whether the current subtask has met the target based on the experimental image after the current subtask is completed.

[0016] If the target is met, then execute the execution steps of the next subtask of the current subtask;

[0017] If the target is not met, the execution steps of the current subtask will be re-executed.

[0018] According to the present invention, an experimental task execution method is provided, wherein the step of guiding a robot to execute the current sub-task by means of a visual language action model, based on the text information, the experimental image, and the visual cue image processing, includes:

[0019] Operation information is generated based on the text information, the experimental image, and the visual cue image processed by the visual language action model.

[0020] Based on the operational information, the robot is controlled to execute the current sub-task.

[0021] According to an experimental task execution method provided by the present invention, the operation information includes at least one operation instruction, and each of the operation instructions carries an operation sequence.

[0022] The step of controlling the robot to execute the current sub-task based on the operation information includes:

[0023] Starting with the first operation instruction in the operation sequence, the robot is controlled to execute the operation corresponding to the current operation instruction until the operation corresponding to the last operation instruction in the operation sequence is completed, and the current subtask is confirmed to be completed.

[0024] The present invention also provides an experimental task execution device, comprising the following modules:

[0025] The partitioning module is configured to partition the task information of the target experimental task based on a visual language model to obtain a sub-task sequence, wherein the sub-task sequence is formed by arranging multiple sub-tasks in the execution order from front to back;

[0026] The execution module is configured to start from the first subtask in the subtask sequence, process the text information corresponding to the current subtask and the experimental image before the execution of the current subtask through the visual language model to obtain a visual cue image, and guide the robot to execute the current subtask based on the text information, the experimental image and the visual cue image through the visual language action model, until the last subtask in the subtask sequence is completed, and the target experimental task is determined to be completed.

[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the experimental task execution method as described above.

[0028] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the experimental task execution method as described above.

[0029] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the experimental task execution method as described above.

[0030] The experimental task execution method and apparatus provided by this invention divides the task information of the target experimental task based on a visual language model to obtain a sub-task sequence. The sub-task sequence consists of multiple sub-tasks arranged in execution order from front to back. Starting from the first sub-task in the sub-task sequence, the visual language model processes the text information corresponding to the current sub-task and the experimental image before the execution of the current sub-task to obtain a visual cue image. Then, a visual language action model, based on the processed text information, experimental image, and visual cue image, guides the robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, thus confirming the completion of the target experimental task. This invention adopts a dual-loop closed-loop control architecture, combining a visual language model and a visual language action model. The visual language model enables semantic-level planning of the experimental task, decomposing complex tasks into controllable basic operation sequences. The visual language action model fuses the visual cue image, text information, and experimental image, accurately guiding the robot's actions. This effectively improves the success rate, operational safety, and procedural compliance of experimental tasks, exhibiting high versatility and safety. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 This is one of the flowcharts illustrating the experimental task execution method provided by the present invention.

[0033] Figure 2 This is the second flowchart of the experimental task execution method provided by the present invention.

[0034] Figure 3 This is a schematic diagram of the visual cue image provided by the present invention.

[0035] Figure 4 This is a schematic diagram of the experimental task execution device provided by the present invention.

[0036] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] First, a brief description of the relevant content involved in this invention will be given.

[0039] Existing robotic chemical experiment techniques, besides being insufficient for long-duration experimental tasks, also suffer from the following problems:

[0040] Challenges of transparent container recognition: Existing vision-language models (VLMs), such as VoxPoser and ReKep, heavily rely on depth sensors and object segmentation techniques. However, these methods perform poorly when dealing with transparent containers commonly used in chemical experiments (such as glass beakers and test tubes), resulting in a high visual recognition error rate and thus affecting the success rate of subsequent operations.

[0041] Limitations of visual cueing methods: Existing visual cueing methods, such as the Marking Open-world Keypoint Affordances (MOKA) method, can provide a certain degree of visual guidance, but due to the lack of direct consideration of textual instructions, non-compliant or unsafe grasping and operation behaviors may occur in chemical experimental scenarios with high safety requirements and strict procedural requirements.

[0042] Insufficient semantic-level feedback and safety compliance: Existing Vision-Language-Action Model (VLA) models, such as RDT and π0, although they perform well in executing specific actions, lack global semantic understanding and closed-loop feedback mechanisms. They are difficult to proactively conduct safety checks and confirm procedural compliance, so they are prone to operational errors or even safety accidents in complex or high-safety-requirement experimental scenarios.

[0043] The aforementioned problems severely limit the widespread application of robotics technology in practical chemical experiments. Therefore, this invention provides a method and apparatus for executing experimental tasks, capable of efficiently and accurately completing long-duration, highly complex chemical experiments while ensuring operational safety and compliance with experimental procedures.

[0044] The following is combined with Figures 1-5 The present invention describes the experimental task execution method and apparatus.

[0045] Figure 1 This is one of the flowcharts illustrating the experimental task execution method provided by the present invention, such as... Figure 1 As shown, the method includes steps 101 and 102.

[0046] Step 101: Based on the visual language model, the task information of the target experimental task is divided to obtain a sub-task sequence, which is formed by arranging multiple sub-tasks in the order of execution from front to back.

[0047] Specifically, the experimental task execution method provided by the present invention is applicable to experimental robot systems. The experimental robot system consists of a visual language model, a visual language action model, a robot, and imaging devices set on the robot. The robot refers to a machine device that automatically performs work, which can be a mechanical arm (such as a dual-arm gripper device) or a humanoid machine that can walk independently and has two arms.

[0048] See Figure 2 , Figure 2This is the second flowchart illustrating the experimental task execution method provided by this invention: The Visual Language Model (VLM) is positioned in the outer loop of the experimental robot system. The VLM can function as a task planner, a visual prompt generator, and a task monitor, operating throughout the entire experimental task execution process. The Vision-Language-Action Model (VLA) is located in the inner loop of the experimental robot system. The VLA model is responsible for fusing the three types of information—the language description and prompt images provided by the VLM, and the experimental images acquired by the imaging equipment—inputting them as input and outputting specific action commands to achieve high-precision control of the robot.

[0049] Furthermore, experimental robot systems can be equipped with multiple robotic arms capable of collaboration, with multiple VLAs scheduled by the same VLM to jointly complete multi-station or serial experimental processes. Current experimental robot systems primarily rely on dual-arm gripper structures to perform various atomic operations. However, for tasks requiring high compliance and precise grasping (such as transferring powder or holding a glass rod), a humanoid two-hand structure with multiple degrees of freedom and sensor feedback capabilities can be used. This can further improve the performance of experimental robot systems in micro-manipulation and complex control tasks, and enhance their versatility.

[0050] The target experimental task refers to the experimental task to be performed, which can be a chemical experiment task, a physical experiment task, etc.

[0051] In practical applications, see Figure 2 The visual language model acts as a planner for task planning: It first receives task information (Task) for the target experiment, which may include a description of the experimental task and equipment information. Further, based on the task information, the visual language model decomposes the complex experimental process into executable atomic operations (Primitive Tasks), forming a subtask list with a clear temporal relationship, such as "transferring solids," "grabbing a glass rod," and "adding acid." The subtasks in the subtask list are arranged from beginning to end according to their execution order, such as "1.xxx" to "4.xxx".

[0052] Step 102: Starting from the first subtask in the subtask sequence, the visual language model is used to process the text information corresponding to the current subtask and the experimental image before the execution of the current subtask to obtain a visual cue image. Based on the text information, the experimental image and the visual cue image, the visual language action model is used to guide the robot to execute the current subtask until the last subtask in the subtask sequence is completed, thus confirming the completion of the target experimental task.

[0053] Specifically, the experimental images are captured by imaging devices; the imaging devices can be red-green-blue (RGB) cameras; the experimental images can be multi-view experimental images, including the front view, top view, left view, and right view of the current experimental scene. The front view is captured by an imaging device installed in front of the robot, the top view can be captured by an imaging device installed at the robot's top angle, the left view is captured by an imaging device installed on the robot's left wrist, and the right view is captured by an imaging device installed on the robot's right wrist.

[0054] Text information can be a text description of the subtask to be performed, such as text instructions.

[0055] In practical applications, for each subtask in the subtask sequence, each subtask is executed sequentially. Once the currently executing subtask is completed, the next subtask is executed, and so on, until all subtasks are executed and the target experimental task is determined to be completed.

[0056] For each subtask, the specific execution process is as follows: An experimental image is acquired via imaging equipment after the completion of the previous subtask or before the execution of the target experimental task, and used as the experimental image before the execution of the current subtask. Specifically, if the current subtask is the first subtask, the experimental image before the execution of the current subtask is the same as the experimental image before the execution of the target experimental task; if the current subtask is not the first subtask, the experimental image before the execution of the current subtask is the same as the experimental image after the completion of the previous subtask.

[0057] See Figure 2 The visual language model, acting as a visual prompt, generates semantically understood visual cue images based on the text information (Text) corresponding to the current subtask and the experimental images (Observed Images), combined with the Qwen2.5-VL model. These visual cue images are labeled with at least one of the following: a bounding box area for the grasping / target object and key points. Furthermore, the visual language action model fuses the Text and Prompted Images provided by the VLM, as well as the Observed Images provided by the imaging device, to obtain action information. This action information is then used to control the robot to perform the corresponding actions, achieving high-precision control of the robot.

[0058] In addition, visual language models can generate other visual cues based on multimodal information such as depth maps, thermal imaging, and spectroscopic data to adapt to more complex experimental scenarios.

[0059] The experimental task execution method provided by this invention divides the task information of the target experimental task based on a visual language model to obtain a sub-task sequence. The sub-task sequence consists of multiple sub-tasks arranged in execution order from front to back. Starting from the first sub-task in the sub-task sequence, the visual language model processes the text information corresponding to the current sub-task and the experimental image before the execution of the current sub-task to obtain a visual cue image. Then, a visual language action model, based on the processed text information, experimental image, and visual cue image, guides the robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, thus confirming the completion of the target experimental task. This invention adopts a dual-loop closed-loop control architecture, combining a visual language model and a visual language action model. The visual language model enables semantic-level planning of the experimental task, decomposing complex tasks into controllable basic operation sequences. The visual language action model fuses the visual cue image, text information, and experimental image, accurately guiding the robot's actions. This effectively improves the success rate, operational safety, and procedural compliance of experimental tasks, exhibiting high versatility and safety.

[0060] Optionally, the step of processing the text information corresponding to the current subtask and the experimental image before the execution of the current subtask using the visual language model to obtain a visual cue image includes:

[0061] Extract the text information corresponding to the current subtask;

[0062] The text information is semantically analyzed using the visual language model to obtain semantic analysis results.

[0063] The visual cues image is obtained by annotating the experimental image using the visual language model and based on the semantic analysis results.

[0064] In practical applications, visual language models can extract text information corresponding to the current sub-task from sub-sequence tasks. Then, acting as a visual cue generator, the visual language model performs semantic understanding and analysis on the text information corresponding to the current sub-task to determine the grasping object / target object, i.e., the semantic analysis result. The grasping object / target object is then labeled in the experimental image, which can be done using bounding boxes and / or key points. This avoids center bias when handling transparent containers, thus ensuring the safety of the experimental task.

[0065] For example, see Figure 3 , Figure 3 This is a schematic diagram of the visual cue image provided by the present invention: the target object "test tube" is marked with a blue rectangle in the visual cue image, and the key point "grabbing point" on the target object "test tube" is marked with yellow dots.

[0066] This invention generates visual cues by combining textual information with visual context and using advanced vision-language models (such as Qwen-2.5-VL) to automatically identify transparent or irregular objects to be manipulated in experiments. It also generates safe and compliant target areas or key point cues in real time by combining language commands, thereby improving the accuracy of object recognition and the robustness of operation. This ensures the consistency of the robot's visual perception and action execution in complex scenarios and solves the problem of insufficient methods for the recognition of transparent containers and visual cues.

[0067] Optionally, the experimental image is a multi-view experimental image; correspondingly, the step of annotating the experimental image based on the semantic analysis results using the visual language model to obtain the visual cue image includes:

[0068] Using the visual language model, the front view in the multi-view experimental image is labeled based on the semantic analysis results to obtain the visual cue image.

[0069] In practical applications, to further reduce data processing volume and improve task execution efficiency while ensuring the effectiveness of visual cues, the front view in multi-view experimental images can be annotated to obtain visual cue images. The top view, left-side view, and side view in multi-view experimental images do not require annotation.

[0070] Optionally, after guiding the robot to perform the current subtask based on the text information, the experimental image, and the visual cue image processing, the method further includes:

[0071] The visual language model is used to acquire and verify whether the current subtask has met the target based on the experimental image after the current subtask is completed.

[0072] If the target is met, then execute the execution steps of the next subtask of the current subtask;

[0073] If the target is not met, the execution steps of the current subtask will be re-executed.

[0074] In practical applications, visual language models can also serve as task monitors to verify the execution results of each subtask.

[0075] Specifically, see Figure 2After the current subtask is completed, the acquisition device transmits the experimental image after the completion of the current subtask. The VLM determines whether the current subtask has achieved its goal. If it has, it continues to execute the next subtask; if it has not, it automatically triggers a repeat. In this way, closed-loop task management with error correction capabilities can be achieved, breaking through the limitation of only having one round of inner loop feedback.

[0076] Thus, by establishing a complete closed-loop monitoring mechanism, VLM acts as a real-time monitor to evaluate the execution of each operation step, provide proactive feedback, and make timely corrections in the event of failure or insecurity, ensuring operational safety and strict adherence to procedures, and solving the problems of insufficient semantic-level feedback and security compliance.

[0077] Optionally, guiding the robot to perform the current sub-task through a visual language action model, based on the text information, the experimental image, and the visual cue image processing, includes:

[0078] Operation information is generated based on the text information, the experimental image, and the visual cue image processed by the visual language action model.

[0079] Based on the operational information, the robot is controlled to execute the current sub-task.

[0080] Specifically, the operation information may include operation instructions and operation conditions. The operation instructions represent the action of the operation, and the operation conditions represent the part or component of the operation. For example, the operation instruction is "grab the test tube", and the operation condition is "grab the part one-third of the distance from the mouth of the test tube".

[0081] In practical applications, the Visual Language Action Model (VLM) serves as the action execution layer. It receives experimental images from four RGB cameras (front view, top view, and both wrists), and combines these with visual cues and text information provided by the VLM to generate operation instructions and conditions. Furthermore, the robot executes the current subtask according to these instructions and conditions. This process enables the generation of operational information, significantly improving operational stability in complex scenarios.

[0082] Optionally, the operation information includes at least one operation instruction, each of the operation instructions carrying an operation sequence; correspondingly, controlling the robot to execute the current sub-task based on the operation information includes:

[0083] Starting with the first operation instruction in the operation sequence, the robot is controlled to execute the operation corresponding to the current operation instruction until the operation corresponding to the last operation instruction in the operation sequence is completed, and the current subtask is confirmed to be completed.

[0084] In practical applications, for each subtask, the visual language action model generates a sequence of operation instructions, which includes at least one operation instruction carrying the order of operations. See also Figure 2 The five yellow squares in the operation information section represent five operation instructions.

[0085] Furthermore, the robot executes each operation instruction sequentially from front to back according to the operation sequence to complete the current subtask.

[0086] In this process, after the robot completes the action corresponding to each operation command, the visual language action model will check whether the current subtask has been completed, forming a closed loop. If the current subtask has not been completed, the visual language action model will continue to control the robot to execute the current subtask. If the current subtask has been completed, the visual language model will check whether the current subtask has met the requirements.

[0087] Furthermore, after each operation command completed by the robot, the visual language action model can verify the success of the operation. If successful, the robot executes the operation corresponding to the next operation command; if unsuccessful, it continues to execute the operation corresponding to the current operation command. That is, starting with the operation command that is first in the operation sequence, the robot is controlled to execute the operation corresponding to the current operation command. After the operation corresponding to the current operation command is successfully executed, the robot continues to be controlled to execute the operation corresponding to the next operation command, until the operation corresponding to the operation command that is last in the operation sequence is completed, confirming the completion of the current subtask. This ensures a high success rate for subtask execution.

[0088] It should be noted that, to improve the robustness of the visual language action model (VLA), a dynamic retry mechanism can be used to introduce failure samples during the training phase, thereby enhancing the VLA's ability to perceive and recover from execution failures. Specifically, the training process for the VLA is as follows: acquiring training samples, which include both positive and negative samples; training the VLA based on these training samples until a training termination condition is met. Training termination can be defined as the VLA achieving a set accuracy rate.

[0089] Furthermore, based on VLM, a small sample discrimination mechanism or a small amount of human demonstration data can be introduced to enable the system to quickly establish evaluation logic when facing unknown experimental tasks, thereby improving its versatility and adaptability.

[0090] The experimental task execution method provided by this invention will be further explained below.

[0091] Task reception and parsing: Users specify experimental tasks in natural language (e.g., "complete the acid-base neutralization experiment") and input the task semantics (task information of the target experimental task) into VLM.

[0092] Task planning:

[0093] Based on the task description and initial experimental scene image, VLM automatically decomposes the complete experimental process into a series of sub-tasks, such as: transferring sodium hydroxide (NaOH) solid → adding water → adding hydrogen chloride (HCl) to neutral.

[0094] Visual cue generation: For each subtask, VLM automatically generates a visual cue map containing gripping points, pouring points, target container boundaries, etc., by combining the experimental platform image with instruction semantics. Figure 3 As shown in the figure, a marker box is automatically generated in the upper part of the glass rod to guide VLA to perform the operation safely.

[0095] Atomic operation execution: It receives three types of input information: ① task language instructions (text information); ② observation images from four perspectives (experimental images); ③ cue images generated by VLM (visual cue images), which are fused to generate control signals (operation information) to achieve atomic-level action execution (such as grasping, pouring, stirring, etc.).

[0096] Execution status monitoring: After each subtask is completed, VLM reads the multi-view experimental images again to determine whether the operation has achieved its goal. For example, in the "adding acid" task, VLM determines whether the "neutralization" goal has been achieved by identifying whether the solution changes color. If not, it automatically triggers the re-execution action until the task is successful.

[0097] Task completion judgment: After all sub-tasks are completed, the system automatically identifies whether the experiment is successful and outputs a task completion signal; there is no manual intervention or verification process, and the entire process is completed automatically in a closed loop by VLM and VLA.

[0098] Specifically, the experimental task execution method includes the following steps: user inputs task → VLM performs semantic parsing and task planning → outputs subtask sequence; for each subtask, VLM generates visual cues → inputs them into VLA along with language descriptions and observation images; VLA performs atomic operations; VLM analyzes the execution status image and determines success or failure → success: proceed to the next task; failure: repeat the current task; after all tasks are completed, VLM issues a "task completed" signal.

[0099] The experimental task execution method provided by this invention completes complex experimental tasks without human intervention through an automated "planning-execution-evaluation-correction" closed-loop control system.

[0100] Comparative experiments were conducted to verify the experimental task execution method provided by this invention. The results show that the experimental task execution method provided by this invention significantly outperforms existing technologies in 7 types of atomic operations and 5 complete chemical experimental procedures.

[0101] In the "grabbing a glass rod" task, this invention uses visual cue images to position the grab point at 1 / 3 of the glass rod, avoiding the risk of slipping. The compliance rate is as high as 0.875, which is significantly better than π0 (0.200) and MOKA (0.350). The compliance rate refers to the proportion of compliant grabs. Compliant grabs mean that the grab position is correct and the grab is successful. Non-compliant grabs include grab failure and / or incorrect grab position.

[0102] In the "acid addition neutralization" experiment, the outer loop judges that the solution color has not completely changed and automatically triggers the "repeated acid addition" operation to complete the gradual neutralization process, demonstrating the task adaptive capability under VLM guidance;

[0103] In summary, the experimental task execution method provided by this invention improves the success rate by 23.57% and the compliance rate by 0.298 compared with the prior art, demonstrating the significant advantages of the experimental task execution method provided by this invention in complex experimental scenarios. The success rate is the ratio of the number of successful tests for each task to the total number of tests.

[0104] The experimental task execution method provided by this invention has the ability to decompose tasks into structures, automatically transforming natural language experimental objectives into structured sub-tasks using a large language model, thereby improving the robot's task understanding ability; it has a multimodal perception and prompting mechanism: introducing instruction-related visual prompts to achieve accurate identification and action planning for complex and transparent experimental vessels; it has semantic-level dynamic feedback control: the VLM has the ability to perceive the experimental state and the degree of achievement of task objectives in real time, constructing a complete outer closed loop, enhancing the robustness of the experimental robot system; and it has the ability to execute the entire process without human supervision: compared with traditional systems that require manual intervention or preset program logic, this invention achieves automatic error correction and adaptive adjustment by autonomously evaluating and controlling the task process through the VLM.

[0105] Specifically, the experimental task execution method provided by the present invention has the following advantages:

[0106] Significantly improves the completion rate of long-sequence tasks: Traditional robot control methods such as ACT and π0 lack task structure understanding when handling complex experimental processes, and are prone to failure at the breakpoint of the operation chain. This invention introduces a task structured decomposition and full-process scheduling mechanism based on a vision-language model (VLM) to realize the multi-step decomposition and sequential execution of experimental tasks, effectively reducing failures caused by discontinuities in intermediate links, and improving the success rate by 23.57%.

[0107] Improving the success rate of recognizing and manipulating complex containers (such as transparent objects): Traditional visual cueing methods, such as MOKA, often rely on depth maps or masks, which frequently produce errors when processing transparent containers. This invention, through an "instruction-aware visual cueing mechanism," can automatically generate a precise grasping region by combining verbal instructions and observed images, solving the recognition problem in scenes with blurred or occluded target boundaries, and significantly improving the grasping success rate and accuracy.

[0108] Achieving closed-loop assurance of safety standards and experimental procedures: This invention uses VLM to perform semantic-level evaluation of the results of each subtask (such as whether the color has changed, whether the container is full, etc.), and automatically retryes when the standards are not met, avoiding experimental failures or safety risks caused by execution deviations, improving the compliance index by 0.298. This mechanism is particularly effective in titration and flame testing tasks, ensuring that experimental results are reproducible and reliable.

[0109] Fully automated execution, requiring no manual annotation, evaluation, or mid-process intervention: Other methods require manual selection of anchor points or manual setting of parameters. This invention is uniformly scheduled by VLM and automatically executed by VLA. All prompts, feedback, and corrections are completed automatically, making it suitable for large-scale deployment and unmanned laboratory environments.

[0110] The system has a clear structure that is easy to expand and migrate: it adopts a modular dual-ring structure (outer ring VLM scheduling, inner ring VLA execution), which can flexibly replace the language model or execution model, adapt to different robot hardware platforms and task requirements, and has good versatility and engineering scalability.

[0111] The experimental task execution device provided by the present invention is described below. The experimental task execution device described below and the experimental task execution method described above can be referred to in correspondence.

[0112] Figure 4 This is a schematic diagram of the structure and flow of the experimental task execution device provided by the present invention, as shown below. Figure 4 As shown, the apparatus method includes:

[0113] The partitioning module 401 is configured to partition the task information of the target experimental task based on a visual language model to obtain a sub-task sequence, wherein the sub-task sequence is formed by arranging multiple sub-tasks in the order of execution from front to back;

[0114] The execution module 402 is configured to start from the first subtask in the subtask sequence, process the text information corresponding to the current subtask and the experimental image before the execution of the current subtask through the visual language model to obtain a visual cue image, and guide the robot to execute the current subtask based on the text information, the experimental image and the visual cue image through the visual language action model, until the last subtask in the subtask sequence is completed, and the target experimental task is determined to be completed.

[0115] The experimental task execution device provided by this invention divides the task information of the target experimental task into a sub-task sequence based on a visual language model. The sub-task sequence consists of multiple sub-tasks arranged in execution order from front to back. Starting from the first sub-task in the sub-task sequence, the visual language model processes the text information corresponding to the current sub-task and the experimental image before the execution of the current sub-task to obtain a visual cue image. Then, a visual language action model, based on the processed text information, experimental image, and visual cue image, guides the robot to execute the current sub-task until the last sub-task in the sub-task sequence is completed, thus confirming the completion of the target experimental task. This invention adopts a dual-loop closed-loop control architecture, combining a visual language model and a visual language action model. The visual language model enables semantic-level planning of the experimental task, decomposing complex tasks into controllable basic operation sequences. The visual language action model fuses the visual cue image, text information, and experimental image, accurately guiding the robot's actions. This effectively improves the success rate, operational safety, and procedural compliance of experimental tasks, exhibiting high versatility and safety.

[0116] Optionally, the execution module 402 is specifically configured as follows:

[0117] Extract the text information corresponding to the current subtask;

[0118] The text information is semantically analyzed using the visual language model to obtain semantic analysis results.

[0119] The visual cues image is obtained by annotating the experimental image using the visual language model and based on the semantic analysis results.

[0120] Optionally, the experimental images are multi-view experimental images;

[0121] The execution module 402 is specifically configured as follows:

[0122] Using the visual language model, the front view in the multi-view experimental image is labeled based on the semantic analysis results to obtain the visual cue image.

[0123] Optionally, the execution module 402 is further configured to:

[0124] The visual language model is used to acquire and verify whether the current subtask has met the target based on the experimental image after the current subtask is completed.

[0125] If the target is met, then execute the execution steps of the next subtask of the current subtask;

[0126] If the target is not met, the execution steps of the current subtask will be re-executed.

[0127] Optionally, the execution module 402 is specifically configured as follows:

[0128] Operation information is generated based on the text information, the experimental image, and the visual cue image processed by the visual language action model.

[0129] Based on the operational information, the robot is controlled to execute the current sub-task.

[0130] Optionally, the operation information includes at least one operation instruction, and each of the operation instructions carries an operation sequence;

[0131] The execution module 402 is specifically configured as follows:

[0132] Starting with the first operation instruction in the operation sequence, the robot is controlled to execute the operation corresponding to the current operation instruction until the operation corresponding to the last operation instruction in the operation sequence is completed, and the current subtask is confirmed to be completed.

[0133] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an experimental task execution method. The method includes: dividing the task information of the target experimental task based on a visual language model to obtain a sub-task sequence, wherein the sub-task sequence is formed by arranging multiple sub-tasks in execution order from front to back; starting from the first sub-task in the sub-task sequence, processing the text information corresponding to the current sub-task and the experimental image before the execution of the current sub-task through the visual language model to obtain a visual cue image, and guiding the robot to execute the current sub-task through a visual language action model based on the text information, the experimental image, and the visual cue image processing, until the last sub-task in the sub-task sequence is completed, thus determining that the target experimental task has been completed.

[0134] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the experimental task execution method provided by the above methods. The method includes: dividing the task information of the target experimental task based on a visual language model to obtain a sub-task sequence, wherein the sub-task sequence is formed by arranging multiple sub-tasks in execution order from front to back; starting from the first sub-task in the sub-task sequence, processing the text information corresponding to the current sub-task and the experimental image before the execution of the current sub-task through the visual language model to obtain a visual cue image, and guiding the robot to execute the current sub-task through a visual language action model based on the text information, the experimental image, and the visual cue image processing, until the last sub-task in the sub-task sequence is completed, thereby determining that the target experimental task has been completed.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the experimental task execution method provided by the above methods. The method includes: dividing the task information of the target experimental task based on a visual language model to obtain a sub-task sequence, wherein the sub-task sequence is formed by arranging multiple sub-tasks in execution order from front to back; starting from the first sub-task in the sub-task sequence, processing the text information corresponding to the current sub-task and the experimental image before the execution of the current sub-task through the visual language model to obtain a visual cue image; and guiding the robot to execute the current sub-task through a visual language action model based on the text information, the experimental image, and the visual cue image processing, until the last sub-task in the sub-task sequence is completed, thereby determining that the target experimental task has been completed.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for executing an experimental task, characterized in that, include: Based on the visual language model, the task information of the target experimental task is divided to obtain a sub-task sequence, which is formed by arranging multiple sub-tasks in the execution order from front to back. Starting from the first subtask in the subtask sequence, the visual language model processes the text information corresponding to the current subtask and the experimental image before the execution of the current subtask to obtain a visual cue image. The visual cue image is generated by the visual language model based on the semantic analysis results of the text information and by annotating the front view in the experimental image. It is used to identify the object to be operated on in the current subtask and its operation position, including the grasping point or the boundary of the target container. Then, based on the text information, the experimental image, and the visual cue image, the visual language action model guides the robot to execute the current subtask until the last subtask in the subtask sequence is completed, thus confirming the completion of the target experimental task. The visual language model operates in the outer loop of task planning and monitoring, and the visual language action model operates in the inner loop of action generation and execution. The visual language model and the visual language action model constitute a dual-loop closed control architecture.

2. The experimental task execution method according to claim 1, characterized in that, The step involves processing the text information corresponding to the current subtask and the experimental image before the execution of the current subtask using the visual language model to obtain a visual cue image, including: Extract the text information corresponding to the current subtask; The text information is semantically analyzed using the visual language model to obtain semantic analysis results. The visual cues image is obtained by annotating the experimental image using the visual language model and based on the semantic analysis results.

3. The experimental task execution method according to claim 2, characterized in that, The experimental images are multi-view experimental images; The step of annotating the experimental image based on the semantic analysis results using the visual language model to obtain the visual cue image includes: Using the visual language model, the front view in the multi-view experimental image is labeled based on the semantic analysis results to obtain the visual cue image.

4. The experimental task execution method according to claim 1, characterized in that, After guiding the robot to perform the current subtask based on the text information, the experimental image, and the visual cue image, the process further includes: The visual language model is used to acquire and verify whether the current subtask has met the target based on the experimental image after the current subtask is completed. If the target is met, then execute the execution steps of the next subtask of the current subtask; If the target is not met, the execution steps of the current subtask will be re-executed.

5. The experimental task execution method according to any one of claims 1-4, characterized in that, The step of guiding the robot to perform the current sub-task using a visual language action model, based on the text information, the experimental image, and the visual cue image, includes: Based on the text information, the experimental image, and the visual cue image, the visual language action model generates operation information. Based on the operational information, the robot is controlled to execute the current sub-task.

6. The experimental task execution method according to claim 5, characterized in that, The operation information includes at least one operation instruction, and each operation instruction carries an operation sequence. The step of controlling the robot to execute the current sub-task based on the operation information includes: Starting with the first operation instruction in the operation sequence, the robot is controlled to execute the operation corresponding to the current operation instruction until the operation corresponding to the last operation instruction in the operation sequence is completed, and the current subtask is confirmed to be completed.

7. An experimental task execution device, characterized in that, include: The partitioning module is configured to partition the task information of the target experimental task based on a visual language model to obtain a sub-task sequence, wherein the sub-task sequence is formed by arranging multiple sub-tasks in the execution order from front to back; The execution module is configured to start from the first subtask in the subtask sequence, and process the text information corresponding to the current subtask and the experimental image before the execution of the current subtask using the visual language model to obtain a visual cue image. The visual cue image is generated by the visual language model based on the semantic analysis results of the text information and by annotating the front view in the experimental image. It is used to identify the object to be operated on in the current subtask and its operation position, including the grasping point or the boundary of the target container. Then, based on the text information, the experimental image, and the visual cue image, the visual language action model guides the robot to execute the current subtask until the last subtask in the subtask sequence is completed, thus determining that the target experimental task has been completed. The visual language model runs in the outer loop of task planning and monitoring, and the visual language action model runs in the inner loop of action generation and execution. The visual language model and the visual language action model constitute a dual-loop closed control architecture.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the experimental task execution method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the experimental task execution method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the experimental task execution method as described in any one of claims 1 to 6.

Images