Multi-slot optical module feeding control method and system based on visual language action model
Patent Information
- Application Number
- CN202611283393.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-24
- Publication Date
- 2026-10-16
Smart Images
Figure CN122914148A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of embodied intelligence and industrial robot control technology, and in particular to a method and system for controlling the loading and unloading of multi-slot optical modules based on a visual language action model. Background Technology
[0002] Optical modules are core components of optical communication systems, characterized by their small size and similar appearance, with closely spaced slots on their trays. In the production and testing phase of optical modules, a complete loading and unloading process includes at least several sub-tasks: target slot identification, approach, grasping, lifting, transporting, placing, releasing, and removal.
[0003] The existing technical solutions have the following main shortcomings: First, fixed teaching trajectories store joint sequences at specific points, which usually require re-teaching when slot positions, module poses, camera positions, or the robot's initial state change. Multiple slots in the same task are stored as unrelated trajectories, making it impossible to generate reusable actions from the semantics and relative geometric relationships of the target slots.
[0004] Secondly, general visual language action models typically predict continuous action blocks directly based on images, language, and ontology states, but do not explicitly represent the process stages of loading and unloading optical modules. When the continuous gripper values and robotic arm movements regress together, small predicted fluctuations near the gripper will be amplified by the execution end into repeated opening and closing of the gripper; after placement, anomalies such as failure to release, premature release, or failure to detach after release are also prone to occur.
[0005] Third, the general motion head integrates target selection, long-distance transport, and fine end-effector alignment into the same continuous motion distribution. For multiple slots with similar appearances, it is difficult for the target slot information in the language command to achieve a stable alignment with the slot geometry, module relative pose, and motion output in the image, causing the model to tend to memorize the training trajectory rather than understand the target position.
[0006] Fourth, when action blocks are executed in fixed-length segments, the model cannot promptly use new observations to correct end-point biases; if re-predicting is required for each control cycle, it will increase inference overhead and introduce motion jitter. Model loss or open-loop motion error alone cannot accurately reflect the grasping success rate and placement accuracy.
[0007] Therefore, a dedicated control method is needed for loading and unloading optical modules in multiple slots. This method enables a single model to generate reusable actions across slots based on the target slot, explicitly distinguishes process stages within the model, and executes action blocks with discrete gripper events and confidence constraints, thereby balancing the generalization ability for long-term tasks with the precision of fine operation. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the existing technology. In order to solve the problems that the single model cannot generalize across multiple similar slots, the gripper is prone to oscillation in the fine stage, and the open-loop execution cannot correct the end deviation in real time, a multi-slot optical module loading and unloading control method and system based on visual language action model is adopted.
[0009] A method for controlling the loading and unloading of multi-slot optical modules based on a visual language action model includes the following steps: Acquire current observations containing at least one scene image, robot state, gripper state, and target slot task information; Based on the target slot task information, workpiece coordinate system, and the visual features of the light module and slot in the scene image, construct the geometric conditions of the target slot. The current observation and the target slot geometry are input into a trained process stage conditional visual language action model, which then outputs the current process stage, robotic arm action block, gripper discrete event, and execution confidence. Based on the current process stage, the effective motion dimension and motion amplitude range of the robotic arm motion block are defined, and the allowed event set of the gripper discrete events is defined. Control the gripper state switching according to the gripper event latching rules, and maintain the previous stable state of the gripper when the state switching conditions are not met; The constrained robotic arm action block is converted into a control target that the robot can execute. After executing part of the action block, new observations are acquired again. Based on the execution confidence, the consistency between the newly generated action block and the unexecuted part of the original action block, the stage completion condition, and the safe stopping condition, it is determined whether to continue execution, truncate and replan, or stop safely.
[0010] Furthermore, the geometric conditions of the target slot include at least one of the following: target slot identifier embedding, target slot spatial position embedding in the workpiece coordinate system, and optical module relationship embedding relative to the target slot. The spatial position of the target slot can be mapped to a unified coordinate expression through calibration transformation between the robot base coordinate system, workpiece coordinate system, and camera coordinate system; the relationship embedding of the optical module relative to the target slot can be obtained from the position difference, posture difference, or visual feature relationship between the two, enabling the same visual language action model to distinguish multiple slots with similar appearances.
[0011] Furthermore, the current process stage jointly output by the process stage conditional model includes at least multiple stages such as approach, closing gripper, stable lifting, handling, placement, release, and withdrawal; the validity constraint based on the current process stage specifically means: limiting the allowed robot arm motion dimensions, motion amplitude range, and allowed set of discrete gripper events for output in that stage.
[0012] Furthermore, the discrete events of the gripper are selected from the set of open, close, hold, and release. A gripper state switch is performed only when the discrete event matches the current process stage, the execution confidence of the event reaches a preset threshold, and the current gripper state has been maintained for a preset duration; otherwise, the previous stable state of the gripper is maintained. After the gripper closes, it remains closed until the process stage allowing release is entered; after the gripper is released, it remains open until retraction is complete.
[0013] Furthermore, the robotic arm motion block includes a standard motion component determined by the target slot task information and the current process stage, and a residual motion component generated based on the real-time geometric error of the optical module relative to the gripper or the optical module relative to the target slot; the combination result of the standard motion component and the residual motion component is converted into the joint target, joint increment or end pose target of the target robot by the body adapter decoder.
[0014] Furthermore, after executing a portion of the action block, the newly predicted action block is compared with the unexecuted portion of the original action block for consistency. Consistency is determined based on at least one of the following: action space distance, end-effector trajectory deviation, and action distribution divergence. Stage completion conditions are determined based on at least one of the following: whether the optical module or robot end-effector has entered the target space region corresponding to the current stage; whether the gripper state meets the stage expectation; stable gripping state; whether the optical module has entered the target slot and remained stable after release; and whether the robot end-effector has completed withdrawal. Execution continues if the safety stop condition is not triggered and the consistency, execution confidence, action distribution boundary, and corresponding stage completion conditions meet preset conditions; if the safety stop condition is not triggered but any preset condition is not met, execution is halted and replanning is performed; safe stopping occurs when a collision or contact force exceeding the limit, joint state exceeding the limit, control target abrupt change, target loss timeout, state or communication timeout, optical module slippage or drop, gripper state mismatch with the stage, or the number of replanning attempts or the duration of no progress exceeds a preset upper limit.
[0015] Furthermore, a method for training the visual language action model described above includes: synchronizing real machine teaching data for loading and unloading multi-slot optical modules in time; automatically generating process stage labels, gripper event labels, and task result labels corresponding to each time step based on the changes in gripper opening, robotic arm motion state, optical module displacement, and target slot position information in the synchronized teaching data; performing balanced sampling on samples from different target slots, and applying random perturbations to the target slot spatial position, camera observation, and robot initial state in the samples to form enhanced training samples; and jointly training the visual language action model using the enhanced training samples and corresponding labels to simultaneously predict process stages, robotic arm action blocks, gripper discrete events, and execution confidence.
[0016] Furthermore, the optimization objectives of the joint training include action block loss, stage classification loss, gripper event loss, adjacent action block consistency loss, slot geometric consistency loss, and task result loss.
[0017] This invention also provides a multi-slot optical module loading and unloading control system employing the above-described control method, comprising at least one of a scene camera and a wrist camera, a robot controller, a robot joint encoder, a gripper driver, a task input device, at least one processor, a memory, and a communication interface. The memory stores a trained process-stage conditional visual language action model and a program executable by the processor; the processor performs target slot geometry construction, model reasoning, gripper event latching, and action block execution and replanning functions; the communication interface connects the processor, the robot controller, and the gripper driver.
[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.
[0019] Compared with the prior art, the present invention has the following technical advantages: By employing the above technical solution, the geometric conditions of the target slot are constructed in the model input, enabling the same model to distinguish multiple slots with similar appearances and reuse motion generation capabilities. Through the conditional output of the process stage and its constraints on motion boundaries and discrete events of the gripper, combined with gripper event latching rules, the risk of repeated opening and closing, premature release, or failure to release of the gripper caused by frame-by-frame prediction fluctuations can be reduced. Through re-observation after executing part of the motion, and hierarchical decision-making based on motion consistency, execution confidence, stage completion conditions, and safe stopping conditions, a balance can be achieved between motion block inference efficiency and end-point deviation closed-loop correction, thereby improving the stability and recoverability of the multi-slot optical module loading and unloading process. Attached Figure Description
[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the process stage conditional visual language action model and action block closed-loop decision-making in an embodiment of this application. Figure 2 This is a schematic diagram of the closed-loop execution process of the model training and deployment phases in an embodiment of this application. Figure 3 This is a schematic diagram of the hardware structure of the multi-slot optical module loading and unloading control system according to an embodiment of this application. Detailed Implementation
[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0022] Example 1 Please refer to Figure 1 and Figure 2 In this embodiment of the invention, a method for controlling the loading and unloading of multi-slot optical modules based on a visual language action model includes the following steps: Step S1: Multimodal observation acquisition and slot geometry construction.
[0023] Acquire current observations containing at least one scene image, robot joint states, gripper states, and target slot task information. Based on the target slot task information in the current observations, and combining the workpiece coordinate system, the position of the target slot in the workpiece coordinate system, and the visual features of the optical module and slot in the image, construct the target slot geometric conditions. These target slot geometric conditions include at least one of target slot identifier embedding, target slot spatial position embedding, and optical module-to-target slot relationship embedding. The robot base coordinate system, workpiece coordinate system, and camera coordinate system are mapped to a unified coordinate representation through pre-calibrated coordinate transformations, enabling the visual language action model to distinguish multiple visually similar slots within the same model without requiring independent model training for each slot or a fixed complete trajectory.
[0024] Step S2: Generation of process stage conditional actions.
[0025] Multi-view visual features, linguistic features, robot state features, and the aforementioned target slot geometric conditions are input into a visual-linguistic-motion model. This model is a process stage conditional model, which includes a process stage prediction head that can jointly output the current process stage, robot arm action blocks, gripper discrete events, and execution confidence. The process stage includes at least several phases such as approach, gripper closing, stable lifting, handling, placement, release, and withdrawal. The stage prediction results are fed back to the motion generation network, limiting the allowed robot arm motion dimensions, motion amplitude range, and allowed set of output gripper discrete events for different stages.
[0026] Step S3: Joint prediction of continuous actions and discrete gripper events.
[0027] The model's motion generation network outputs robotic arm motion blocks and gripper discrete events simultaneously. The gripper discrete events are selected from a set consisting of open, close, hold, and release events. The robotic arm motion blocks include canonical motion components determined by the task and process stages, and residual motion components generated based on real-time geometric errors of the optical module relative to the gripper or the optical module relative to the target slot. These two types of components are combined by a body-adaptive decoder to convert into joint targets, joint increments, or end-effector pose targets for the target robot.
[0028] Step S4: Grip event latching.
[0029] The gripper state is switched only when the current output of the model's discrete gripper event matches the current process stage, the execution confidence of the event reaches a preset threshold, and the current gripper state has been maintained for a preset duration; otherwise, the gripper's previous stable state is maintained. After a closing event is latched, the gripper remains closed until a release-allowed process stage is entered and the release condition is met; after a release event is latched, the gripper remains open until the evacuation stage is completed. This reduces the risk of repeated opening and closing, premature release, or failure to release after placement caused by frame-by-frame output fluctuations.
[0030] Step S5: Closed-loop execution and replanning of action blocks.
[0031] After executing the first few actions in a robotic arm action block, new observations are acquired and a new action block is generated. The consistency of the new action block is compared with the unexecuted portion of the original action block, and a decision is made based on the priority of safe stop, truncation and replanning, or continued execution. If a safe stop condition is triggered, the current action block is immediately terminated and enters a safe state; if the safe stop condition is not triggered but the consistency, execution confidence, action distribution boundary, or stage completion condition corresponding to the current process stage does not meet preset requirements, the original action block is truncated and inference is recalculated based on the new observations; otherwise, the remaining portion of the original action block continues to be executed.
[0032] Step S6: Model training.
[0033] The machine teaching data for loading and unloading multi-slot optical modules is synchronized in time. Based on the changes in gripper opening, robotic arm motion state, optical module displacement, and target slot position information in the teaching data, process stage labels, gripper event labels, and task result labels are automatically generated for each time step. Samples from different target slots are uniformly sampled, and random perturbations are applied to the target slot spatial position, camera observation, and robot initial state to form enhanced training samples. The visual language action model is jointly trained using the enhanced training samples and corresponding labels. The joint optimization objectives include action block loss, stage classification loss, gripper event loss, adjacent action block consistency loss, slot geometric consistency loss, and task result loss. Through multi-slot equalization and geometric perturbation training, the model learns transferable slot condition relationships, rather than memorizing a few taught joint sequences.
[0034] Example 2 This embodiment, based on embodiment 1, provides a detailed explanation of the determination methods for consistency, phase completion, and safe stopping. This method can be deployed on dual-arm or single-arm robots.
[0035] Step S1: Multimodal Observation and Slot Geometry Construction. Acquire scene camera images, wrist camera images, robot joint states, gripper states, and optional end-effector force information; parse the target slot identifier and operation type from the language task. Construct slot geometry conditions based on the workpiece coordinate system, the target slot's position in the workpiece coordinate system, and the visual features of the module / slot in the image. Slot geometry conditions must include at least one of the following: target slot identifier embedding, target slot spatial embedding, and module-relative-target slot relationship embedding.
[0036] Step S2: Conditional coding of process stages.
[0037] Multi-view visual features, linguistic features, robot state features, and slot geometry are input into a shared visual-linguistic encoding network, and the head output stage probability is predicted by the process stage. The process stages include at least several stages such as approach, alignment, gripper closure, stabilization lifting, handling, placement, release, and withdrawal. The stage probabilities or stage embeddings are fed back to the motion generation network, enabling different stages to employ different effective motion dimensions, motion amplitude ranges, and gripper event sets.
[0038] Step S3: Joint prediction of continuous motions and discrete gripper events. The motion generation network outputs the robot arm motion block, gripper events, motion confidence, and optional stage completion probability at the same time. Gripper events are selected from a discrete set consisting of open, close, hold, and release, and are not directly driven by continuous gripper values. The robot arm motion block is a combination of ontology-independent canonical motions and residual motions corresponding to the current geometric error, which is converted into the joint target, joint increment, or end-effector pose target of the target robot by the ontology-adaptive decoder.
[0039] Step S4: Grip Event Latching. The open, closed, or release state switch is only executed if the gripper event is allowed during the prediction phase, the event confidence meets the threshold, and the previous gripper state is maintained for the set duration; otherwise, the previous stable state is maintained. After closing, the gripper remains closed until the placement phase begins; after releasing, the gripper remains open until evacuation is complete, eliminating frame-by-frame gripper oscillations from the algorithm output layer.
[0040] Step S5: Closed-loop execution of the action block. Let the K unexecuted actions in the original action block form the sequence Arem, and the first K predicted actions corresponding to the new observation form the sequence Anew. The action space distance can be expressed as the average of the normalized joint target difference, joint increment difference, or end-effector difference over K time steps; the end-effector trajectory deviation can be expressed as the weighted sum of the position and attitude deviations at the corresponding time steps; when the model outputs the action probability distribution, the action distribution divergence can be KL divergence, JS divergence, or other distribution distances. Any of the above indicators can be selected, or multiple normalized indicators can be weighted and combined into a consistency indicator, and consistency is determined when this indicator is not greater than the corresponding threshold. Each threshold is preset based on the robot's motion range, control cycle, and validation set statistical results, and is not limited to a fixed value. The completion conditions for each stage correspond to the current process stage: the approach or alignment stage can be determined based on the robot end effector or optical module entering the target space area and the pose error; the closing gripper or stable lifting stage can be determined based on the gripper state, contact state, the pose stability of the optical module relative to the end effector, and the lifting height; the placement or release stage can be determined based on the optical module entering the space area corresponding to the target slot, the gripper opening, and the optical module remaining stable after release; the withdrawal stage can be determined based on the distance between the robot end effector and the target slot.
[0041] Safety stop conditions include, but are not limited to: collision signals or contact forces exceeding safety thresholds, joint positions, velocities, or accelerations exceeding robot safety boundaries, sudden changes in the control target or exceeding the preset workspace, target loss reaching a preset duration, observation timestamps expiring or communication timeouts, gripper status mismatch with the current process stage, detection of optical module slippage or drop, and replanning counts or no-progress duration exceeding a preset upper limit. Upon triggering a safety stop condition, the control system terminates the current action block, maintaining the robot in its current safe pose or executing a preset evacuation action; while still carrying the optical module, the gripper remains closed; when release is complete, the gripper remains open, and alarm information including the trigger condition, process stage, and robot status is output.
[0042] Step S6: Training Sample Construction and Multi-Task Optimization. Synchronize the vision, state, and actions during real-machine teaching. Generate stage labels, gripper event labels, and task result labels based on gripper opening changes, robotic arm speed, optical module displacement, target slot position, and optional contact signals. Perform balanced sampling on samples from each slot, and apply perturbations to slot identification, slot spatial position, image cropping, camera extrinsic parameters, and the robot's initial joint state. Jointly train at least several of the following loss methods: action block loss, stage classification loss, gripper event loss, adjacent action block consistency loss, slot geometric consistency loss, and task result loss. The basic visual language network can be frozen or partially updated, and the action generation network, stage prediction head, gripper event head, and ontology adapter decoder participate in at least some of the training.
[0043] Step S7: Task Completion Determination. The placement phase determines completion based on at least whether the optical module has entered the target slot's corresponding spatial area and whether the gripper is in a released state; in one embodiment, the placement task is determined to be complete only when the optical module enters the spatial area, the gripper has released, and the robot's end effector has withdrawn. The grasping phase determines completion based at least on one of the following: the stability of the optical module's pose relative to the end effector after the gripper closes, the contact state, or the lifting displacement of the optical module caused by the end effector. Failure samples retain the failure stage and failure type for classification, result prediction, or candidate action ranking during the training phase.
[0044] The technical advantages of the above solution are: A single model can handle multiple placement locations using slot conditions; Phase conditionalization and gripper event latching reduce gripper anomalies caused by continuous regression; Geometric residuals and motion block replanning improve grasping, placement, and disturbance recovery capabilities; Automatic stage labeling and multi-slot equalization enable existing real-device teaching to generate visual language action model training data specific to optical module processes. The base model can be a model with visual language encoding and continuous action generation capabilities, and the action generation network can adopt diffusion, flow matching, or autoregressive structures; the base visual language network can be trained with full parameters, partial parameters, or low-rank adaptation. This invention does not limit the specific name and parameter scale of the base model.
[0045] Example 3 This embodiment provides a multi-slot optical module loading and unloading control system using the method described in Embodiment 1. For example... Figure 3 As shown, the control system includes at least one of a scene camera and a wrist camera, a robot controller, a robot joint encoder, a gripper driver, a task input device, at least one processor, a memory, and a communication interface; it may also include an end effector force sensor if necessary.
[0046] At least one of the scene camera and the wrist camera constitutes a vision acquisition module for acquiring scene images or end-effector local images; the robot controller acquires joint position and joint speed from the robot joint encoder, the gripper driver provides gripper opening or gripper state, and an optional end-effector force sensor provides contact force or torque; the task input device is a host computer, a human-machine interface terminal, or an industrial control computer for receiving target slot task information.
[0047] The processor is a central processing unit, a graphics processing unit, an artificial intelligence accelerator, or a combination thereof; the memory stores a trained process-stage conditional visual language action model, slot geometry parameters, robot safety boundaries, and programs executable by the processor. The processor performs visual feature extraction, robot state processing, target slot geometry construction, process-stage conditional model inference, gripper event latching, and action block execution and replanning functions. These functions can run on the same computing device or distributed across multiple computing devices connected via a communication network.
[0048] The gripper event latching function receives the gripper discrete events, execution confidence, current process stage, and gripper feedback status output by the model, and determines whether to send a state switching command to the gripper driver based on the gripper event latching rules.
[0049] The action block execution and replanning function triggers a reobservation after executing part of the action block of the robotic arm, and determines whether to continue execution, truncate replanning, or safely stop based on execution confidence, action consistency, stage completion conditions, and safe stop conditions.
[0050] The communication interface adopts at least one of Ethernet, fieldbus, robot manufacturer control interface or robot middleware interface, and connects the processor, robot controller, gripper driver, camera and optional end force sensor; the robot controller sends the joint target, joint increment or end pose target determined by the processor to the robot execution unit.
[0051] The aforementioned hardware and its functional modules work together to execute the control method described in Embodiment 1 or Embodiment 2.
[0052] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.
[0053] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.
Claims
1. A method for controlling the loading and unloading of multi-slot optical modules based on a visual language action model, characterized in that, Includes the following steps: Acquire current observations containing at least one scene image, robot state, gripper state, and target slot task information; Based on the target slot task information, workpiece coordinate system, and the visual features of the light module and slot in the scene image, construct the geometric conditions of the target slot. The current observation and the target slot geometry are input into a trained process stage conditional visual language action model, which then outputs the current process stage, robotic arm action block, gripper discrete event, and execution confidence. Based on the current process stage, the effective motion dimension and motion amplitude range of the robotic arm motion block are defined, and the allowed event set of the gripper discrete events is defined. Control the gripper state switching according to the gripper event latching rules, and maintain the previous stable state of the gripper when the state switching conditions are not met; The constrained robotic arm action block is converted into a control target that the robot can execute. After executing part of the action block, new observations are acquired again. Based on the execution confidence, the consistency between the newly generated action block and the unexecuted part of the original action block, the preset stage completion conditions, and the safe stop conditions, it is determined whether to continue execution, truncate and replan, or stop safely.
2. The multi-slot optical module loading and unloading control method based on visual language action model according to claim 1, characterized in that, The geometric conditions for constructing the target slot include: Based on the target slot identifier and workpiece coordinate system in the target slot task information, construct at least one of the following: target slot identifier embedding, target slot spatial position embedding, and optical module relative to target slot relationship embedding. The target slot geometry conditions enable the visual language action model to distinguish multiple slots with similar appearances within the same model.
3. The multi-slot optical module loading and unloading control method based on visual language action model according to claim 1, characterized in that, The current process stage includes at least several stages including approach, alignment, closing the gripper, stabilizing lifting, handling, placing, releasing and withdrawing; The validity constraint based on the current process stage specifically means: limiting the allowed robot arm movement dimensions, movement amplitude range, and allowed set of discrete gripper events for output at this stage.
4. The multi-slot optical module loading and unloading control method based on visual language action model according to claim 1, characterized in that, The discrete events of the gripper are selected from a set consisting of open, close, hold, and release; The specific rules for latching the gripper event are as follows: gripper state switching is performed only when the gripper discrete event matches the current process stage, the execution confidence of the gripper discrete event reaches a preset threshold, and the current state of the gripper has been maintained for a preset duration; otherwise, the previous stable state of the gripper is maintained; after the gripper is closed, it remains closed until the process stage that allows release is entered, and after the gripper is released, it remains open until the withdrawal is completed.
5. The multi-slot optical module loading and unloading control method based on visual language action model according to claim 1, characterized in that, The robotic arm motion block includes standard motion components determined by the target slot task information and the current process stage, as well as residual motion components generated based on the real-time geometric error of the optical module relative to the gripper or the optical module relative to the target slot. The method further includes converting the combination result of the canonical motion component and the residual motion component into a joint target, joint increment, or end-effector pose target of the target robot through a body adaptation decoder.
6. The multi-slot optical module loading and unloading control method based on visual language action model according to claim 1, characterized in that, The decision to continue execution, truncate replanning, or safely stop specifically includes: The consistency of the new action block generated based on the new observation is compared with the unexecuted part of the original action block; the consistency is determined based on at least one of the action space distance, end trajectory deviation and action distribution divergence, and the consistency is determined to be satisfied when the adopted consistency index is not greater than the corresponding preset threshold. The completion conditions of the stage include at least one of the following conditions: the optical module or robot end effector enters the target space area corresponding to the current process stage; the gripper state meets the expected state of the current process stage; the pose change or contact state of the optical module relative to the robot end effector meets the stable gripping conditions; the optical module enters the space area corresponding to the target slot and remains stable after the gripper releases; and the robot end effector completes withdrawal. If the safety stop condition is not triggered, and the consistency, execution confidence, action distribution boundary, and stage completion condition corresponding to the current process stage meet their respective preset conditions, the remaining part of the original action block continues to be executed; if the safety stop condition is not triggered but any of the preset conditions are not met, the original action block is truncated and reasoning is re-inferred based on the new observation. The safety stopping conditions include at least one of the following: the collision signal or contact force exceeds the safety threshold, the joint position, joint speed or joint acceleration exceeds the safety boundary, the control target changes abruptly or exceeds the preset workspace, the target is lost for a preset duration, the status data or communication times out, the gripper status does not match the current process stage, the optical module is detected to slip or fall, and the number of replanning attempts or the duration of no progress exceeds the preset upper limit. When the safety stop condition is triggered, the current action block is terminated, so that the robot maintains the current safe posture or performs a preset evacuation action, maintains the gripper state adapted to the carrying state of the optical module, and outputs alarm information.
7. A method for training a visual language action model according to any one of claims 1 to 6, characterized in that, include: Synchronize the real machine teaching data for loading and unloading multi-slot optical modules in time; Based on the changes in gripper opening, robotic arm motion status, optical module displacement, and target slot position information in the synchronized teaching data, the system automatically generates process stage labels, gripper event labels, and task result labels for each time step. Equal sampling is performed on samples from different target slots, and random perturbations are applied to the spatial position of the target slots, camera observations, and the robot's initial state in the samples to form enhanced training samples; The visual language action model is jointly trained using the enhanced training samples and corresponding labels, enabling it to simultaneously predict process stages, robotic arm action blocks, gripper discrete events, and execution confidence.
8. The method according to claim 7, characterized in that, The optimization objectives of the joint training include at least several of the following: action block loss, stage classification loss, gripper event loss, adjacent action block consistency loss, slot geometric consistency loss, and task result loss.
9. A multi-slot optical module loading and unloading control system, characterized in that, The system includes at least one of a scene camera and a wrist camera, a robot controller, a robot joint encoder, a gripper actuator, a task input device, at least one processor, a memory, and a communication interface; the memory stores a trained process-stage conditional visual language motion model and a program executable by the processor. At least one of the scene camera and wrist camera is used to constitute a vision acquisition module; the robot controller, robot joint encoder, and gripper driver are used to constitute a robot state acquisition module; the processor is used to run a task input module, a target slot geometry construction module, the visual language action model, a gripper event latching module, and an action block execution and replanning module; the communication interface connects the processor, robot controller, and gripper driver to enable the control system to execute the method described in any one of claims 1 to 6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method described in any one of claims 1 to 8.