Palletizing and transferring task plan iterative planning method and system based on digital twin multi-modal feedback

Abstract

The application discloses a palletizing and transferring task plan iterative planning method and system based on digital twin multi-modal feedback, relates to the technical field of industrial robot task iterative planning, and comprises the following steps: acquiring a current structured task plan, and analyzing and identifying the action semantics of each action step; converting each action step into a corresponding motion verification task according to the action semantics, obtaining a motion verification task sequence, and then gradually executing the motion verification in sequence in a digital twin verification environment; if the verification fails in the verification process, generating structured error information and virtual environment visual feedback information corresponding to the current action step; taking the original task target, the current structured task plan, the structured error information and the virtual environment visual feedback information as multi-modal feedback information, regenerating a structured task plan by a task plan generation subject, and then jumping to the analysis step for continuous iteration until a stop condition is met, and outputting the current structured task plan.

Description

Technical Field

[0001] This application relates to the field of industrial robot task iterative planning technology, and in particular to a palletizing and transfer task planning iterative planning method and system based on digital twin multimodal feedback. Background Technology

[0002] In industrial palletizing, loading / unloading, and transfer scenarios, collaborative operations involving multiple robots, transfer platforms, fixtures, and target palletizing locations are becoming increasingly common. These scenarios typically involve multi-step transfers of workpieces from the loading area to the transfer platform and from the transfer platform to the target palletizing location, requiring consistency between robot movements, platform movements, workpiece status, and the palletizing space. With the increasing demands for flexible manufacturing and small-batch, multi-variety production, traditional fixed-program robot task configuration methods are struggling to quickly adapt to changing scenarios.

[0003] The combination of large-scale visual language models, visual language action models, and retrieval-enhanced generation with large-scale language models can directly generate task plans based on on-site images, task objectives, or natural language instructions. These models possess strong semantic understanding and adaptability to open scenarios and have begun to be introduced into industrial task planning scenarios. However, these models suffer from several issues when generating robot task plans, including spatial understanding biases, insufficient consideration of collisions and interference, inconsistencies in state consistency between multiple steps, inconsistent timing and poses of multiple executing entities in relay coordination, and failure to consider the occupancy status of stacking positions in stacking tasks.

[0004] Existing robot motion planning and digital twin verification systems typically only verify given trajectories and programs. Their inputs are mostly low-level robot trajectories, target poses, or program instructions, lacking semantic parsing mechanisms for generating structured task plans for upper-level visual language models. Even if unreachability, collisions, or state anomalies can be detected, they usually lack structured descriptions of failure reasons for upper-level model replanning, visual feedback of failure scenarios, and multimodal closed-loop feedback mechanisms that form with upper-level generative models.

[0005] Therefore, there is a semantic gap between the upper-level task planning and the lower-level motion verification. The industry lacks a method that can automatically map the structured task plan generated by the large visual language model to the motion verification task, and feed back the reasons for verification failure to the upper-level model in a multimodal form of structured errors plus digital twin screenshot annotations, thereby supporting closed-loop iterative planning. Summary of the Invention

[0006] The purpose of this application is to provide a method and system for iterative planning of palletizing and transport tasks based on digital twin multimodal feedback. This method can perform motion verification of structured task plans in a digital twin verification environment, and trigger the task plan generation subject to perform iterative replanning through structured error information and virtual environment visualization feedback information, thereby improving the executability, security and stability of the task plan.

[0007] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, comprising the following steps: Obtain a structured task plan generated based on the task objective, and use the structured task plan as the current structured task plan; the structured task plan contains multiple action steps arranged in execution order, and each action step includes at least the execution subject, action semantics, and operation object.

[0008] The current structured task plan is parsed, the action semantics of each action step are identified, and each action step is converted into a corresponding motion verification task based on the action semantics to obtain a motion verification task sequence.

[0009] Motion verification is performed step by step in the motion verification task sequence within the digital twin verification environment.

[0010] If the current action step fails to be verified during the verification process, a structured error message and a virtual environment visualization feedback message corresponding to the current action step are generated; the virtual environment visualization feedback message includes at least a screenshot of the virtual environment and the annotation information on the screenshot of the virtual environment.

[0011] The original task objective, the current structured task plan, the structured error information, and the virtual environment visualization feedback information are fed back to the task plan generation entity as multimodal feedback information to regenerate the structured task plan.

[0012] The regenerated structured task plan is used as the current structured task plan. The process jumps to the step "parse the current structured task plan, identify the action semantics of each action step, and convert each action step into a corresponding motion verification task according to the action semantics to obtain a motion verification task sequence" until the stopping condition is met. Then, the current structured task plan is output or a failure report is output and manual review is triggered.

[0013] Optionally, each action step in the structured task plan further includes at least one of source pose, target pose, constraints, and step dependencies; the action semantics include at least one of grasping, placing on the transfer platform, platform movement, grasping from the transfer platform, and placing on the target position; each action step is converted into a corresponding motion verification task according to the action semantics to obtain a motion verification task sequence, specifically including the following steps: For action steps with the action semantics of "grab", generate motion verification tasks that include approaching, clamp alignment, grabbing, attaching to the operation object, and withdrawing.

[0014] For action steps with the action semantics of "place on the transfer platform", generate motion verification tasks that include moving to the platform placement position, releasing the operation object, and withdrawing.

[0015] For action steps with the action semantics of "platform movement", generate a motion verification task that includes the transfer platform moving along the constraint axis and carrying the operation object.

[0016] For action steps with the semantic meaning of "grab from the transfer platform", generate a motion verification task that includes the robot grabbing the operation object from the platform handover position.

[0017] For action steps with the action semantics of "place to target location", generate a motion verification task that includes moving to target location, placing, releasing and withdrawing.

[0018] Optionally, before performing motion verification step-by-step according to the motion verification task sequence in the digital twin verification environment, the following steps are also included: In the digital twin verification environment, the robot object, transfer platform object, operation workpiece object, fixture object, obstacle object and target position object participating in the task execution are bound together. The mapping relationship between each field in the current structured task plan and each digital twin verification environment object is established, and the geometric model, kinematic model, coordinate system, state variables, collision attributes and constraint parameters of each digital twin verification environment object are bound together.

[0019] Optionally, motion verification is performed step-by-step in the digital twin verification environment according to the motion verification task sequence, specifically including the following steps: Verify whether the robot's end effector can reach the target pose or intermediate pose.

[0020] Verify whether there are collisions between the robot bodies, between the robot and the transfer platform, between the robot and the workpiece being operated, and between the robot and obstacles during the movement process.

[0021] Verify whether the travel, speed, limit, and load status of the transfer platform, as well as the follow-up relationship of the operated workpiece, meet the constraints.

[0022] Verify whether the relationship between the workpiece and its state changes is consistent before and after the action.

[0023] Verify whether the state of the workpiece released by the previous executing entity and the state of the workpiece grasped by the subsequent executing entity are continuous in space and time.

[0024] Verify whether the target position is occupied, whether the target orientation meets the stacking requirements, and whether there are spatial conflicts between the manipulated workpieces.

[0025] Optionally, the structured error information includes at least one of the following: verification result identifier, failed action step identifier, failed execution subject, failed action, involved object, failure type, failure reason, modification suggestion, and associated screenshot identifier; the failure type includes at least one of the following: target pose unreachable, motion path collision, inconsistent operation workpiece relationship state, platform target position exceeding limit, target position occupied, handover sequence or pose mismatch, and fixture posture mismatch.

[0026] Optionally, generating virtual environment visual feedback information corresponding to the current action step specifically includes the following steps: During the current action step verification process, record the viewpoint parameters, poses of each digital twin verification environment object, collision geometry, and relational state variables of the manipulated workpiece.

[0027] Based on the pose of the object involved in the current action step in the digital twin verification environment, select the observation viewpoint, trigger the screenshot rendering of the viewpoint, and obtain a screenshot of the virtual environment.

[0028] Based on the failure type, the corresponding visualization overlay rule is matched from the preset annotation template library, and annotation information is overlaid on the virtual environment screenshot, outputting the virtual environment screenshot and the annotation information on the virtual environment screenshot.

[0029] Optionally, the stopping condition includes at least one of the following: All actions and steps in the digital twin verification environment have been verified.

[0030] The number of replanning iterations reached the preset maximum number of iterations.

[0031] The current failure type is a preset, non-automatically recoverable error type.

[0032] The confidence level of the task plan generator's output is lower than the preset threshold.

[0033] The number of action steps that passed verification did not increase in consecutive iterations.

[0034] Triggering preset conditions for manual review.

[0035] Optionally, the task plan generation entity can be any one of the following: a large visual language model, a visual language action model, a combination of retrieval enhancement generation and a large language model, an expert system, a rule system, or a process planning system.

[0036] Optionally, when performing motion verification step by step according to the motion verification task sequence in the digital twin verification environment, a first strategy is adopted to perform motion verification: when any motion step verification failure is detected, the current round of verification is immediately interrupted and the structured error information of the failed motion step is output, and the feedback and replanning process is entered.

[0037] Alternatively, a second strategy can be used to perform motion verification: after verifying all motion steps, all failed motion steps are summarized into a structured error message list and output uniformly.

[0038] Secondly, this application provides an iterative planning system for palletizing and transporting tasks based on digital twin multimodal feedback, including the following functional modules: The task plan access module is used to obtain a structured task plan generated based on the task objective and use the structured task plan as the current structured task plan; the structured task plan contains multiple action steps arranged in execution order, and each action step includes at least the execution subject, action semantics and operation object.

[0039] The structured task plan parsing module is used to parse the current structured task plan and identify the action semantics of each action step.

[0040] The motion verification task generation module is used to convert each action step into a corresponding motion verification task based on the action semantics, thereby obtaining a motion verification task sequence.

[0041] The motion feasibility and collision verification module is used to perform motion verification step by step in the motion verification task sequence in the digital twin verification environment; if the current action step fails to verify during the verification process, the structured error generation module and the digital twin screenshot and annotation module are called.

[0042] The structured error generation module is used to generate structured error information corresponding to the current action step.

[0043] The digital twin screenshot and annotation module is used to generate virtual environment visualization feedback information corresponding to the current action step; the virtual environment visualization feedback information includes at least a virtual environment screenshot and annotation information on the virtual environment screenshot.

[0044] The iterative replanning module is used to feed back the original task objective, the current structured task plan, the structured error information, and the virtual environment visualization feedback information as multimodal feedback information to the task plan generation entity to regenerate the structured task plan. The regenerated structured task plan is used as the current structured task plan, and the structured task plan parsing module, motion verification task generation module, and motion feasibility and collision verification module are called sequentially until the stopping condition is met. The current structured task plan is then output or a failure report is output and manual review is triggered.

[0045] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides an iterative planning method and system for palletizing and transporting task planning based on digital twin multimodal feedback. The method first acquires a structured task plan as the current structured task plan. This plan contains action steps arranged in execution order, with each step including at least the executing subject, action semantics, and operation object. By parsing the current plan and identifying the action semantics of each step, it is converted into a motion verification task sequence, thereby establishing an interface mapping between the semantic-level task plan and the underlying motion verification, eliminating the semantic gap between the two, and improving the adaptability of the task plan and the motion system. Subsequently, motion verification is executed step by step in the digital twin verification environment, avoiding the safety risks and debugging costs associated with direct execution on the actual robot. When a certain action step fails verification, structured error information containing the failed action step identifier, failure type, failure reason, and modification suggestions is generated. Simultaneously, a digital twin screenshot and annotation image of the corresponding failed step are automatically generated. This structured error information allows for precise location of the failure reason and machine-readable verification. The screenshots and annotations visually present spatial conflicts and unreachable areas. Combining these two elements fully leverages the multimodal understanding capabilities of the task plan generator, enhancing its awareness of spatial relationships in failure scenarios and thus improving the accuracy and interpretability of replanning and correction. Based on this, the original task objective, the current structured task plan, structured error information, and screenshots and annotations of the virtual environment are fed back to the task plan generator as multimodal feedback information. This allows the generator to perform overall replanning or local corrections while preserving the original task constraints. The regenerated plan then serves as the current plan, jumping back to the parsing step to continue iterating until the stopping condition is met, at which point the current structured task plan is output. This forms a closed-loop iterative mechanism of "generation—verification—multimodal feedback—replanning," ensuring that the final output task plan is executable, secure, and stable under digital twin verification. Furthermore, the entire iterative process is completed in a virtual environment, eliminating the need for real equipment resources and effectively saving on-site debugging costs. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a schematic diagram of a multi-robot palletizing and transfer scenario.

[0048] Figure 2 This is a flowchart illustrating an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, provided as an embodiment of this application.

[0049] Figure 3 This is a schematic diagram of JSON code illustrating a structured task plan example in an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, provided as an embodiment of this application.

[0050] Figure 4 This is a schematic diagram illustrating the transformation of the motion verification task generation module in an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, provided in an embodiment of this application.

[0051] Figure 5 This is a schematic diagram illustrating the transformation of the workpiece relationship state management module in an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, provided as an embodiment of this application.

[0052] Figure 6 This is a schematic diagram of JSON code illustrating an example of structured error information in an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, provided as an embodiment of this application.

[0053] Figure 7 This is a schematic diagram of the multimodal feedback interaction process in an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, provided as an embodiment of this application.

[0054] Figure 8 This is a schematic diagram of the functional modules of a palletizing and transfer task planning iterative planning system based on digital twin multimodal feedback, provided as an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0057] The task plan verification and iterative feedback method provided in this application is applicable to industrial operation scenarios such as palletizing, loading and unloading, multi-robot relay transfer, and stacking. In an exemplary palletizing and transfer scenario, such as... Figure 1 As shown, the first robot (robot_1) picks up the workpiece from the loading area and places it on the XY transfer platform; the XY transfer platform moves the workpiece to the handover position; the second robot (robot_2) picks up the workpiece from the XY transfer platform and places it on the target palletizing position. The digital twin verification environment verifies the motion of the above process step by step; if the verification fails, it sends structured error information, virtual environment screenshots, and annotations back to the task plan generation entity, which then regenerates the task plan.

[0058] This application provides an iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback. In one exemplary embodiment, such as... Figure 2 As shown, the method includes the following steps: A1. Obtain the structured task plan generated based on the task objective, and use the structured task plan as the current structured task plan.

[0059] Specifically, the task planning access module obtains the structured task plan generated by the task planning generation entity. The structured task plan contains multiple action steps arranged in execution order. Each action step includes at least an actor, action semantics, and an object. Each action step may also include a source pose, a target pose, optional constraints, and step dependencies. The actor represents the execution device for that step (e.g., a robot, a transfer platform), the action semantics represents the action type corresponding to that step (e.g., grasping, placing, platform movement), the object represents the object operated on in that step (e.g., the workpiece to be transferred), the source pose is the starting position of the action, the target pose is the target position of the action, the constraints are the constraints that the step must satisfy (e.g., safety constraints, cycle time constraints, posture constraints), and the step dependencies are the dependencies between different action steps.

[0060] The task plan generation entity can be any one of the following: a Vision-Language Model (VLM), a Vision-Language-Action Model (VLA), a combination of Retrieval-Augmented Generation (RAG) and Large Language Model (LLM), an expert system, a rule system, or a process planning system. The core of this application is not about generating the task plan itself by a specific type of task plan generation entity, but rather about performing virtual verification of the task plan and achieving a complete closed loop of iterative replanning through structured error information and visual feedback.

[0061] The task planning access module can be implemented in the form of a large visual language model, a visual language action model, a combination of retrieval enhancement generation and large language model, an expert system, a rule system or a process planning system, or a combination of the above methods.

[0062] In a specific example, the visual language big model generates a structured task plan based on the scene images, task objectives, and scene description. An example of a structured task plan is shown below. Figure 3As shown in the figure, `task_type` represents the task type; `steps` is a list of action steps arranged in execution order; in each step, `id` is the step identifier, `actor` is the execution entity of the step, `action` is the action semantics corresponding to the step, `object` is the operation object, `source_pose` is the source pose, `target_pose` is the target pose, `platform` is the transfer platform involved in the step, and `target` is the target location. As an example, the action semantics include at least one of "pick", "place on platform", "platform move", "pick from platform", and "place to pallet".

[0063] Before obtaining the structured task plan, scene information also needs to be acquired. Specifically, the scene information acquisition module obtains the following information from the on-site visual sensors, user input interface, virtual scene description file, and robot model library, transfer platform model library, and workpiece model library: (1) On-site images: These are images taken from on-site cameras, simulation views, screenshots of the digital twin environment, or images uploaded manually, including top views of the work area and robot perspective views; (2) Task objectives: type, quantity, source location, target location, stacking order, posture requirements, safety constraints, and cycle time requirements of the workpieces to be operated; (3) Virtual scene information: robot model, transfer platform model, operation workpiece model, fixture model, obstacle model, target position model, workspace boundary, reachable area, collision geometry and motion constraints.

[0064] A2. Parse the current structured task plan, identify the action semantics of each action step, and convert each action step into a corresponding motion verification task based on the action semantics to obtain a motion verification task sequence.

[0065] The structured task plan parsing module parses the fields of the structured task plan output in step A1, identifies the execution subject, action semantics, operation object, source pose, target pose, transfer platform, target position, constraints, preconditions, poststates, and step dependencies of each action step, and outputs an intermediate representation that can be used by downstream modules.

[0066] The motion verification task generation module (i.e., the interface conversion module) converts each motion step into a corresponding motion verification task or motion planning task sequence based on the parsed motion semantics. This conversion is not a simple field mapping, but rather a motion verification task generated based on motion semantics, which includes the initial pose, key process poses, target pose, constraints, attachment relationship switching conditions, and other related information.

[0067] In one exemplary embodiment, such as Figure 4 As shown, each action step is converted into a corresponding motion verification task based on action semantics, specifically including: A21. For action steps with the action semantics of "pick", generate a motion verification task that includes approaching, clamp alignment, grabbing, attaching to the operation object, and withdrawing.

[0068] A22. For action steps with the action semantics of "place on the transfer platform" (place_on_platform), generate a motion verification task that includes moving to the platform placement position, aligning, placing, releasing the operation object, and withdrawing.

[0069] A23. For action steps with the action semantics of "platform_move", generate a motion verification task that includes motion planning, motion execution, position arrival, workpiece follow-up, and motion stabilization. The transfer platform moves along the constraint axis and carries the operation object.

[0070] A24. For action steps with the action semantics of "pick from platform", generate a motion verification task that includes the robot approaching the platform, aligning, picking up the object, attaching, and withdrawing.

[0071] A25. For action steps with the action semantics "place to target position" (place_to_pallet), generate a motion verification task that includes moving to the target position, aligning, placing, releasing, and withdrawing.

[0072] A3. Perform motion verification step by step in the motion verification task sequence within the digital twin verification environment.

[0073] Before motion verification in this step, the digital twin object binding module first binds the robot objects, transfer platform objects, manipulated workpiece objects, fixture objects, obstacle objects, and target position objects participating in this task within the digital twin verification environment. This establishes a mapping relationship between the execution subject field, manipulated object field, transfer platform field, and target position field in the structured task plan and the digital twin verification environment object instances. It also binds the geometric model, kinematic model, coordinate system, state variables, collision attributes, and constraint parameters of each digital twin verification environment object. For multi-robot scenarios, it also binds the relative positions, shared workspaces, and handover areas between different robots.

[0074] Subsequently, the workpiece relationship state management module updates the workpiece relationship state relative to the world coordinate system, the robot end effector, the transfer platform, and the target position based on the action semantics of each action step. In the example palletizing and transfer scenario, such as... Figure 5 As shown, the relational states of the manipulated workpiece switch sequentially as follows: world_fixed (world fixed, the manipulated workpiece is stationary relative to the world coordinate system) → attached_to_robot_1 (attached to the end effector of the first robot, the workpiece's pose moves with the end effector of the first robot) → supported_on_platform (supported on the transfer platform, the workpiece's pose moves with the carrier component of the transfer platform, and the workpiece moves synchronously with the platform during the XY transfer platform movement) → attached_to_robot_2 (attached to the end effector of the second robot) → placed_on_pallet (placed at the target position). These relational states of the manipulated workpiece are used to ensure the correct attribution, support, attachment, follow-up, and release relationships of the workpiece during the relay process involving multiple execution entities.

[0075] It should be noted that during the period when the workpiece is supported on the transfer platform, in the digital twin verification environment, the workpiece's motion relationship is modeled using a kinematic approach that is rigidly attached to the load-bearing components of the transfer platform. That is, the workpiece's pose is driven by and synchronized with the pose of the load-bearing components of the transfer platform. Correspondingly, the constraint verification of the transfer platform's stroke, speed, limits, load-bearing state, and the workpiece's motion relationship refers to verifying whether the platform's motion parameters and the load-bearing and motion synchronization relationships fall within the preset constraint range under the aforementioned rigid attachment motion model. In other words, the workpiece's pose motion is modeled using a kinematic attachment approach, while the load-bearing and motion constraint verification verifies the platform's motion parameters and constraint boundaries. These two aspects belong to two separate levels: pose motion modeling and constraint verification, and they do not conflict with each other.

[0076] The motion feasibility and collision verification module performs the verification step by step in the digital twin verification environment according to the motion verification task sequence generated in step A2. In an exemplary embodiment, the verification content includes at least the following six aspects: (1) Verify whether the robot end can reach the target pose or intermediate pose, that is, whether the target pose or intermediate pose is within the robot's workspace and whether there is a feasible solution for inverse kinematics.

[0077] (2) Verify whether there are collisions between the robot bodies, between the robot and the transfer platform, between the robot and the workpiece, and between the robot and the obstacle during the movement process, including robot self-collision.

[0078] (3) Verify whether the travel, speed, limit, load status and the follow-up relationship of the workpiece of the transfer platform meet the constraints.

[0079] (4) Verify whether the relationship state switching of the workpiece before and after the action is consistent, that is, whether the state switching of the workpiece such as attachment, support, release, etc. meets the semantic requirements of the action.

[0080] (5) Verify whether the state of the workpiece released by the previous execution subject and the state of the workpiece grasped by the next execution subject are continuous in space and time, that is, the relay continuity of multiple execution subjects.

[0081] (6) Verify whether the target position is occupied, whether the target posture meets the stacking requirements, and whether there is a spatial conflict between the operation workpieces.

[0082] The above verification can employ one of two structured error output strategies: The first strategy (fail-fast mode): Upon detecting any action step verification failure, the current round of verification is immediately interrupted. The structured error generation module outputs the structured error information of the failed action step, and the feedback and replanning process begins. This strategy is suitable for scenarios where steps are highly dependent and the failure of a previous verification step renders subsequent steps meaningless.

[0083] The second strategy (batch mode): After verifying all action steps, all failed action steps are summarized into a structured error message list and output uniformly. This strategy is suitable for scenarios where it is desirable to fix as many problems as possible in a single iteration. This strategy is switched by configuration parameters.

[0084] During the execution of step A3, the digital twin engine in the digital twin verification environment automatically records the viewpoint parameters, poses of each digital twin verification environment object, collision geometry and the relationship state variables of the manipulated workpiece during each action step verification process.

[0085] If all actions in step A3 pass verification, an executable task plan and verification report will be output. The verification report includes the verification status of each task step, the robot target pose, the platform target position, the workpiece state transition record, collision detection results, reachability results, and the target position occupancy status.

[0086] If the verification of the current action step fails during the verification process of step A3, then steps A4 to A5 will be executed.

[0087] A4. Generate structured error information and virtual environment visual feedback information corresponding to the current action step; the virtual environment visual feedback information includes at least a screenshot of the virtual environment and annotation information on the screenshot of the virtual environment.

[0088] Structured error information is generated by the structured error generation module. This structured error information includes at least: a verified result identifier (verified), a failed action step identifier (failed_step), a failed actor, a failed action (action), an object involved (object), a failure type (failure_type), a failure reason (reason), a suggested modification, and a associated screenshot identifier (snapshot). The failure types include at least: unreachable target pose (unreachable_target_pose), collision on the motion path (collision), inconsistent state of the workpiece relationship (state_mismatch), platform target position out of range (platform_out_of_range), target slot occupied (pallet_slot_occupied), mismatch in handoff order or pose (handoff_mismatch), and mismatch in gripper pose (gripper_misalignment).

[0089] An example of structured error messages is as follows: Figure 6As shown in the diagram, "verified" indicates whether the verification passed; "failed_step" indicates the identifier of the failed action step; "actor" indicates the executing entity of the failed action step; "action" indicates the failed action; "object" indicates the object involved; "failure_type" indicates the failure type, such as "unreachable_target_pose" (target pose unreachable), "collision" (motion path collision), "state_mismatch" (workpiece relationship state inconsistency), "platform_out_of_range" (platform target position out of range), "pallet_slot_occupied" (target slot occupied), "handoff_mismatch" (handoff order or pose mismatch), "gripper_misalignment" (gripper pose mismatch), etc.; "reason" indicates a natural language description of the failure reason; "suggestion" indicates the corresponding modification suggestion; and "snapshot" indicates the associated virtual environment screenshot identifier.

[0090] Simultaneously, the digital twin screenshot and annotation module automatically generates visual feedback information for the virtual environment. Specifically: (1) Based on the pose of the objects involved in the failed step (robot, transfer platform, operation workpiece, target position, etc.) in the digital twin verification environment, an observation viewpoint is automatically selected, and the virtual environment screenshot rendering of the viewpoint is triggered.

[0091] (2) Match the corresponding visualization overlay rules from the preset annotation template library according to the failure type. For example, for the collision type, automatically overlay the red collision bounding box; for the unreachable target pose type, automatically overlay the yellow target pose marker and reachable workspace boundary; for the pallet slot occupied type, automatically overlay the target position and the highlight of the conflicting object.

[0092] (3) Output PNG format image and corresponding annotation metadata.

[0093] The output virtual environment screenshots and annotations include: screenshots of failed steps, annotations of collision areas, annotations of poses of unreachable targets, snapshots of the relationship status between the robot / operating workpiece / transfer platform, annotations of spatial conflicts in the target location, and optional top view, side view, trajectory map, depth map, point cloud map, heat map or multi-view combination map.

[0094] A5. The original task objective, the current structured task plan, the structured error information, and the virtual environment visualization feedback information are fed back to the task plan generation entity as multimodal feedback information to regenerate the structured task plan.

[0095] The iterative replanning module combines the following four types of information into multimodal feedback information: (i) Task objectives (including user-defined palletizing or transport task objectives, such as target location, cycle time requirements, safety constraints, etc.); (ii) Current structured task plan; (iii) The structured error information output in step A4; (iv) Screenshots and annotations of the virtual environment output in step A4.

[0096] The above four types of information are fed back to the task plan generating entity in an input format acceptable to the task plan generating entity (such as an image-text interleaved format or a combination of image and text descriptions), so that the entity can correct the task plan based on structured error information and a visual scenario while preserving the task objective constraints.

[0097] In an exemplary embodiment, prior to step A1, in order to perform preliminary processing on the input image to ensure the consistency of the input image, the method may further include steps B1 to B3.

[0098] B1. Obtain the input image.

[0099] B2. Preprocess the input image.

[0100] B3. Output the preprocessed image.

[0101] like Figure 7 As shown, the multimodal feedback interaction process is as follows: The task plan generation entity (in this embodiment, a Visual Language Model (VLM)) re-plans or partially modifies the task plan based on the task objective, the current structured task plan, structured error information, and screenshots and annotations of the virtual environment (e.g., modifying only the failed action steps and their related steps before and after), generating a new version of the structured task plan. Re-planning methods include: modifying the platform target position, adjusting the grasping or placement posture, changing the robot's execution order, changing the target position, adding intermediate obstacle avoidance steps, introducing manual review nodes, or reducing task constraints. The Visual Language Model (VLM) can be replaced by a Visual Language Action Model, a combination of retrieval-enhanced generation and a large language model, an expert system, a rule system, a process planning system, or a hybrid of the above methods.

[0102] A6. Use the regenerated structured task plan as the current structured task plan, jump to step A2, until the stopping condition is met, output the current structured task plan or output a failure report and trigger manual review.

[0103] The iterative replanning module controls the repeated execution of steps A2 to A6, as follows: Figure 7 As shown, each time partial validation fails, a structured error report and virtual environment visual feedback are generated, providing multimodal feedback, and a structured task plan is regenerated. This continues until any of the following stopping conditions are met, where the validation passes if: (1) All action steps in the digital twin verification environment are verified and an executable task plan is output.

[0104] The conditions under which verification fails can be as follows: (1) The number of replanning iterations reaches the preset maximum number of iterations.

[0105] (2) The current failure type belongs to the preset non-automatically repairable error type.

[0106] (3) The output confidence of the task plan generator is lower than the preset threshold.

[0107] (4) The number of action steps that have been verified in multiple consecutive iterations has not increased.

[0108] (5) Trigger the preset conditions for manual review.

[0109] If any of the above verification failure conditions are triggered, a failure report will be output and forwarded to manual review.

[0110] The iterative replanning module also records each round of task plan, verification results, error type, screenshot identifier, modification content, and reason for stopping, for process traceability.

[0111] The execution entities of each of the above steps are computer systems and their functional modules; the task plan generation entity exists as a model component called by the system; physical actuators such as robots and transfer platforms are modeled as digital twin verification environment objects in this method, participating in motion verification without directly performing actions.

[0112] Furthermore, those skilled in the art will understand that, in addition to palletizing / transfer, it can also be extended to scenarios requiring task-level planning and motion verification, such as loading and unloading, pre-assembly transfer, workpiece sorting, multi-robot handling, inter-equipment power transfer, production line buffer transfer, warehouse handling, welding, and spraying.

[0113] Based on the same inventive concept, this application also provides a system for implementing the above-described method. The solution provided by this system is similar to the solution described in the above-described method. In an exemplary embodiment, such as... Figure 8 As shown, the system includes the following modules: (1) Scene information acquisition module: used to acquire on-site images, task objectives and virtual scene information; output scene input data and task objective data, and connect to the task plan access module and the digital twin object binding module.

[0114] (2) Task plan access module: used to receive the structured task plan generated by the task plan generation subject; during the iteration phase, the corresponding task plan generation subject is called to generate a new version of the structured task plan based on multimodal feedback information; the generation algorithm of the task plan generation subject itself is not a protected object of this system, but is only used as a model component for system calls.

[0115] (3) Structured task plan parsing module: used to parse the step identifier, execution subject, action type, operation object, source location, target location, transfer platform, target position, constraints and step dependencies in the structured task plan; output task semantic data organized by step.

[0116] (4) Motion verification task generation module (interface conversion module): It is used to generate motion verification tasks according to action semantics based on task semantic data, including robot approach, grasp, attach, move, place, release, evacuate, platform carrying movement, target position check and multi-executor handover check sub-tasks.

[0117] (5) Digital Twin Object Binding Module: Used to bind robot objects, transfer platform objects, operation workpiece objects, fixture objects, obstacle objects and target position objects in the virtual verification environment, and bind the model, coordinate system, state and constraint data of each object.

[0118] (6) Workpiece Relationship Status Management Module: Connected to the motion verification task generation module and the digital twin object binding module, it is used to maintain the relationship status of the workpiece, including fixed in the world coordinate system, attached to the end of the robot, supported on the transfer platform, moving with the transfer platform, attached to the end of another robot, and placed in the target position.

[0119] (7) Motion Feasibility and Collision Verification Module: Connected to the Digital Twin Object Binding Module and the Workpiece Relationship Status Management Module, it is used to perform motion verification tasks in the virtual verification environment and output reachability verification results, collision detection results, state switching consistency verification results, platform constraint verification results, target position conflict verification results and multi-executor relay consistency verification results.

[0120] (8) Structured error generation module: connected to the motion feasibility and collision verification module, used to output structured error information including verification result identifier, failed action step identifier, failed execution subject, failed action, involved object, failure type, failure reason, modification suggestion, and associated screenshot identifier when verification fails.

[0121] (9) Digital Twin Screenshot and Annotation Module: Connected to the motion feasibility and collision verification module and the structured error generation module, it is used to generate virtual environment screenshots, collision area annotation maps, unreachable area annotation maps, status snapshots, target position conflict annotation maps, and optional top view, trajectory map, depth map, and point cloud map based on the failure steps and failure types.

[0122] (10) Iterative replanning module: It is connected to the structured error generation module, digital twin screenshot and annotation module, task plan access module and scene information acquisition module respectively; it is used to encapsulate the task target, the previous version of the structured task plan, the structured error information and the virtual environment screenshot and annotation map into multimodal feedback information input to the task plan generation subject, trigger the task plan replanning, control the number of iterations, stopping conditions and manual review conditions, and record the process of each iteration.

[0123] The data flow relationships between the modules in the system are as follows: The scene information acquisition module transmits on-site images, task objectives, and virtual scene information to the task planning access module; the task planning access module transmits the structured task plan to the structured task plan parsing module; the structured task plan parsing module transmits the parsed task semantic data to the motion verification task generation module and the workpiece relationship status management module; the motion verification task generation module transmits the motion verification task sequence to the motion feasibility and collision verification module; the digital twin object binding module and the workpiece relationship status management module transmit the bound virtual verification environment object and the relationship status of the operating workpiece to the motion feasibility and collision verification module; the motion feasibility and collision verification module transmits the verification results, including whether the verification passed and details of the failure steps, to the structured error generation module and the digital twin screenshot and annotation module; the structured error generation module and the digital twin screenshot and annotation module transmit structured error information and virtual environment screenshots and annotation diagrams to the iterative replanning module; the iterative replanning module transmits multimodal feedback information to the task planning access module, triggering the task plan generation entity to regenerate the task plan; the motion feasibility and collision verification module outputs an executable task plan and verification report to the system on the verification-passed branch.

[0124] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0126] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0127] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0128] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback, characterized in that, include: Obtain a structured task plan generated based on the task objectives, and use the structured task plan as the current structured task plan; The structured task plan contains multiple action steps arranged in execution order, and each action step includes at least the execution subject, action semantics, and operation object; The current structured task plan is parsed, the action semantics of each action step are identified, and each action step is converted into a corresponding motion verification task based on the action semantics to obtain a motion verification task sequence. Motion verification is performed step by step in the motion verification task sequence within the digital twin verification environment; If the current action step fails during the verification process, a structured error message and a virtual environment visualization feedback message corresponding to the current action step are generated; the virtual environment visualization feedback message includes at least a screenshot of the virtual environment and the annotation information on the screenshot of the virtual environment. The original task objective, the current structured task plan, the structured error information, and the virtual environment visualization feedback information are fed back to the task plan generation entity as multimodal feedback information to regenerate the structured task plan. The regenerated structured task plan is used as the current structured task plan. The process jumps to the step "parse the current structured task plan, identify the action semantics of each action step, and convert each action step into a corresponding motion verification task according to the action semantics to obtain a motion verification task sequence" until the stopping condition is met. Then, the current structured task plan is output or a failure report is output and manual review is triggered.

2. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback as described in claim 1, characterized in that, Each action step in the structured task plan further includes at least one of source pose, target pose, constraints, and step dependencies; the action semantics include at least one of grasping, placing on the transfer platform, platform movement, grasping from the transfer platform, and placing on the target position; each action step is converted into a corresponding motion verification task based on the action semantics to obtain a motion verification task sequence, specifically including: For action steps with the action semantics of "grab", generate motion verification tasks that include approaching, clamp alignment, grabbing, attaching to the operation object, and withdrawing. For action steps with the action semantics of "place on the transfer platform", generate motion verification tasks that include moving to the platform placement position, releasing the operation object, and withdrawing. For action steps with the action semantics of "platform movement", generate a motion verification task that includes the transfer platform moving along the constraint axis and carrying the operation object following the movement. For action steps with the action semantics of "grab from the transfer platform", generate a motion verification task that includes the robot grabbing the operation object from the platform handover position; For action steps with the action semantics of "place to target location", generate a motion verification task that includes moving to target location, placing, releasing and withdrawing.

3. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, Before performing motion verification step-by-step according to the motion verification task sequence in the digital twin verification environment, the following steps are also included: In the digital twin verification environment, the robot object, transfer platform object, operation workpiece object, fixture object, obstacle object and target position object participating in the task execution are bound together. The mapping relationship between each field in the current structured task plan and each digital twin verification environment object is established, and the geometric model, kinematic model, coordinate system, state variables, collision attributes and constraint parameters of each digital twin verification environment object are bound together.

4. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, In the digital twin verification environment, motion verification is performed step by step according to the motion verification task sequence, specifically including: Verify whether the robot's end effector can reach the target pose or intermediate pose; Verify whether there are collisions between robot bodies, between the robot and the transfer platform, between the robot and the workpiece being manipulated, and between the robot and obstacles during the movement process; Verify whether the travel, speed, limit, and load status of the transfer platform, as well as the follow-up relationship of the operated workpiece, meet the constraints. Verify whether the relationship state switching of the workpiece before and after the action is consistent; Verify whether the state of the workpiece released by the previous executing entity and the state of the workpiece grasped by the next executing entity are continuous in space and time; Verify whether the target position is occupied, whether the target orientation meets the stacking requirements, and whether there are spatial conflicts between the manipulated workpieces.

5. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, The structured error information includes at least one of the following: verification result identifier, failed action step identifier, failed execution subject, failed action, involved object, failure type, failure reason, modification suggestion, and associated screenshot identifier; the failure type includes at least one of the following: target pose unreachable, motion path collision, inconsistent operation workpiece relationship state, platform target position exceeding limit, target position occupied, handover sequence or pose mismatch, and fixture pose mismatch.

6. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, Generate visual feedback information of the virtual environment corresponding to the current action step, specifically including: During the current action step verification process, record the viewpoint parameters, poses of each digital twin verification environment object, collision geometry, and relational state variables of the manipulated workpiece; Select an observation viewpoint based on the pose of the object involved in the current action step in the digital twin verification environment, trigger the screenshot rendering of that viewpoint, and obtain a screenshot of the virtual environment; Based on the failure type, the corresponding visualization overlay rule is matched from the preset annotation template library, and annotation information is overlaid on the virtual environment screenshot, outputting the virtual environment screenshot and the annotation information on the virtual environment screenshot.

7. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, The stopping condition includes at least one of the following: All actions and steps in the digital twin verification environment have passed verification; The number of replanning iterations has reached the preset maximum number of iterations. The current failure type belongs to the preset non-automatically recoverable error type; The confidence level of the task plan generator's output is lower than a preset threshold; The number of verified action steps did not increase in multiple consecutive iterations; Triggering preset conditions for manual review.

8. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, The task plan generation entity can be any one of the following: a large visual language model, a visual language action model, a combination of retrieval enhancement generation and a large language model, an expert system, a rule system, or a process planning system.

9. The iterative planning method for palletizing and transporting tasks based on digital twin multimodal feedback according to claim 1, characterized in that, When performing motion verification step by step according to the motion verification task sequence in the digital twin verification environment, the first strategy is adopted to perform motion verification: when any motion step verification failure is detected, the current round of verification is immediately interrupted and the structured error information of the failed motion step is output, and the feedback and replanning process is entered. Alternatively, a second strategy can be used to perform motion verification: after verifying all motion steps, all failed motion steps are summarized into a structured error message list and output uniformly.

10. An iterative planning system for palletizing and transporting tasks based on digital twin multimodal feedback, characterized in that, include: The task plan access module is used to obtain a structured task plan generated based on the task objective and use the structured task plan as the current structured task plan; The structured task plan contains multiple action steps arranged in execution order, and each action step includes at least the execution subject, action semantics, and operation object; The structured task plan parsing module is used to parse the current structured task plan and identify the action semantics of each action step; The motion verification task generation module is used to convert each action step into a corresponding motion verification task according to the action semantics, so as to obtain a motion verification task sequence. The motion feasibility and collision verification module is used to perform motion verification step by step in the motion verification task sequence in the digital twin verification environment; if the current action step fails to verify during the verification process, the structured error generation module and the digital twin screenshot and annotation module are called. The structured error generation module is used to generate structured error information corresponding to the current action step; A digital twin screenshot and annotation module is used to generate virtual environment visual feedback information corresponding to the current action step; the virtual environment visual feedback information includes at least a virtual environment screenshot and annotation information on the virtual environment screenshot; The iterative replanning module is used to feed back the original task objective, the current structured task plan, the structured error information, and the virtual environment visualization feedback information as multimodal feedback information to the task plan generation subject, so as to regenerate the structured task plan; The regenerated structured task plan is used as the current structured task plan. The structured task plan parsing module, motion verification task generation module, and motion feasibility and collision verification module are called in sequence until the stopping condition is met. The current structured task plan is then output or a failure report is output and manual review is triggered.

Images