Human-robot interaction system with body intelligence

Abstract

The present application relates to the technical field of mechanical arm, and in particular to a body intelligent mechanical arm human-computer interaction system. The system decomposes long time sequence tasks into subtasks through a large language model, and introduces task acquisition, task queue management, task re-planning and control execution modules: temporary tasks are triggered by visual detection and recognition of gestures and environmental events, interruptable markers and safety anchors are maintained based on priority and interruption risk, the state of the mechanical arm and the environment with timestamps is collected when interrupted, the state is compared and local re-planning is performed when recovered, transition subtasks are automatically generated for failed subtasks, and the mechanical arm is controlled to execute after constraint verification. The system can realize safe interruption and efficient recovery in long time sequence tasks while ensuring structural consistency, spatial consistency and temporal consistency, significantly improving the autonomy, real-time performance and operational safety of the mechanical arm in human-machine collaborative scenarios.

Description

Technical Field

[0001] This invention relates to the field of robotic arm technology, and more specifically to a human-machine interaction system for an embodied intelligent robotic arm. Background Technology

[0002] With the development of intelligent manufacturing and human-machine collaboration technologies, the autonomous execution capability of robotic arms in complex and dynamic environments has attracted much attention. In multi-task human-machine interaction scenarios, robotic arms often need to handle long-term planned tasks and uncertain temporary tasks simultaneously. Traditional robotic arm systems mainly rely on preset scripts and fixed control processes, and their task execution modes lack flexibility, making it difficult to dynamically adjust according to environmental changes or operator intentions.

[0003] Existing task switching mechanisms typically lack descriptions of task interruptibility and fail to systematically assess the priority, urgency, and safety of different task phases. This can lead to risks such as end-effector instability, object detachment, or trajectory conflicts when a task is interrupted. Furthermore, when execution resumes after a task interruption, the robot's posture, target object position, and environmental layout may differ from those at the time of interruption. However, current technologies often directly reuse the task parameters and path planning from before the interruption, making it difficult to guarantee geometric feasibility and operational safety.

[0004] On the other hand, existing task replanning technologies mostly adopt a global replanning approach, which is computationally complex and difficult to meet real-time requirements. Furthermore, they do not systematically consider multi-dimensional constraints such as task structure consistency, spatial consistency, and temporal consistency, resulting in deficiencies in the semantic logic, execution continuity, and security feasibility of the generated recovery task sequences.

[0005] Therefore, how to achieve intelligent trigger recognition of temporary tasks, task interruption decision based on interruptibility and safety policies, task executability analysis for interruption recovery scenarios, and local task replanning that meets multidimensional consistency constraints has become a key technical issue in improving the autonomy and safety of robotic arms in human-computer interaction scenarios. Summary of the Invention

[0006] The technical problem this invention aims to solve is to address the shortcomings of existing technologies by providing an embodied intelligent robotic arm human-machine interaction system. This system decomposes long-sequential tasks into sub-tasks using a large language model and introduces modules for task acquisition, task queue management, task replanning, and control execution. It utilizes visual detection to recognize gestures and environmental events to trigger temporary tasks, maintains interruptible markers and safety anchors based on priority and interruption risk, collects timestamped robotic arm and environmental states during interruptions, compares the states upon recovery, performs local replanning, automatically generates transitional sub-tasks for failed sub-tasks, and controls the robotic arm execution after constraint verification. This system can achieve safe interruption and efficient recovery in long-sequential tasks while ensuring structural, spatial, and temporal consistency, significantly improving the autonomy, real-time performance, and operational safety of the robotic arm in human-machine collaborative scenarios.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An embodied intelligent robotic arm human-computer interaction system is applied to human-computer interaction scenarios that combine long-term and temporary tasks. The long-term tasks are decomposed into multiple sub-tasks using a preset large language model to generate an initial task sequence. The system includes:

[0009] The task acquisition module is used to identify the triggering conditions of temporary tasks through a visual detection algorithm, determine the priority of the temporary tasks, and generate the priority of the temporary tasks.

[0010] The task queue management module is used to input the initial task sequence into the dynamic task queue. The dynamic task queue generates a task interruption instruction based on the priority of the temporary task and the safety interruption policy, and records the robotic arm status data and the sequence of tasks to be executed at the time of interruption.

[0011] The task replanning module is used to perform local replanning on the sequence of tasks to be executed based on the robotic arm state data at the time of the interruption and the current robotic arm state data after the interruption is resumed, and output the replanned task sequence.

[0012] The control execution module is used to perform pre-execution verification of the replanning task sequence through a visual detection algorithm, and generate motion control commands for the robotic arm based on the verification results, thereby controlling the robotic arm to execute interactive tasks.

[0013] The task queue management module includes a queue maintenance unit, an interruption decision unit, and a status recording unit, including:

[0014] The queue maintenance unit is used to maintain the interruptible flag and safe interruption anchor point for each subtask in the dynamic task queue. The safe interruption anchor point represents the time node during the execution of the subtask that is allowed to be interrupted.

[0015] The interruption decision unit is used to generate a task interruption instruction based on the priority of the temporary task and the interruptibility flag of the currently executing subtask when a temporary task is received.

[0016] The state recording unit is used to record the robotic arm state data, environmental state data, and the sequence of tasks to be executed at the time of interruption, and to generate a snapshot of the robotic arm state.

[0017] The interrupt decision unit is configured with interrupt point selection logic, which includes:

[0018] Get the priority and urgency coefficient of temporary tasks;

[0019] Identify the interruptible flag of the currently executing subtask. If the currently executing subtask is marked as uninterruptible, search the dynamic task queue for the safe interrupt anchor point closest to the current executing subtask.

[0020] Calculate the interruption risk score based on the priority, urgency coefficient, and most recent security interruption anchor point of the temporary task;

[0021] When the interruption risk score is lower than the preset safety interruption threshold, an immediate interruption command is generated to control the robotic arm to interrupt the current sub-task after it is completed.

[0022] When the interruption risk score is higher than or equal to the preset safety interruption threshold, a delayed interruption command is generated to control the robotic arm to interrupt after reaching the next safety interruption anchor point.

[0023] The task replanning module includes a state comparison unit, a task evaluation unit, and a local replanning unit, comprising:

[0024] The state comparison unit is used to obtain the joint angle, end pose and position of the current robotic arm and compare them with the robotic arm state data at the time of interruption to generate state difference data.

[0025] The task evaluation unit is used to determine the executability of each subtask in the sequence of tasks to be executed based on the state difference data, and to identify executable subtasks, subtasks that need to modify parameters, and failed subtasks.

[0026] The local replanning unit performs a local replanning operation on the sequence of tasks to be executed based on the task consistency index, thereby generating a replanned task sequence.

[0027] The task evaluation unit is configured with sub-task judgment logic, which includes:

[0028] Obtain the target location and execution prerequisites for each subtask in the sequence of tasks to be executed;

[0029] The positions of environmental objects in the state difference data are matched and analyzed with the target positions of each sub-task to calculate the change in the target position of the sub-task.

[0030] When the change in the target position of a subtask is less than the preset position tolerance threshold and the execution prerequisites are met, the subtask is marked as an executable subtask.

[0031] When the change in the target position of a subtask is greater than or equal to the preset position tolerance threshold, but the execution prerequisites meet the execution requirements, the subtask is marked as a subtask that needs parameter correction, and the change in the target position is used as the correction parameter.

[0032] When the preconditions for the execution of a subtask are no longer met, the subtask is marked as an invalid subtask.

[0033] The local replanning unit is configured with replanning logic, which includes:

[0034] The execution order of all executable subtasks remains unchanged, maintaining structural consistency;

[0035] For subtasks that require parameter modification, update the target position and target attitude based on the modified parameters to maintain spatial consistency;

[0036] Identify all failed subtasks and generate transitional subtasks through a large language model. These transitional subtasks are used to connect executable subtasks before and after the failed subtasks to maintain time consistency.

[0037] An optimization objective function is constructed based on the task consistency index. The optimization objective is to maximize the weighted sum of structural consistency, spatial consistency and temporal consistency. The task sequence optimization algorithm is used to locally adjust the sequence of tasks to be executed and generate a replanning task sequence.

[0038] The step of locally adjusting the sequence of tasks to be executed using a task sequence optimization algorithm to generate a replanning task sequence includes:

[0039] Initialize the replanning sequence by adding all executable subtasks to the replanning sequence in their original order, and generate candidate replanning sequences.

[0040] Traverse all failed subtasks, analyze the robot arm pose and environmental state before and after the failed subtasks through visual detection, obtain the corresponding transition subtasks, and generate a set of transition subtasks.

[0041] For each failed subtask location, select the transition subtask from the transition subtask set that maximizes the increment of the optimization objective function and insert it;

[0042] For subtasks that require parameter modification, update the subtask parameters by minimizing the end-effector pose error;

[0043] By adjusting the execution order and parameter configuration of subtasks in the replanning sequence through an iterative optimization algorithm, the optimization objective function converges to a local optimum, and the final replanning task sequence is output.

[0044] The control execution module includes a verification unit and an instruction generation unit, including:

[0045] The verification unit is used to perform pre-execution verification on the replanning task sequence using a visual detection algorithm to determine whether the constraints are met. The constraints include resource conflict detection, robotic arm posture safety detection, and environmental collision detection.

[0046] The instruction generation unit is used to generate motion control instructions based on the replanning task sequence when the replanning task sequence meets the constraints, control the robotic arm to return to the execution state before the interruption and continue execution, and generate a replanning trigger signal when the replanning task sequence does not meet the constraints, and replanning is performed through the task replanning module.

[0047] The verification unit is configured with constraint condition judgment logic, which includes:

[0048] Kinematic simulations are performed on each subtask in the replanning task sequence to detect whether the robotic arm's execution trajectory collides with environmental obstacles, whether there are conflicts or insufficient resources in the end effector, sensors, and computing resources required for each subtask, and whether the joint angles and joint velocities of each subtask are within the safe operating range of the robotic arm. Constraints are then generated for the replanning task sequence.

[0049] When all the constraints are met, a verification pass flag is generated, and action control instructions are generated based on the replanning task sequence.

[0050] When any constraint is not met, a constraint violation report and a replanning trigger signal are generated. The constraint violation report is then fed back to the task replanning module for iterative replanning until all constraints are met.

[0051] The state recording unit is configured with state snapshot generation logic, which includes:

[0052] Upon receiving a task interruption command, multi-source state data at the moment of interruption is collected using a vision sensor and a robotic arm torque sensor. This multi-source state data includes:

[0053] Record the joint angles, joint velocities, joint torques, and end-effector pose of the robotic arm to generate motion state data of the robotic arm;

[0054] Record the position, orientation, and category of objects within the workspace to generate environmental status data;

[0055] Record the identifier, execution progress, and sequence of tasks to be executed for the currently executing subtask, and generate task execution status data;

[0056] The robotic arm motion state data, environmental state data, and task execution state data are encapsulated in a preset JSON format to generate a robotic arm state snapshot.

[0057] Add a timestamp and a unique identifier to the snapshot of the robotic arm's state, store it in a local database, and establish a mapping index with the sequence of tasks to be executed.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] 1. This invention sets interruptible flags and safe interruption anchors for each subtask in a dynamic task queue, and generates immediate or delayed interruption instructions by combining the priority, urgency and interruption risk score of temporary tasks. This enables quantitative assessment and hierarchical control of task interruption behavior, effectively avoiding risks such as end-effector attitude instability, object detachment and trajectory conflict, and ensuring the execution continuity and safety of long-sequence tasks when temporary tasks are frequently inserted.

[0060] 2. This invention transforms the replanning problem into a constrained sequence optimization problem by comparing states, evaluating task executability, and using a local replanning mechanism based on task structure consistency, spatial consistency, and temporal consistency. It only makes local adjustments to the sequence of tasks to be executed, which significantly reduces computational complexity and improves real-time performance compared to global replanning, while maintaining the consistency of task semantic logic, motion path, and execution timing. Attached Figure Description

[0061] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0062] Figure 1 This is a schematic diagram of the structure of the human-computer interaction system of the embodied intelligent robotic arm according to an embodiment of this application;

[0063] Figure 2 This is a flowchart illustrating the configuration strategy of the human-computer interaction system for the intelligent robotic arm, as described in this application embodiment.

[0064] Figure 3 This is a schematic diagram of the process for generating interrupt instructions in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram illustrating the partial replanning process in an embodiment of this application. Detailed Implementation

[0066] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0067] Please see Figure 1 This invention provides an embodiment of an embodied intelligent robotic arm human-computer interaction system, applied to human-computer interaction scenarios that mix long-term tasks and temporary tasks. The long-term tasks are decomposed into multiple sub-tasks through a preset large language model to generate an initial task sequence. The system includes:

[0068] The task acquisition module is used to identify the triggering conditions of temporary tasks through a visual detection algorithm, determine the priority of the temporary tasks, and generate the priority of the temporary tasks.

[0069] The task queue management module is used to input the initial task sequence into the dynamic task queue. The dynamic task queue generates a task interruption instruction based on the priority of the temporary task and the safety interruption policy, and records the robotic arm status data and the sequence of tasks to be executed at the time of interruption.

[0070] The task replanning module is used to perform local replanning on the sequence of tasks to be executed based on the robotic arm state data at the time of the interruption and the current robotic arm state data after the interruption is resumed, and output the replanned task sequence.

[0071] The control execution module is used to perform pre-execution verification of the replanning task sequence through a visual detection algorithm, and generate motion control commands for the robotic arm based on the verification results, thereby controlling the robotic arm to execute interactive tasks.

[0072] Furthermore, the task queue management module includes a queue maintenance unit, an interrupt decision unit, and a status recording unit, comprising:

[0073] The queue maintenance unit is used to maintain the interruptible flag and safe interruption anchor point for each subtask in the dynamic task queue. The safe interruption anchor point represents the time node during the execution of the subtask that is allowed to be interrupted.

[0074] The interruption decision unit is used to generate a task interruption instruction based on the priority of the temporary task and the interruptibility flag of the currently executing subtask when a temporary task is received.

[0075] The state recording unit is used to record the robotic arm state data, environmental state data, and the sequence of tasks to be executed at the time of interruption, and to generate a snapshot of the robotic arm state.

[0076] Furthermore, the task replanning module includes a state comparison unit, a task evaluation unit, and a local replanning unit, comprising:

[0077] The state comparison unit is used to obtain the joint angle, end pose and position of the current robotic arm and compare them with the robotic arm state data at the time of interruption to generate state difference data.

[0078] The task evaluation unit is used to determine the executability of each subtask in the sequence of tasks to be executed based on the state difference data, and to identify executable subtasks, subtasks that need to modify parameters, and failed subtasks.

[0079] The local replanning unit performs a local replanning operation on the sequence of tasks to be executed based on the task consistency index, thereby generating a replanned task sequence.

[0080] Furthermore, the control execution module includes a verification unit and an instruction generation unit, comprising:

[0081] The verification unit is used to perform pre-execution verification on the replanning task sequence using a visual detection algorithm to determine whether the constraints are met. The constraints include resource conflict detection, robotic arm posture safety detection, and environmental collision detection.

[0082] The instruction generation unit is used to generate motion control instructions based on the replanning task sequence when the replanning task sequence meets the constraints, control the robotic arm to return to the execution state before the interruption and continue execution, and generate a replanning trigger signal when the replanning task sequence does not meet the constraints, and replanning is performed through the task replanning module.

[0083] Please see Figure 2 The present invention provides an embodiment of a configuration strategy for a human-machine interaction system of an embodied intelligent robotic arm, which is applied to the human-machine interaction system of the embodied intelligent robotic arm. The specific steps of the configuration strategy are as follows:

[0084] S1: Identify the triggering conditions of temporary tasks through a visual detection algorithm, determine the priority of the temporary tasks, and generate the priority of the temporary tasks.

[0085] Specifically, in human-computer interaction scenarios that mix long-term and ad-hoc tasks, ad-hoc tasks often arise from real-time changes in the environment and operator behavior, such as operator gestures, newly appearing workpieces on the workbench, the illumination of abnormal status indicator lights, objects falling, or blocked passageways. Relying solely on manual buttons, preset scripts, or single sensor signals to trigger ad-hoc tasks fails to adequately reflect operator intent and makes it difficult to promptly detect subtle changes in the work environment, easily leading to delayed ad-hoc task responses and even missing opportunities for safety intervention. Therefore, in this step, the task acquisition module introduces a visual detection algorithm to continuously acquire and analyze scene images within the robotic arm's workspace, automatically identifying ad-hoc task triggering conditions that meet predefined rules. Based on this, the triggered ad-hoc tasks are prioritized, generating ad-hoc task priorities that can be directly invoked by the dynamic task queue and interruption decisions.

[0086] For example, at least one set of visual acquisition devices, such as industrial cameras or depth cameras, are arranged above the working area of ​​the robotic arm. These devices are either fixedly installed or linked with the end effector of the robotic arm to acquire real-time image sequences of the scene, including the worktable, tooling fixtures, workpieces to be processed, material pallets, and the operator's upper limb area. The visual detection algorithm first preprocesses the acquired raw images, including distortion correction, adaptive adjustment of brightness and contrast, and cropping of regions of interest, to improve the robustness of subsequent feature extraction and target recognition. Subsequently, based on a pre-trained target detection network and gesture recognition network, key objects and human-machine interaction actions in the images are identified and located, resulting in multi-category detection results, including workpiece category and location, operator hand position and posture, indicator light status, and alarm symbols, which are then uniformly mapped to a spatial representation in the robotic arm's base coordinate system.

[0087] Furthermore, to transform the visual detection results into semantic events that can be used to drive temporary tasks, this embodiment predefines a set of temporary task triggering conditions, each triggering condition corresponding to one or more combinations of visual patterns. For example, if an operator is detected making a predefined gesture such as "waving," "pointing to a workstation," or "raising a thumb or clenching a fist" in a specific area, it can be parsed into different categories of temporary tasks such as "requesting to insert an auxiliary handling task," "requesting to immediately check a designated workstation," or "confirming the current process has passed." If a new workpiece is detected in an area and the workpiece is in an unprocessed state, it can trigger the insertion of an additional loading or sorting temporary task. If a workpiece is detected falling to the ground or deviating from the fixture's working area, it can trigger an emergency sorting or abnormal handling temporary task. The task acquisition module matches the above visual detection results with the predefined triggering conditions. When a triggering condition is met, a corresponding temporary task description is generated, including the task type, target location, target object, and associated safety attributes.

[0088] After triggering a temporary task, this embodiment further differentiates the importance and urgency of different temporary tasks to enable the dynamic task queue to make reasonable interruption and scheduling decisions when competing for resources with long-sequence tasks. To this end, the task acquisition module prioritizes each triggered temporary task based on visually detected event characteristics, environmental context, and the current task status. Priority determination considers at least the following factors: First, the safety level of the task type; for example, temporary tasks involving personnel safety protection, equipment failure warnings, or dangerous item drops have a higher safety level than general handling or sorting temporary tasks. Second, the spatial location and relationship to risk areas; for example, temporary tasks triggered at the edge of the robotic arm's movement range, near personnel activity channels, or around high-value equipment have a correspondingly higher priority. Third, time sensitivity; for example, when visual detection determines that a workpiece is about to exceed its allowable dwell time or a workstation is about to block the production line's cycle time, the corresponding temporary task has a higher timeliness requirement.

[0089] S2: Input the initial task sequence into the dynamic task queue. The dynamic task queue generates a task interruption instruction based on the priority of the temporary task and the safety interruption policy, and records the robotic arm status data and the task sequence to be executed at the time of interruption.

[0090] Specifically, since long-sequence tasks are typically decomposed into multiple subtasks with dependencies based on a pre-defined large language model according to user intent, executing them sequentially in a static list format makes it impossible to flexibly insert temporary tasks during execution and to respond promptly to high-priority operation requests. Therefore, this embodiment introduces a dynamic task queue mechanism in the task queue management module. Using the initial task sequence as the basic input, and combining it with identified temporary tasks and their priority information, the execution order is dynamically adjusted, and a task interruption command is generated while meeting the constraints of the safety interruption policy. Simultaneously, at the moment of interruption triggering, the robotic arm's state data and the sequence of tasks to be executed are structured and recorded, providing a reliable data foundation for task recovery and local replanning.

[0091] S3: After the interruption is recovered, based on the robotic arm state data at the time of the interruption and the current robotic arm state data, perform local replanning on the sequence of tasks to be executed, and output the replanned task sequence.

[0092] Specifically, long-running tasks undergo multiple interruptions and recoveries during execution. During interruptions, the environmental state, target object position, and the robot arm's own posture may all change. Simply using the task plan and motion path from before the interruption after recovery could not only lead to misalignment between the end effector and the target but also introduce new collision risks. Therefore, in this step, the local replanning module, upon receiving the interruption recovery command, does not directly continue execution from the interruption point. Instead, it jointly analyzes the robot arm state data recorded at the time of the interruption with the currently detected robot arm state data. While ensuring task consistency and safety constraints, it performs local replanning of the task sequence to be executed, generating a replanned task sequence adapted to the current situation, enabling the long-running task to resume execution.

[0093] S4: The replanning task sequence is pre-execution verified using a visual detection algorithm, and motion control commands for the robotic arm are generated based on the verification results to control the robotic arm to execute interactive tasks.

[0094] Specifically, the replanning task sequence is regenerated after the interruption is recovered, combining the current robotic arm state data and environmental state data. If it is executed directly without unified consistency and safety verification, on the one hand, resource conflicts, posture out-of-bounds, or potential collision risks that were missed during the replanning process may occur; on the other hand, slight changes in the environment between replanning and actual execution may lead to alignment failures or unstable grasping and placement actions. To address this, this embodiment introduces a pre-execution verification mechanism based on a visual detection algorithm to centrally verify the executability and safety of the replanning task sequence in the current real-time environment. After successful verification, motion control commands corresponding one-to-one with the replanning task sequence are generated to drive the robotic arm to complete the interactive task in a safe and coherent manner. If the verification fails, the reason for the failure is fed back to the task replanning module, triggering a new round of task replanning.

[0095] Please see Figure 3 The flowchart for generating interrupt instructions provided in this embodiment of the invention shows the specific steps of S2 as follows:

[0096] S2.1: Maintain the interruptibility flag and safe interruption anchor point for each subtask in the dynamic task queue, wherein the safe interruption anchor point represents the time node during the execution of the subtask that is allowed to be interrupted.

[0097] Specifically, since long-sequence tasks typically consist of multiple subtasks with sequential dependencies, and the robotic arm involves various motion patterns such as grasping, handling, obstacle avoidance, and human-machine collaboration during execution, interrupting the current subtask at any point in time can easily lead to safety risks such as end effector instability, object detachment, or collision with the environment. Therefore, in this step, all subtasks in the initial task sequence are uniformly connected to a dynamic task queue, and an interruptibility flag and safe interruption anchor point description associated with the execution progress are established for each subtask. This clarifies the "permitted interruption time" and "prohibited interruption period" in both the time and task structure dimensions, providing basic data support for the subsequent interruption decision unit to generate task interruption instructions.

[0098] For example, the dynamic task queue is maintained using a sequential linked list or circular queue data structure. Each subtask in the queue corresponds to a task record, which includes at least the following fields: subtask ID, subtask type, target location and target posture, execution prerequisites, estimated execution duration, current execution progress, interruptibility flag, and a set of safe interruption anchor points. The interruptibility flag characterizes the interruptibility status of the current subtask within the overall execution cycle, and can be implemented using a multi-level flagging method, such as dividing the interruptibility status into three levels: completely uninterruptible, conditionally interruptible, and completely interruptible. The set of safe interruption anchor points stores the critical time nodes during the execution of the subtask, with each safe interruption anchor point corresponding one-to-one with the specific execution progress, robotic arm posture, and environmental constraints. By continuously maintaining the above structured information for subtasks, the dynamic task queue can reflect the current system's interruptibility safety boundaries in real time during task execution.

[0099] Furthermore, in this embodiment, the generation of safety interruption anchor points is segmented based on the motion characteristics and safety attributes of the sub-task. Specifically, for trajectory-type sub-tasks (e.g., moving from the initial pose to the grasping pose, or from the grasping pose to the placement pose), the complete trajectory is divided into several physically meaningful stages based on the offline planned trajectory or online generated trajectory of the robotic arm, such as acceleration, constant speed, deceleration, and end-effector stabilization. The system evaluates safety indicators such as joint velocity, acceleration, and distance between the end effector and obstacles at each stage, and sets safety interruption anchor points only at stages that meet the conditions of "controllable speed, sufficient space redundancy, and no easily detachable target carried by the end effector." For state-sensitive sub-tasks (e.g., grasping, insertion, docking, etc.), safety interruption anchor points are preferentially set after the action is completed and the end effector is in stable contact or completely separated from the target object, avoiding interruptions during the contact process or high-collision-risk stages, thereby reducing the impact risk to the target object and the robotic arm itself.

[0100] S2.2: Upon receiving a temporary task, generate a task interruption instruction based on the priority of the temporary task and the interruptibility flag of the currently executing subtask.

[0101] Specifically, in this embodiment, an interruption command generation mechanism based on a joint decision-making process using priority, urgency coefficient, and interruptibility flag is introduced. The fundamental purpose is to ensure that while long-sequence tasks are executed continuously, the response of temporary tasks reflects the importance of the task at hand without compromising the safety and stability of the robotic arm's current actions. By explicitly introducing a safety interruption strategy during the interruption decision-making process, the interruption command is no longer a simple immediate interruption. Instead, it dynamically generates various strategic control commands such as immediate interruption, delayed interruption, or prohibition of interruption based on the interruptibility attribute of the currently executing subtask, the distribution of safety interruption anchor points, and the urgency of the temporary task. This achieves a balance between task response efficiency and operational safety.

[0102] The specific steps of S2.2 are as follows:

[0103] S2.2.1: Obtain the priority and urgency coefficient of temporary tasks;

[0104] S2.2.2: Identify the interruptible flag of the currently executing subtask. If the currently executing subtask is marked as uninterruptible, search the dynamic task queue for the safe interrupt anchor point closest to the current executing subtask.

[0105] S2.2.3: Calculate the interruption risk score based on the priority, urgency coefficient, and nearest security interruption anchor point of the temporary task;

[0106] S2.2.4: When the interruption risk score is lower than the preset safety interruption threshold, an immediate interruption command is generated to control the robotic arm to interrupt after the current sub-task is completed;

[0107] S2.2.5: When the interruption risk score is higher than or equal to the preset safety interruption threshold, a delayed interruption command is generated to control the robotic arm to interrupt after reaching the next safety interruption anchor point.

[0108] Specifically, in human-computer interaction scenarios that mix long-term and ad-hoc tasks, embodied intelligent robotic arms need to maintain continuous and stable execution of the initial task sequence planned in the long term, while also responding promptly to user-issued instructions or unexpected ad-hoc tasks in the environment. Simply interrupting the current subtask immediately upon receiving an ad-hoc task can easily lead to the robotic arm's end effector being in an unsafe position, the gripped object falling off, or interference with the surrounding environment. However, delaying ad-hoc tasks indiscriminately can cause lag in the response of high-priority ad-hoc tasks, reducing the human-computer interaction experience. Therefore, in this step, the interruption decision unit comprehensively considers the priority and urgency coefficient of ad-hoc tasks, as well as the interruptibility flags and safe interruption anchor information of the currently executing subtasks in the dynamic task queue. Based on a preset safety strategy, it quantifies the interruption risk and generates either an immediate interruption command or a delayed interruption command accordingly, thus achieving a dynamic trade-off between task responsiveness and execution safety.

[0109] For example, the interruption decision unit first receives the priority and urgency coefficient of the temporary task from the task acquisition module. The priority characterizes the importance of the temporary task within the overall task sequence, while the urgency coefficient characterizes the temporary task's sensitivity to response latency. Both the priority and urgency coefficients can be automatically generated using a rule base or a large language model combined with user intent parsing results. Subsequently, the interruption decision unit reads the interruptibility flag and corresponding safe interruption anchor point information of the currently executing subtask from the queue maintenance unit: if the current subtask is marked as interruptible, a safe interruption anchor point near the current execution progress is selected as a candidate interruption point; if the current subtask is marked as uninterruptible, the unit searches the dynamic task queue for the safe interruption anchor point closest to the current execution position and calculates the time distance or trajectory distance from the current moment to that anchor point on the time axis, serving as a quantitative indicator of the impact of the interruption behavior on the continuity of the current subtask.

[0110] Furthermore, in this embodiment, the interruption decision unit constructs an interruption risk scoring function R based on the temporary task priority P, the urgency coefficient E, the time distance T from the nearest safe interruption anchor point, and the interruption cost coefficient C corresponding to the interruptibility marker of the currently executing subtask. For example, through a weighted combination or normalized linear model, the above factors are mapped to risk score values ​​within the interval [0,1]. The larger the time distance T and the interruption cost coefficient C, the further away from the safe interruption anchor point or the less suitable the current subtask is for interruption, and R increases accordingly. Conversely, the higher the priority P and the urgency coefficient E, the more the system allows interruption closer to the current moment, and R can decrease accordingly. By introducing the priority and urgency coefficients, the interruption decision unit can distinguish between "high urgency, high priority" temporary tasks (e.g., safety avoidance instructions, human-machine direct interaction instructions) and "low urgency, normal priority" temporary tasks (e.g., non-critical handling requests), adaptively adjusting interruption behavior in different scenarios.

[0111] Furthermore, in the safety strategy of this embodiment, an interruption risk threshold R_th is preset, and the interruption risk score R is compared with this threshold: when R is lower than the preset safety interruption threshold R_th, it indicates that the impact of immediately interrupting after the current action is completed on system safety and task continuity is acceptable. At this time, the interruption decision unit generates an immediate interruption command, instructing the control execution module to stop continuing to execute the subsequent trajectory after the current micro-action or the current sub-action atomic unit is completed, and triggering the status recording unit to record the robotic arm status data and environmental status data at the time of interruption; when R is higher than the preset safety interruption threshold R_th, it indicates that there is a high risk of interruption near the current time. At this time, the interruption decision unit generates a delayed interruption command, instructing the system to continue to execute along the current sub-task trajectory, and triggering an interruption when reaching the next safety interruption anchor point or the nearest risk-acceptable anchor point, thereby reserving the nearest safe insertion window for temporary tasks without destroying the critical stage of the current sub-task.

[0112] Furthermore, to reduce the risk of interruption for the embodied intelligent robotic arm during complex operations, this embodiment sets mandatory constraint rules for specific types of sub-tasks in the safety strategy: For sub-tasks involving grasping, insertion, assembly, collision-sensitive operations, or high-speed movement, even if their interruptibility is marked as "conditionally interruptible," the interruption decision unit still mandates that interruption be performed only at pre-marked safe interruption anchor points, prohibiting mid-process interruptions during the contact establishment phase, high acceleration phase, or unstable gripping phase. When the current sub-task is detected to belong to the above-mentioned high-risk category, the system automatically increases the interruption risk score R or raises the corresponding safe interruption threshold R_th, causing the interruption decision logic to tend to choose delayed interruption or wait until the specified safe interruption anchor point before interruption, thereby avoiding abnormal force on the robotic arm, unexpected collisions between the end effector and the environment, or damage to the manipulated object due to overly aggressive interruption decisions.

[0113] Specifically, upon receiving a temporary task, the interruption decision unit no longer simply performs coarse-grained interruptions according to first-come, first-served or fixed rules. Instead, it quantitatively assesses and grades the interruption risk based on the coupling relationship between the temporary task's priority, urgency coefficient, the interruptibility flag of the currently executing subtask, and the safe interruption anchor point, combined with preset safety policies. This results in the generation of task interruption instructions that meet safety constraints and the real-time requirements of human-machine interaction. This risk-scoring and safety-policy-based interruption decision mechanism enables the embodied intelligent robotic arm to ensure the continuity of complex task execution and provide more flexible and reliable responses to user temporary needs and unexpected environmental events in scenarios where long-sequential and temporary tasks are intertwined.

[0114] S2.3: Record the robotic arm status data, environmental status data, and sequence of tasks to be executed at the moment of interruption, and generate a snapshot of the robotic arm status.

[0115] The specific steps of S2.3 are as follows:

[0116] S2.3.1: Upon receiving a task interruption command, multi-source state data at the moment of interruption is collected using a vision sensor and a robotic arm torque sensor. The multi-source state data includes:

[0117] Record the joint angles, joint velocities, joint torques, and end-effector poses of the robotic arm to generate robotic arm motion state data; record the object positions, object postures, and object categories within the workspace to generate environmental state data; record the identifiers, execution progress, and sequence of tasks to be executed for the currently executing sub-task to generate task execution state data.

[0118] S2.3.2: Encapsulate the robotic arm motion state data, environmental state data, and task execution state data according to a preset JSON format to generate a robotic arm state snapshot;

[0119] S2.3.3: Add a timestamp and a unique identifier to the snapshot of the robotic arm's state, store it in the local database, and establish a mapping index with the sequence of tasks to be executed.

[0120] Specifically, during the execution of long-sequence tasks, the robotic arm's own motion state, the spatial layout of objects in the workspace, and the sequence of tasks to be executed in the dynamic task queue are all constantly changing. If control commands are simply paused upon interruption without fully recording these multiple states, it will be difficult to accurately reconstruct the system context at the time of interruption during subsequent resumption of execution and task replanning. This could lead to discrepancies between the resumed execution trajectory and the original plan, disruption of pre- and post-task conditions, or neglect of environmental changes. Therefore, this step uses a state recording unit to synchronously collect robotic arm state data and environmental state data when a task interruption is triggered. Combined with the current sequence of tasks to be executed in the dynamic task queue, these three types of information are uniformly encapsulated into a structured state snapshot, providing a unified data benchmark for subsequent state comparison, executability assessment, and local replanning.

[0121] For example, once the interruption decision unit generates a task interruption command and the control execution module completes the current micro-action or the action at the current safety interruption anchor point, the state recording unit immediately triggers the interruption state acquisition process. First, the state recording unit simultaneously acquires multi-source state data at the moment of interruption through the vision detection module and the robotic arm's body sensors. The vision detection module is responsible for perceiving the layout information of environmental objects within the workspace, while the robotic arm sensors are responsible for acquiring the kinematic and dynamic state information of the robotic arm itself. By synchronously acquiring visual and sensor data under a unified time reference, the alignment of different types of data in the structured state snapshot across the time dimension is ensured, thereby avoiding state deviations introduced by inconsistent acquisition times.

[0122] Furthermore, for the robotic arm itself, the state recording unit reads the current joint angles, joint angular velocities, and joint torque parameters of each joint from the joint encoder, speed sensor, and torque sensor, and combines this with forward kinematics calculations to obtain the pose information of the end effector, including the position coordinates of the end effector in the working coordinate system and its attitude Euler angles (or quaternions), thereby generating robotic arm motion state data. In the robotic arm motion state data, the joint angles are uniformly converted to preset standard units and joint number sequence, and the end effector pose adopts a coordinate system and attitude representation consistent with the task planning module to ensure that the state snapshot can be directly reused by the subsequent task replanning module and control execution module. For robotic arms with joint overload protection or collision detection functions, this embodiment can also record the current joint current, contact force estimate, or collision detection flag for subsequent analysis to determine whether the interruption occurred under abnormal force conditions.

[0123] Furthermore, regarding the environmental state, the state recording unit uses visual detection algorithms to perform target detection and pose estimation on the images or point cloud data acquired at the time of the interruption. This identifies the position, pose, and category information of each target object and obstacle within the workspace, generating environmental state data. In this environmental state data, the position and pose of each object are represented in a unified coordinate system. Object categories can be encoded using preset semantic labels or category numbers, allowing the subsequent task evaluation unit to determine the executability of corresponding subtasks based on whether the target object has been displaced or removed. Simultaneously, to improve the expressive power of state snapshots for complex scenes, the environmental state data can also include auxiliary information such as ambient lighting conditions, camera pose, and workbench reference markers, used to assist in restoring environmental geometry when the interruption interval is long.

[0124] Furthermore, regarding task execution status, the status recording unit reads the currently executing subtask identifier, current execution progress (e.g., completed action stages, trajectory execution ratio), and task sequences still in the pending execution state from the dynamic task queue at the time of interruption, and organizes them into task execution status data. For each pending subtask, the status recording unit preferably records its target position, target posture, execution preconditions, interruptibility flag, and associated safety interruption anchor point information to ensure that during subsequent local replanning, the task replanning module can perform fine-grained rearrangement and correction based on the task semantics and constraints before the interruption. By incorporating the complete state of the dynamic task queue at the time of interruption into a structured state snapshot, task-level time freezing can be achieved, avoiding the inconsistency problem caused by only recording the current subtask and ignoring subsequent task chains.

[0125] Please see Figure 4 The schematic diagram of local replanning provided in this embodiment of the invention shows the specific steps of S3 as follows:

[0126] S3.1: Obtain the current joint angles, end-effector pose, and environmental object positions of the robotic arm, compare and analyze them with the robotic arm state data at the time of interruption, and generate state difference data.

[0127] Specifically, when an embodied intelligent robotic arm resumes execution of an interrupted task after completing a temporary task, the joint postures of the robotic arm itself, the spatial pose of the end effector, and the layout of various target objects and obstacles in the workspace may have changed. If the planned trajectory and task parameters before the interruption are directly used without verifying the consistency between the current state and the state at the time of interruption, it is easy to cause risks such as trajectory deviation, grasping failure, or collision with newly added obstacles during the recovery process. Therefore, in this step, the state comparison unit obtains the current joint angles, end effector pose, and environmental object positions of the robotic arm based on visual detection, and compares and analyzes them one by one with the robotic arm state data recorded in the structured state snapshot at the time of interruption. This generates state difference data that can quantitatively describe the differences from before the interruption to the present, providing input basis for subsequent executability judgment and local replanning.

[0128] In one embodiment, the state comparison unit first acquires images or point cloud data of the current workspace using a camera installed near the robot arm base or at the end effector. It then uses a vision detection module to perform object detection and pose estimation on this data to obtain the current positions of objects in the environment, including the position coordinates and attitude information of each target object and obstacle in a unified world coordinate system or the robot arm base coordinate system. Simultaneously, the state comparison unit communicates with the robot arm controller to read in real-time the current joint angles output by each joint encoder, as well as the current pose of the end effector obtained based on forward kinematics calculations. To ensure consistency in the comparative analysis, this embodiment uniformly converts the current joint angles, end effector pose, and environmental object positions to the same coordinate system, attitude representation, and data precision as in the structured state snapshot at the time of interruption.

[0129] Furthermore, regarding the state of the robotic arm itself, the state comparison unit performs differential calculations between the current joint angle vector and the joint angle vector recorded in the structured state snapshot at the time of interruption, obtaining the angle difference for each joint. Based on the displacement difference and attitude difference of the end effector (e.g., position difference vector and attitude difference quaternion), it quantitatively characterizes the degree of spatial offset of the robotic arm's end effector. Preferably, allowable deviation thresholds and end effector pose allowable deviation thresholds can be set for each joint. When the angle difference of a joint or the end effector pose offset exceeds the corresponding threshold, the joint or end effector state is marked as significantly deviated in the state difference data; otherwise, it is marked as acceptable deviation. Through the above processing, while ensuring numerical accuracy, complex joint and pose differences can be abstracted into engineering-meaning difference labels, facilitating rapid decision-making by subsequent modules.

[0130] Furthermore, regarding the state of environmental objects, the state comparison unit establishes data associations between the current set of environmental objects and the set of objects in the environmental state data at the time of the interruption, based on object identification information, category labels, and spatial proximity relationships. For object pairs that have successfully established associations, their position difference vectors and orientation differences are calculated and compared with preset displacement and rotation thresholds: when both the object's displacement and orientation changes are within the threshold range, the object is marked as not significantly moved; when the object's displacement or orientation changes exceed the threshold, the object is marked as moved, and its specific displacement and rotation amounts are recorded in the state difference data. Objects that exist only in the snapshot at the time of the interruption but are missing in the current detection results are marked as removed; objects that exist only in the current detection results but are not present in the snapshot at the time of the interruption are marked as newly added objects, and their position, size, and category information are recorded. Through the above association and difference process, the changes in the object layout within the workspace before and after the interruption can be intuitively reflected.

[0131] S3.2: Based on the state difference data, perform an executability judgment on each subtask in the sequence of tasks to be executed, and identify executable subtasks, subtasks that require parameter modification, and failed subtasks.

[0132] The specific steps of S3.2 are as follows:

[0133] S3.2.1: Obtain the target location and execution prerequisites of each subtask in the sequence of tasks to be executed;

[0134] S3.2.2: Match the positions of environmental objects in the state difference data with the target positions of each sub-task, and calculate the change in the target position of the sub-task;

[0135] S3.2.3: When the change in the target position of a subtask is less than the preset position tolerance threshold and the execution prerequisites meet the execution requirements, the subtask is marked as an executable subtask.

[0136] S3.2.4: When the target position change of a subtask is greater than or equal to the preset position tolerance threshold, but the execution prerequisites meet the execution requirements, the subtask is marked as a subtask that needs parameter correction, and the target position change is used as the correction parameter.

[0137] S3.2.5: When the execution preconditions of a subtask are no longer met, mark the subtask as an invalid subtask.

[0138] Specifically, since the actual operating state of the embodied intelligent robotic arm during the interruption recovery phase may differ from its state at the time of interruption—for example, the joint angles of the robotic arm may drift, the end-effector pose may deviate, the target object may move, or some objects may be removed or added—if the sub-tasks in the task sequence to be executed are directly executed according to the task parameters recorded before the interruption, it may lead to safety risks such as target misalignment, insertion and alignment failure, or collision between the motion trajectory and the environment. Therefore, in this step, the executability judgment unit, based on the state difference data output by the state comparison unit, performs condition verification and geometric feasibility analysis on each sub-task in the task sequence to be executed. Combining the pre-set execution prerequisites and safety constraints of the sub-tasks, each sub-task is divided into directly executable sub-tasks, parameter-modifying sub-tasks requiring adjustment of target pose / path parameters, and failed sub-tasks that cannot continue execution due to the complete destruction of prerequisites. This achieves automatic hierarchical management of the task sequence after the interruption.

[0139] In one embodiment, the executability determination unit first reads the sequence of tasks to be executed and their associated subtask information from a structured state snapshot. Each subtask includes at least: subtask type (e.g., "approach," "grab," "move," "place," "plug," etc.), target object identifier, target pose, predetermined path or safe workspace constraints, execution prerequisites, and interruptibility flags. Subsequently, the subtask information is associated with state difference data, which includes joint angle deviations, end-effector pose deviations, and displacement, attitude changes, and addition / disappearance flags of the target object and obstacles. By matching the difference data with the corresponding subtasks according to the target object identifier and task stage, a subtask difference description containing "current environment / attitude change" information is constructed for each subtask.

[0140] Furthermore, in this embodiment, the executability judgment unit preferably adopts a hierarchical threshold judgment strategy, performing a precondition consistency check and a geometric feasibility check sequentially for each subtask. During the precondition consistency check phase, based on the execution preconditions recorded in the subtask (e.g., "the target object exists and is not obscured by other objects," "the target object is within the designated area of ​​the workbench," "a certain upstream subtask has been completed," etc.), and combined with the object presence marker, object position label, and task completion marker in the state difference data, it is determined whether the logical premise of the subtask still holds. If it is detected that the target object has been removed, a new obstacle completely obscures the target object, or a preceding critical subtask has been determined to be invalid, then the subtask is directly marked as an invalid subtask.

[0141] Furthermore, in the geometric feasibility check phase, for subtasks where the preconditions still hold, the executability assessment unit constructs geometric executability indicators based on end-effector pose deviation, target object position deviation, and environmental obstacle changes. For example, for motion subtasks such as "approaching," "moving," and "leaving," the executability assessment unit compares the spatial relationship between the target pose and obstacles in the current environment, combined with the kinematic constraints and safety distance constraints of the robotic arm, to assess whether a collision-free trajectory still exists. For fine-operation subtasks such as "grasping," "placing," and "inserting," the focus is on checking whether the offset between the current position of the target object and the original target pose is within a preset compensable threshold range, and whether the robotic arm end-effector can still meet operational constraints such as normal contact and gripping space under the adjusted posture. If no feasible solution is found in the trajectory planning or constraint verification, the subtask is judged as a failed subtask; if a feasible solution exists, but the original target pose, grasping posture, or intermediate path needs to be adjusted, the subtask is marked as a subtask requiring parameter modification.

[0142] Specifically, after obtaining the state difference data, this embodiment performs fine-grained executability judgment and classification management on each subtask in the sequence of tasks to be executed. It can not only automatically filter out executable subtasks that can be directly resumed, but also identify subtasks that require local replanning or parameter updates and need to modify parameters. It can also promptly remove failed subtasks whose logical premises have been broken or which are no longer geometrically feasible, providing clear task-level input for subsequent task replanning and execution sequence reorganization, and ensuring that the execution behavior of the embodied intelligent robotic arm is safe, consistent and efficient in interruption recovery scenarios.

[0143] S3.3: Based on the task consistency index, perform a local replanning operation on the sequence of tasks to be executed to generate a replanned task sequence.

[0144] The specific steps of S3.3 are as follows:

[0145] S3.3.1: Maintain the execution order of all executable subtasks to preserve structural consistency;

[0146] S3.3.2: For subtasks that require parameter modification, update the target position and target attitude according to the modified parameters to maintain spatial consistency;

[0147] S3.3.3: Identify all failed subtasks and generate transitional subtasks through a large language model, wherein the transitional subtasks are used to connect the executable subtasks before and after the failed subtasks to maintain time consistency.

[0148] S3.3.4: Construct an optimization objective function based on the task consistency index, with the weighted sum of structural consistency, spatial consistency and temporal consistency as the optimization objective. Use the task sequence optimization algorithm to locally adjust the sequence of tasks to be executed and generate a replanning task sequence.

[0149] Specifically, due to the shift in execution context of the original initial task sequence after temporary task execution and environmental changes, simply continuing the remaining subtasks in the original order after removing failed subtasks may lead to broken task semantic chains, disrupted dependencies, or even duplicated operations or omissions of key steps. Therefore, in this step, the task replanning module introduces a task consistency index based on the aforementioned state difference data and executability judgment results to perform local replanning operations on the task sequence to be executed: while ensuring that the overall task intent remains unchanged and key constraints are met, the module adjusts the order of local subtask chains, replaces parameters, and inserts necessary compensation tasks, thereby generating a replanned task sequence that is semantically consistent, temporally reasonable, and motion feasible in the current environment and robotic arm state.

[0150] In one embodiment, task consistency metrics are used to quantify the overall consistency between the replanned task sequence after interruption recovery and the target task before interruption. These metrics include at least: semantic consistency metrics, dependency consistency metrics, temporal continuity metrics, and motion feasibility metrics. Semantic consistency metrics measure the degree of agreement between the replanned subtask set and the original task intent in terms of target objects, operation types, and results. For example, whether it still completes tasks such as "moving a specified object from area A to area B" or "completing a certain assembly step." Dependency consistency metrics constrain the causal order between subtasks, such as whether the "approach → grasp → move → place" chain is maintained, avoiding out-of-order execution or missing key actions. Temporal continuity metrics assess the rationality of the connection between replanned subtasks on the timeline, avoiding meaningless back-and-forth movements and frequent long-distance switching. Motion feasibility metrics, combined with the current obstacle layout and robotic arm kinematic constraints, evaluate whether the replanned subtask set can generate a collision-free trajectory as a whole. These metrics can be implemented by combining a rule base with a task semantic graph generated by a large language model, mapping the natural language task description into computable structured constraints.

[0151] Furthermore, in this embodiment, after receiving the classification results of executable subtasks, subtasks requiring parameter modification, and failed subtasks output in step S3.2, the task replanning module first removes all nodes marked as failed subtasks from the sequence of tasks to be executed, and organizes the parameter update requirements of subtasks requiring parameter modification (such as new target pose, corrected grasping posture, replanned approach path constraints, etc.) into a parameter adjustment list. Subsequently, using the most recently completed subtask before the interruption as the anchor point, it takes this as the starting point for replanning, and performs local rearrangement and parameter injection on the subsequent subtask sequence without changing the already executed task segments and remote task segments that are not directly coupled with this segment. This achieves the replanning strategy of "local modification, global preservation," avoiding the computational overhead and logical complexity caused by completely reconstructing the entire task sequence.

[0152] Furthermore, during the local replanning process, the task replanning module preferentially employs graph search or constraint optimization methods to combine and sort executable subtasks and subtasks requiring parameter modification. Specifically, subtasks in the sequence of tasks to be executed can be modeled as nodes in a weighted directed graph. Edges between nodes represent dependencies and switchability relationships, and edge weights can comprehensively consider temporal continuity and motion feasibility in the task consistency index. For example, shorter spatial movement distances and fewer pose transitions correspond to smaller weights. Based on this, a feasible path with the minimum total weight is searched while satisfying dependency consistency constraints, resulting in a subtask execution order that balances execution efficiency and action continuity. For subtasks requiring parameter modification, the target pose and trajectory constraints after parameter adjustment are introduced during the graph search process, and their impact on motion feasibility index is dynamically evaluated to ensure that the replanning result is geometrically feasible.

[0153] Specifically, this step, without disrupting the overall task objectives and key dependencies, performs local replanning operations on the sequence of tasks to be executed, based on task consistency indicators. This fully utilizes the value of executable and repairable subtasks while effectively eliminating interference from failed subtasks to subsequent execution. The resulting replanned task sequence can adapt to changes in the current state of the robotic arm and the environment, ensuring that the embodied intelligent robotic arm can complete the remaining tasks in a logically coherent, motion-safe, and semantically consistent manner even after task interruptions and temporary task insertions.

[0154] The specific steps of S3.3.4 are as follows:

[0155] S3.3.4.1: Initialize the replanning sequence by adding all executable subtasks to the replanning sequence in their original order, and generating candidate replanning sequences;

[0156] S3.3.4.2: Traverse all failed subtasks, analyze the robot arm pose and environmental state before and after the failed subtasks through visual detection, obtain the corresponding transition subtasks, and generate a set of transition subtasks;

[0157] S3.3.4.3: For each failed subtask location, select the transitional subtask from the transitional subtask set that maximizes the increment of the optimization objective function and insert it;

[0158] S3.3.4.4: For subtasks that require parameter modification, update the subtask parameters by minimizing the end-effector pose error;

[0159] S3.3.4.5: Adjust the execution order and parameter configuration of subtasks in the replanning sequence through iterative optimization algorithm, so that the optimization objective function converges to a local optimum and outputs the final replanning task sequence.

[0160] Specifically, after interruption recovery and the insertion of temporary tasks, the order, spatial distribution, and expected execution time windows of the subtasks in the task sequence to be executed may change. Simply relying on heuristic rules for reordering is insufficient to ensure both semantic integrity and efficient execution timing. Therefore, in this step, the task replanning module, based on the aforementioned task consistency index, integrates structural consistency, spatial consistency, and temporal consistency into a single optimization framework. By constructing a weighted summation-based objective function, the replanning problem is transformed into a constrained sequence optimization problem. A task sequence optimization algorithm is then used to adjust the task sequence to be executed within the local search space, thereby obtaining a replanned task sequence with optimal or near-optimal overall consistency.

[0161] In one embodiment, the task replanning module first constructs a task graph model based on the sequence of tasks to be executed, representing each subtask as a node in the graph. Directed edges between nodes are used to characterize the sequential dependencies and switchable relationships between tasks. Based on this, and combining the executability judgment results and task semantic information, three types of consistency indicators are calculated for each candidate replanning sequence: The structural consistency indicator measures the fidelity of the replanned task sequence in the task chain structure, such as whether the original key subtask chain order ("approach → grasp → move → place") is maintained, and whether the task grouping and stage division in the original sequence are preserved as much as possible; the spatial consistency indicator characterizes the rationality of the subtasks' spatial execution paths, such as whether the spatial distance between the target poses of adjacent subtasks is small, and whether the overall motion trajectory avoids large-scale ineffective back-and-forth movements; the temporal consistency indicator reflects the temporal rationality of the task execution after replanning, such as whether the subtask time window constraints are met, whether excessive delays to high-priority tasks are avoided, and whether the time overhead caused by frequent context switching is reduced.

[0162] Furthermore, in this embodiment, the task replanning module constructs an optimization objective function based on structural consistency indicators, spatial consistency indicators, and temporal consistency indicators. The calculation formula is as follows:

[0163]

[0164] in, To obtain the structural consistency score, calculate the proportion of subtask types retained between the original task sequence and the replanning sequence; To obtain the spatial consistency score, calculate the reciprocal of the spatial deviation between the target position of each subtask in the replanning sequence and the original target position; To obtain a time consistency score, calculate the proximity between the expected execution time of the replanning sequence and the original execution time; , , The weighting coefficients are and satisfy the following conditions: + + =1, used to characterize the importance of structural consistency, spatial consistency, and temporal consistency in the current application scenario. For scenarios emphasizing strict adherence to the original process flow, the weight of α can be appropriately increased to ensure that the optimization process prioritizes maintaining the task structure without disruption. For scenarios emphasizing motion efficiency and energy consumption, the weight of β can be increased to make the optimization process more inclined to shorten spatial paths and reduce redundant actions. For scenarios involving time window constraints or requiring rapid response to high-priority tasks, the weight of γ can be increased to ensure the rationality of the overall execution sequence after replanning. In this embodiment, α, β, and γ can be adaptively adjusted through preset configurations, empirical rules, or based on historical execution data.

[0165] Furthermore, to transform the objective function into a solvable task sequence optimization problem, this embodiment introduces a set of constraints on the candidate replanning sequences, including but not limited to: task dependency constraints (downstream subtasks must not execute before upstream subtasks), safety constraints (during task execution, constraints on robotic arm joint limits, velocity and acceleration limits, and collision avoidance must be met), executability constraints (only subtasks determined to be executable or requiring parameter modification are allowed, and the latter must use the updated task parameters), and interruption recovery constraints (the replanning start point should be compatible with the ending state of the most recently completed subtask before the interruption). Based on this, all task sequences that satisfy the constraints are considered as a feasible solution set. With maximizing the objective function J as the optimization objective, the optimal or near-optimal task execution order and parameter combination are selected from the feasible solution set.

[0166] In a preferred embodiment, the task sequence optimization algorithm employs a graph search-based heuristic optimization method, including but not limited to: A* search algorithm with a heuristic cost function, heuristic depth-first / breadth-first search, or metaheuristic methods such as genetic algorithms encoded by task sequences and simulated annealing algorithms. Specifically, the task sequence to be executed can be modeled as the starting node, and all legal swap, insertion, or merging operations that satisfy dependency constraints are considered as state transitions. By swapping adjacent subtasks or fine-tuning the positions of subtasks requiring parameter modification within a local window, candidate replanning sequences are gradually generated. During the search process, the aforementioned objective function J is used as the evaluation function to comprehensively evaluate the structural consistency, spatial consistency, and temporal consistency of the candidate sequences, and the search direction is guided by the heuristic function, thereby quickly converging to a highly consistent task sequence under limited computing resources.

[0167] Furthermore, to limit computational overhead, this embodiment preferably employs a local replanning strategy for task sequence optimization: that is, sequence adjustments are only performed within a local window centered on the interruption point, covering several subtasks after the interruption, while the sequences of distant tasks outside the window remain unchanged. This "local adjustment, global inheritance" approach significantly reduces the search space size and accelerates optimization convergence; it also helps maintain the macroscopic stability of the overall task structure and improves structural consistency metrics. Simultaneously, within the local replanning window, for subtasks requiring parameter modifications, their "target pose update" or "path constraint adjustment" can be introduced as additional parameter dimensions into the optimization process, allowing the spatial consistency metric to consider both task order and trajectory smoothness and energy consumption.

[0168] Furthermore, in some embodiments, the task replanning module can also adjust the weight configuration and some evaluation items of the optimization objective function online based on historical execution data and human-computer interaction feedback. For example, when the system detects frequent instances of user manual intervention to reorder or cancel some tasks after replanning, it can adjust the contribution weights of different types of subtasks in the structural consistency and time consistency evaluation through statistical analysis, making the subsequent replanning results more in line with the user's implicit preferences. This adaptive adjustment mechanism of the optimization objective based on task consistency indicators enables the human-computer interaction system of the intelligent robotic arm of this invention to have a continuous "self-evolving" replanning capability.

[0169] Through the above design, this step constructs a unified weighted optimization objective function based on the clear definition of three types of task consistency indicators: structural consistency, spatial consistency, and temporal consistency. It also combines a task sequence optimization algorithm to make local adjustments to the sequence of tasks to be executed, and finally obtains a replanning task sequence with better overall consistency under the current state of the robotic arm and environmental layout. This improves the execution efficiency and interactive experience after interruption recovery while ensuring safety and semantic integrity.

[0170] The specific steps for S4 are as follows:

[0171] S4.1: Perform pre-execution verification on the replanning task sequence using a visual detection algorithm to determine whether it meets the constraints, which include resource conflict detection, robotic arm posture safety detection, and environmental collision detection.

[0172] The specific steps of S4.1 are as follows:

[0173] S4.1.1: Perform kinematic simulation on each subtask in the replanning task sequence, detect whether the execution trajectory of the robotic arm collides with environmental obstacles, detect whether there are conflicts or insufficient resources in the end effector, sensors and computing resources required for each subtask, detect whether the joint angle and joint speed of each subtask are within the safe operating range of the robotic arm, and generate the constraints of the replanning task sequence.

[0174] S4.1.2: When all the constraints are met, a verification pass flag is generated, and action control instructions are generated according to the replanning task sequence;

[0175] S4.1.3: When any constraint condition is not met, a constraint violation report and a replanning trigger signal are generated, and the constraint violation report is fed back to the task replanning module for iterative replanning until all constraints are met.

[0176] Specifically, the replanning task sequence is regenerated based on the current state difference data after interruption recovery and temporary task insertion. Although task consistency and motion feasibility have been considered during the replanning process, there is still a possibility that some sub-tasks may encounter safety hazards such as resource contention, robotic arm posture exceeding limits, or potential environmental collisions due to further changes in the environment, changes in upper-level system resource usage, or deviations caused by approximate solutions in the replanning process. If the replanning task sequence is not uniformly validated for constraints before execution and is directly sent to the control execution module, the risk of robotic arm instability, misuse of grippers, or interference with the environment and personnel will be significantly increased. Therefore, in this step, the system performs a pre-execution validation of the replanning task sequence based on the latest visual detection results and the current state of the robotic arm. It comprehensively checks whether the constraints are met from three dimensions: resource layer, posture layer, and environment layer. Only when all constraints pass the validation is the replanning task sequence sent as the final execution plan.

[0177] In one embodiment, the pre-execution verification module first acquires images or point cloud data of the current workspace using a camera, and uses a vision detection module to identify the position and orientation of environmental resources such as workbenches, fixtures, tools, materials, and obstacles, generating a latest environmental state description. Simultaneously, it reads the current joint angles, joint speeds, and end-effector pose information from the robotic arm controller as initial attitude boundary conditions for verification. Subsequently, the pre-execution verification module parses the replanning task sequence into a series of sub-task execution segments with time-sequence markers. Each segment includes a target object identifier, target pose, expected tool or fixture resources, and planned trajectory or critical path points. Based on the above parsing results, the resource conflict detection module, attitude safety detection module, and environmental collision detection module are sequentially invoked to perform multi-dimensional constraint checks on the replanning task sequence.

[0178] Furthermore, in terms of resource conflict detection, this embodiment constructs a resource occupancy timeline to uniformly model the resource requests of each subtask in the replanning task sequence. Specifically, the system determines the set of resources that each subtask needs to occupy during execution based on its semantic information and parameter configuration. These resources include, but are not limited to, end effector types (such as two-finger grippers, three-finger grippers, and suction cups), dedicated tooling fixtures, temporary storage trays, area identifiers of the work area, and the specific target object itself. The pre-execution verification module maps each subtask to a "resource-time interval" occupancy record under a unified time reference. By scanning the overlapping relationships between resource occupancy records in the replanning task sequence, it determines whether the same resource is requested simultaneously by multiple subtasks within the same time interval. For example, if a target object has been marked as "grabbed but not placed" by a previous subtask during a certain time period, subsequent subtasks that simultaneously use this object as a grabbing object or as a workpiece fixed on a tooling fixture will be considered to have a resource conflict. Similarly, for fixtures or dedicated tooling that require exclusive use, if multiple subtasks have overlapping usage intervals on the timeline, they will also be marked as having a resource conflict. If a resource conflict is detected, the pre-execution verification module can return the conflict subtask identifier and the list of conflicting resources, triggering the previous step of local replanning or task adjustment.

[0179] Furthermore, regarding the posture safety detection of the robotic arm, this embodiment performs offline simulation and point-by-point safety verification of the posture changes of the robotic arm during execution based on the planned trajectory corresponding to each subtask in the replanning task sequence. Specifically, the posture safety detection module takes the current joint angle at the start of the replanning task sequence as the starting point, performs time-discrete sampling on each trajectory according to the planned trajectory, calculates the joint angle, joint velocity, and joint acceleration at the sampling point, and compares them with the pre-set joint angle limit and velocity / acceleration upper limit to determine whether there is a violation of the limit. At the same time, the system calculates the Jacobian matrix and operability index of the robotic arm at each sampling point to identify dangerous postures that are close to singular configurations or have low operability. For robotic arms with a self-collision model, the minimum distance between links and between links and the base can also be checked during the simulation. If it is lower than the self-collision safety distance threshold, it is considered that the posture safety is not met. If there are continuous or multiple violations of posture safety constraints on a certain trajectory, the pre-execution verification module marks the subtask as "posture infeasible" and feeds back the specific violation location to the replanning module for trajectory replanning.

[0180] Furthermore, in terms of environmental collision detection, this embodiment utilizes the latest environmental model generated by visual detection (e.g., represented as a point cloud, voxel grid, or bounding box set) to perform simulated collision detection on each trajectory in the replanning task sequence. Specifically, the collision detection module calculates the geometric envelope of each link and end effector of the robotic arm in space under the same temporal distance, queries the distance between this envelope and obstacles, tooling, and personnel safety zones in the environmental model, and calculates the minimum safe distance. When the minimum safe distance at any sampling point is less than a preset collision safety distance threshold, the trajectory is determined to have a potential collision risk at the corresponding time. If the minimum distance is zero or the geometric shapes intersect, it is determined to be a certain collision risk. For scenarios involving human bodies or human-machine collaboration areas, the system can also set a higher safe distance threshold around personnel to achieve a more conservative safety strategy. If an environmental collision risk is detected, the pre-execution verification module also marks the corresponding sub-task as not meeting environmental constraints and feeds back the risk location and risk object to the upstream replanning module.

[0181] Preferably, this embodiment can also comprehensively evaluate the resource conflict detection results, attitude safety detection results, and environmental collision detection results to form a pre-execution verification report. The pre-execution verification module can calculate the overall constraint satisfaction score for the replanning task sequence. For example, resource constraints, attitude constraints, and environmental constraints are mapped to sub-scores in the interval [0,1], and the total score is obtained by weighted summation. When the total score is lower than a preset threshold or any key constraint has a serious violation, the replanning task sequence is considered to be unqualified for execution. At this time, the system can automatically trigger a local replanning process or request user confirmation or task adjustment through the human-computer interaction interface. When all constraints are satisfied and the score is higher than the threshold, the replanning task sequence is marked as "safe and executable", and the verification pass mark is stored in the task log as the basis for subsequent traceability and security audit.

[0182] Through the above design, this step introduces a visual detection-based pre-execution verification mechanism before the replanning task sequence is issued and executed. It conducts a comprehensive constraint check on the replanning task sequence from three dimensions: resource conflict, robotic arm posture safety, and environmental collision. This can detect potential risks in advance before the control commands are actually applied to the robotic arm, avoiding safety accidents caused by resource competition, posture exceeding limits, or environmental collisions. This significantly improves the execution safety and system robustness of the embodied intelligent robotic arm in human-computer interaction scenarios.

[0183] S4.2: When the replanning task sequence meets the constraints, generate motion control instructions based on the replanning task sequence to control the robotic arm to return to the execution state before the interruption and continue execution. When the replanning task sequence does not meet the constraints, generate a replanning trigger signal and replan through the task replanning module.

[0184] Specifically, since this embodiment addresses a human-computer interaction scenario where long-term tasks and temporary tasks are intertwined, the actual state of the robotic arm after experiencing interruption, state evaluation, and task replanning often deviates from the ideal state recorded at the time of interruption. If action instructions corresponding to the replanned task sequence are directly issued without unified coordination, problems such as "inconsistent execution starting points," "sudden action jumps," or "local task duplication / omission" can easily occur, leading to task logic confusion or new safety risks. Therefore, based on the pre-execution verification results, this step involves the control execution module uniformly parsing the replanned task sequence and the interruption state snapshot. On the one hand, when the verification passes, it generates action control instructions that conform to the controller interface specification and can smoothly connect the states before and after the interruption; on the other hand, when the verification fails, it feeds back the reason why the current constraints cannot be met to the task replanning module in the form of a replanning trigger signal, forming a closed-loop control link of "verification-execution-replanning".

[0185] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An embodied intelligent robotic arm human-machine interaction system, applied to human-machine interaction scenarios that combine long-term sequential tasks with temporary tasks triggered by operator interaction or workspace environmental events, characterized in that, The long-sequence task is decomposed into multiple sub-tasks using a pre-defined large language model to generate an initial task sequence. The system includes: The task acquisition module is used to identify the triggering conditions of temporary tasks formed by operator interaction behavior or workspace environmental events through visual detection algorithms, and to determine the priority of the temporary tasks and generate the priority of the temporary tasks. The task queue management module is used to input the initial task sequence into a dynamic task queue. The dynamic task queue includes multiple task records maintained in execution order. Each task record includes at least a subtask identifier ID, subtask type, target position and target posture, execution preconditions, current execution progress, interruptible flag, and a set of safe interruption anchor points. The dynamic task queue generates a task interruption instruction based on the priority of the temporary task and the safe interruption policy, and records the robotic arm status data and the sequence of tasks to be executed at the time of interruption. The task replanning module is used to perform local replanning on the sequence of tasks to be executed based on the robotic arm state data at the time of the interruption and the current robotic arm state data after the interruption is resumed, and output the replanned task sequence. The control execution module is used to perform pre-execution verification of the replanning task sequence through a visual detection algorithm, and generate motion control commands for the robotic arm based on the verification results, thereby controlling the robotic arm to execute interactive tasks.

2. The human-machine interaction system of the embodied intelligent robotic arm according to claim 1, characterized in that, The task queue management module includes a queue maintenance unit, an interruption decision unit, and a status recording unit, including: The queue maintenance unit is used to maintain the interruptible flag and safe interruption anchor point for each subtask in the dynamic task queue. The safe interruption anchor point represents the time node during the execution of the subtask that is allowed to be interrupted. The interruption decision unit is used to generate a task interruption instruction based on the priority of the temporary task and the interruptibility flag of the currently executing subtask when a temporary task is received. The state recording unit is used to record the robotic arm state data, environmental state data, and the sequence of tasks to be executed at the time of interruption, and to generate a snapshot of the robotic arm state.

3. The human-machine interaction system of the embodied intelligent robotic arm according to claim 2, characterized in that, The interrupt decision unit is configured with interrupt point selection logic, which includes: Get the priority and urgency coefficient of temporary tasks; Identify the interruptible flag of the currently executing subtask. If the currently executing subtask is marked as uninterruptible, search the dynamic task queue for the safe interrupt anchor point closest to the current executing subtask. Calculate the interruption risk score based on the priority, urgency coefficient, and most recent security interruption anchor point of the temporary task; When the interruption risk score is lower than the preset safety interruption threshold, an immediate interruption command is generated to control the robotic arm to interrupt the current sub-task after it is completed. When the interruption risk score is higher than or equal to the preset safety interruption threshold, a delayed interruption command is generated to control the robotic arm to interrupt after reaching the next safety interruption anchor point.

4. The human-machine interaction system of the embodied intelligent robotic arm according to claim 1, characterized in that, The task replanning module includes a state comparison unit, a task evaluation unit, and a local replanning unit, comprising: The state comparison unit is used to obtain the joint angle, end pose and position of the current robotic arm and compare them with the robotic arm state data at the time of interruption to generate state difference data. The task evaluation unit is used to determine the executability of each subtask in the sequence of tasks to be executed based on the state difference data, and to identify executable subtasks, subtasks that need to modify parameters, and failed subtasks. The local replanning unit performs a local replanning operation on the sequence of tasks to be executed based on the task consistency index, thereby generating a replanned task sequence. The task consistency index is used to quantify the overall consistency between the replanned task sequence after interruption recovery and the target task before the interruption.

5. The human-machine interaction system of the embodied intelligent robotic arm according to claim 4, characterized in that, The task evaluation unit is configured with sub-task judgment logic, which includes: Obtain the target location and execution prerequisites for each subtask in the sequence of tasks to be executed; The positions of environmental objects in the state difference data are matched and analyzed with the target positions of each sub-task to calculate the change in the target position of the sub-task. When the change in the target position of a subtask is less than the preset position tolerance threshold and the execution prerequisites are met, the subtask is marked as an executable subtask. When the change in the target position of a subtask is greater than or equal to the preset position tolerance threshold, but the execution prerequisites meet the execution requirements, the subtask is marked as a subtask that needs parameter correction, and the change in the target position is used as the correction parameter. When the preconditions for the execution of a subtask are no longer met, the subtask is marked as an invalid subtask.

6. The human-machine interaction system of the embodied intelligent robotic arm according to claim 5, characterized in that, The local replanning unit is configured with replanning logic, which includes: The execution order of all executable subtasks remains unchanged, maintaining structural consistency; For subtasks that require parameter modification, update the target position and target attitude based on the modified parameters to maintain spatial consistency; Identify all failed subtasks and generate transitional subtasks through a large language model. These transitional subtasks are used to connect executable subtasks before and after the failed subtasks to maintain time consistency. An optimization objective function is constructed based on the task consistency index. The optimization objective is to maximize the weighted sum of structural consistency, spatial consistency and temporal consistency. The task sequence optimization algorithm is used to locally adjust the sequence of tasks to be executed and generate a replanning task sequence.

7. The human-machine interaction system of the embodied intelligent robotic arm according to claim 6, characterized in that, The step of locally adjusting the sequence of tasks to be executed using a task sequence optimization algorithm to generate a replanning task sequence includes: Initialize the replanning sequence by adding all executable subtasks to the replanning sequence in their original order, and generate candidate replanning sequences. Traverse all failed subtasks, analyze the robot arm pose and environmental state before and after the failed subtasks through visual detection, obtain the corresponding transition subtasks, and generate a set of transition subtasks. For each failed subtask location, select the transition subtask from the transition subtask set that maximizes the increment of the optimization objective function and insert it; For subtasks that require parameter modification, update the subtask parameters by minimizing the end-effector pose error; By adjusting the execution order and parameter configuration of subtasks in the replanning sequence through an iterative optimization algorithm, the optimization objective function converges to a local optimum, and the final replanning task sequence is output.

8. The human-machine interaction system of the embodied intelligent robotic arm according to claim 1, characterized in that, The control execution module includes a verification unit and an instruction generation unit, including: The verification unit is used to perform pre-execution verification on the replanning task sequence using a visual detection algorithm to determine whether the constraints are met. The constraints include resource conflict detection, robotic arm posture safety detection, and environmental collision detection. The instruction generation unit is used to generate motion control instructions based on the replanning task sequence when the replanning task sequence meets the constraints, control the robotic arm to return to the execution state before the interruption and continue execution, and generate a replanning trigger signal when the replanning task sequence does not meet the constraints, and replanning is performed through the task replanning module.

9. The human-machine interaction system of the embodied intelligent robotic arm according to claim 8, characterized in that, The verification unit is configured with constraint condition judgment logic, which includes: Kinematic simulations are performed on each subtask in the replanning task sequence to detect whether the robotic arm's execution trajectory collides with environmental obstacles. Based on the execution time interval and resource requests of each subtask, resource conflicts are detected in the end effector, sensors, and computing resources, where the same resource is requested by multiple subtasks simultaneously in the same time interval. The joint angles and joint speeds of each subtask are detected to be within the safe operating range of the robotic arm, and constraints are generated for the replanning task sequence. When all the constraints are met, a verification pass flag is generated, and action control instructions are generated based on the replanning task sequence. When any constraint is not met, a constraint violation report and a replanning trigger signal are generated. The constraint violation report is then fed back to the task replanning module for iterative replanning until all constraints are met.

10. The human-machine interaction system of the embodied intelligent robotic arm according to claim 2, characterized in that, The state recording unit is configured with state snapshot generation logic, which includes: Upon receiving a task interruption command, multi-source state data at the moment of interruption is collected using a vision sensor and a robotic arm torque sensor. This multi-source state data includes: Record the joint angles, joint velocities, joint torques, and end-effector pose of the robotic arm to generate motion state data of the robotic arm; Record the position, orientation, and category of objects within the workspace to generate environmental status data; Record the identifier, execution progress, and sequence of tasks to be executed for the currently executing subtask, and generate task execution status data; The robotic arm motion state data, environmental state data, and task execution state data are encapsulated in a preset JSON format to generate a robotic arm state snapshot. Add a timestamp and a unique identifier to the snapshot of the robotic arm's state, store it in a local database, and establish a mapping index with the sequence of tasks to be executed.

Images