Human-computer interaction system of intelligent mechanical arm with body

Abstract

The invention relates to the technical field of mechanical arms, in particular to a man-machine interaction system of an intelligent mechanical arm with a body. The system decomposes a long time sequence task into sub-tasks through a large language model, and introduces a task acquisition module, a task queue management module, a task re-planning module and a control execution module, uses visual detection to identify gestures and environment events to trigger temporary tasks, and maintains interruptible marks and safety anchor points based on priorities and interruption risks. And during interruption, the mechanical arm with the timestamp and the environment state are collected, during recovery, the states are compared, local re-planning is carried out, transition sub-tasks are automatically generated for failure sub-tasks, and the mechanical arm is controlled to execute after constraint verification. According to the system, on the premise that the structure consistency, the space consistency and the time consistency are guaranteed, safe interruption and efficient recovery in a long-time-sequence task can be achieved, and the autonomy, the real-time performance and the operation safety of the mechanical arm in a man-machine cooperation scene are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mechanical arms, in particular to a body-intelligent mechanical arm human-computer interaction system. BACKGROUND

[0002] With the development of intelligent manufacturing and human-computer collaboration technology, the autonomous execution capability of mechanical arms in complex dynamic environments has attracted much attention. In multi-task human-computer interaction scenarios, mechanical arms often need to handle long-time sequence planning tasks and temporary tasks with uncertainty at the same time. Traditional mechanical arm systems mainly rely on pre-set scripts and fixed control processes, and their task execution modes lack flexibility, making it difficult to dynamically adjust according to environmental changes or operator intentions.

[0003] Existing task switching mechanisms usually lack descriptions of task interruptibility, and do not systematically evaluate the priority, urgency and safety of action phases between different tasks, resulting in risks such as end pose instability, object falling or trajectory conflict when the mechanical arm is interrupted. In addition, when the mechanical arm resumes execution after task interruption, the body pose, target object position and environmental layout may have changed since the interruption, but existing technologies usually directly use the task parameters and path planning before the interruption, making it difficult to guarantee geometric feasibility and operational safety.

[0004] On the other hand, existing task re-planning techniques mostly use global re-planning methods, which are computationally complex and difficult to meet real-time requirements, and do not systematically consider multi-dimensional constraints such as task structure consistency, spatial consistency and temporal consistency, resulting in deficiencies in the generated recovery task sequence in terms of semantic logic, execution continuity and safety feasibility.

[0005] Therefore, how to realize intelligent trigger recognition of temporary tasks, task interruption decision based on interruptibility and safety strategy, task executability analysis for interruption recovery scenarios, and local task re-planning that meets multi-dimensional consistency constraints, has become a key technical problem to improve the autonomy and safety of mechanical arms in human-computer interaction scenarios. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a body-integrated intelligent robot arm human-computer interaction system, which can realize safe interruption and efficient recovery in long-time sequence tasks under the premise of ensuring structural consistency, spatial consistency and time consistency, and significantly improve the autonomy, real-time performance and operation safety of the robot arm in human-computer collaborative scenarios.

[0007] To achieve the above-mentioned purpose, the present application provides the following technical scheme:

[0008] The body-integrated intelligent robot arm human-computer interaction system is applied to a human-computer interaction scene mixed with long-time sequence tasks and temporary tasks, the long-time sequence tasks are decomposed into multiple subtasks by a preset large language model to generate an initial task sequence, and the system comprises:

[0009] A task acquisition module is configured to identify a triggering condition of a temporary task by a visual detection algorithm, determine a priority of the temporary task, and generate a priority of the temporary task.

[0010] A task queue management module is configured to input the initial task sequence into a dynamic task queue, generate a task interruption instruction according to the priority of the temporary task and a safe interruption strategy, and record robot arm state data and a to-be-executed task sequence at the time of interruption.

[0011] A task re-planning module is configured to perform local re-planning on the to-be-executed task sequence based on the robot arm state data at the time of interruption and current robot arm state data after interruption recovery, and output a re-planned task sequence.

[0012] A control execution module is configured to perform pre-execution verification on the re-planned task sequence by a visual detection algorithm, generate a robot arm action control instruction according to a verification result, and control the robot arm to execute an interactive task.

[0013] The task queue management module comprises a queue maintenance unit, an interruption decision unit and a state recording unit, and comprises:

[0014] The queue maintenance unit is configured to maintain an interruptable mark and a safe interruption anchor point of each subtask in the dynamic task queue, and the safe interruption anchor point represents a time node allowed to be interrupted in the execution process of the subtask.

[0015] The interrupt decision unit is configured to generate a task interrupt instruction according to the priority of the temporary task and the interruptable flag of the currently executed subtask when the temporary task is received.

[0016] The state recording unit is configured to record the mechanical arm state data, the environment state data and the to-be-executed task sequence at the interrupt moment, and generate a mechanical arm state snapshot.

[0017] The interrupt decision unit is configured with an interrupt point selection logic, and the interrupt point selection logic comprises:

[0018] The priority and the emergency coefficient of the temporary task are obtained.

[0019] The interruptable flag of the currently executed subtask is identified, and if the currently executed subtask is marked as an un-interruptable state, a nearest safe interrupt anchor point in the dynamic task queue from the position of the currently executed subtask is searched.

[0020] The interrupt risk score is calculated according to the priority of the temporary task, the emergency coefficient and the nearest safe interrupt anchor point.

[0021] When the interrupt risk score is lower than a preset safe interrupt threshold, an immediate interrupt instruction is generated to control the mechanical arm to interrupt after the completion of the currently executed subtask.

[0022] When the interrupt risk score is higher than or equal to the preset safe interrupt threshold, a delayed interrupt instruction is generated to control the mechanical arm to interrupt after reaching the next safe interrupt anchor point.

[0023] The task re-planning module comprises a state comparison unit, a task evaluation unit and a local re-planning unit, and comprises:

[0024] The state comparison unit is configured to obtain the joint angle, the end pose and the environment object position of the current mechanical arm, and compare and analyze the state data of the mechanical arm at the interrupt moment to generate state difference data.

[0025] The task evaluation unit is configured to perform an executability judgment on each subtask in the to-be-executed task sequence according to the state difference data, and identify an executable subtask, a parameter modification required subtask and an invalid subtask.

[0026] The local re-planning unit performs a local re-planning operation on the to-be-executed task sequence based on a task consistency index to generate a re-planned task sequence.

[0027] The task evaluation unit is configured with a subtask judgment logic, and the subtask judgment logic comprises:

[0028] The target position and the execution prerequisite condition of each subtask in the to-be-executed task sequence are obtained.

[0029] matching the environmental object positions in the state difference data with target positions of each subtask, and calculating a target position change amount of the subtask;

[0030] When the target position change amount of the subtask is less than a preset position tolerance threshold, and the execution prerequisite meets the execution requirement, the subtask is marked as an executable subtask.

[0031] When the target position change amount of the subtask is greater than or equal to the preset position tolerance threshold, but the execution prerequisite meets the execution requirement, the subtask is marked as a parameter correction required subtask, and the target position change amount is taken as a correction parameter.

[0032] When the execution prerequisite of the subtask no longer meets the execution requirement, the subtask is marked as an invalid subtask.

[0033] The local re-planning unit is configured with re-planning logic, and the re-planning logic includes:

[0034] The execution order of all executable subtasks is kept unchanged, and structural consistency is maintained.

[0035] For the parameter correction required subtask, the target position and target attitude are updated according to the correction parameter, and spatial consistency is maintained.

[0036] All invalid subtasks are identified, and transition subtasks are generated through a large language model, wherein the transition subtasks are used to link executable subtasks before and after the invalid subtasks, and time consistency is maintained.

[0037] An optimization objective function is constructed based on the task consistency index, and the weighted sum of structural consistency, spatial consistency and time consistency is maximized as the optimization objective. A task sequence optimization algorithm is used to locally adjust the to-be-executed task sequence, and a re-planned task sequence is generated.

[0038] The to-be-executed task sequence is locally adjusted through the task sequence optimization algorithm, and the re-planned task sequence is generated, including:

[0039] Initialize the re-planned sequence, add all executable subtasks in the original order to the re-planned sequence, and generate a candidate re-planned sequence.

[0040] All invalid subtasks are traversed, the robot pose and environment state before and after the invalid subtask are analyzed through visual detection, the corresponding transition subtask is obtained, and a transition subtask set is generated.

[0041] For each invalid subtask position, the transition subtask that maximizes the increment of the optimization objective function is selected from the transition subtask set and inserted.

[0042] For the parameter correction required subtask, the subtask parameters are updated by minimizing the end pose error.

[0043] The execution order and parameter configuration of the sub-tasks in the to-be-replanned sequence are adjusted through an iterative optimization algorithm, so that the optimization objective function converges to a local optimal solution, and a final replanned task sequence is output.

[0044] The control execution module comprises a verification unit and an instruction generation unit, comprising:

[0045] The verification unit is configured to perform pre-execution verification on the replanned task sequence through a visual detection algorithm, and determine whether the constraint conditions are met, wherein the constraint conditions comprise resource conflict detection, mechanical arm posture safety detection and environmental collision detection.

[0046] The instruction generation unit is configured to generate an action control instruction according to the replanned task sequence to control the mechanical arm to return to the execution state before interruption and continue execution when the replanned task sequence meets the constraint conditions, and generate a replanning trigger signal to perform re-planning through the task replanning module when the replanned task sequence does not meet the constraint conditions.

[0047] The verification unit is configured with constraint condition judgment logic, and the constraint condition judgment logic comprises:

[0048] Kinematics simulation is performed on each sub-task in the replanned task sequence to detect whether the mechanical arm execution trajectory collides with the environmental obstacles, whether the end effector, sensor and computing resources required by each sub-task are in conflict or lack of resources, and whether the joint angle and joint speed of each sub-task are within the safe operating range of the mechanical arm, and constraint conditions of the replanned task sequence are generated.

[0049] When the constraint conditions are all met, a verification pass flag is generated, and an action control instruction is generated according to the replanned task sequence;

[0050] When any constraint condition is not met, a constraint violation report and a replanning trigger signal are generated, and the constraint violation report is fed back to the task replanning module for iterative re-planning until all constraint conditions are met.

[0051] The state recording unit is configured with state snapshot generation logic, and the state snapshot generation logic comprises:

[0052] When the task interruption instruction is received, multi-source state data at the interruption time is collected through a visual sensor and a mechanical arm torque sensor, and the multi-source state data comprises:

[0053] The joint angle, joint speed, joint torque and end pose of the mechanical arm are recorded, and the mechanical arm motion state data is generated.

[0054] Record the object position, object posture and object category in the workspace, and generate environment state data;

[0055] Record the identification of the currently executed subtask, the execution progress and the sequence of tasks to be executed, and generate task execution state data;

[0056] Encapsulate the mechanical arm motion state data, environment state data and task execution state data according to the preset JSON format, and generate a mechanical arm state snapshot;

[0057] Add a timestamp and a unique identifier to the mechanical arm state snapshot, store it in a local database, and establish a mapping index with the sequence of tasks to be executed.

[0058] Compared with the prior art, the beneficial effects of the present application are:

[0059] 1、The present application sets an interruptable flag and a safe interrupt anchor point for each subtask in the dynamic task queue, and generates an immediate or delayed interrupt instruction in combination with the temporary task priority, urgency and interrupt risk score, thereby realizing quantitative evaluation and hierarchical control of task interrupt behavior, effectively avoiding risks such as end posture instability, object falling and trajectory conflict, and ensuring the execution continuity and safety of long-time sequence tasks when temporary tasks are frequently inserted.

[0060] 2、The present application converts the re-planning problem into a sequence optimization problem under constraints through state comparison, task executability evaluation and local re-planning mechanism based on task structure consistency, spatial consistency and temporal consistency, only adjusts the sequence of tasks to be executed, significantly reduces the computational complexity and improves the real-time performance compared with global re-planning, while maintaining the consistency of task semantic logic, motion path and execution time sequence. BRIEF DESCRIPTION OF DRAWINGS

[0061] Other features, objects and advantages of the present application will become more apparent through reading the following detailed description of the non-limiting embodiments made with reference to the accompanying drawings:

[0062] Figure 1 A structural schematic diagram of the embodiment of the body-equipped intelligent mechanical arm human-computer interaction system of the present application;

[0063] Figure 2 A configuration strategy flowchart of the embodiment of the body-equipped intelligent mechanical arm human-computer interaction system of the present application;

[0064] Figure 3 A flowchart of the generation of interrupt instructions in the embodiment of the present application;

[0065] Figure 4 A working schematic diagram of local re-planning in the embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0067] Please refer to Figure 1 The present application provides an embodiment: a body intelligent mechanical arm human-computer interaction system applied to a human-computer interaction scene mixed with long-time sequence tasks and temporary tasks, the long-time sequence tasks are decomposed into multiple subtasks by a preset large language model to generate an initial task sequence, and the system comprises:

[0068] A task acquisition module is configured to identify a triggering condition of a temporary task by a visual detection algorithm, determine a priority of the temporary task, and generate a priority of the temporary task;

[0069] A task queue management module is configured to input the initial task sequence into a dynamic task queue, generate a task interruption instruction according to the priority of the temporary task and a safety interruption strategy, and record mechanical arm state data and a to-be-executed task sequence at the time of interruption;

[0070] A task re-planning module is configured to, after interruption recovery, perform local re-planning on the to-be-executed task sequence based on the mechanical arm state data at the time of interruption and current mechanical arm state data, and output a re-planned task sequence;

[0071] A control execution module is configured to perform pre-execution verification on the re-planned task sequence by a visual detection algorithm, generate a mechanical arm action control instruction according to a verification result, and control the mechanical arm to execute an interactive task.

[0072] Further, the task queue management module comprises a queue maintenance unit, an interruption decision unit and a state recording unit, comprising:

[0073] The queue maintenance unit is configured to maintain an interruptable flag and a safety interruption anchor point of each subtask in the dynamic task queue, and the safety interruption anchor point represents a time node allowed to be interrupted in a subtask execution process;

[0074] The interruption decision unit is configured to, when receiving a temporary task, generate a task interruption instruction according to the priority of the temporary task and an interruptable flag of a currently executed subtask;

[0075] The state recording unit is configured to record mechanical arm state data, environment state data and a to-be-executed task sequence at the time of interruption, and generate a mechanical arm state snapshot.

[0076] Further, the task re-planning module comprises a state comparison unit, a task evaluation unit and a local re-planning unit, comprising:

[0077] The state comparison unit is configured to acquire joint angles, end poses and environmental object positions of the current mechanical arm, compare and analyze mechanical arm state data at the time of interruption, and generate state difference data.

[0078] The task evaluation unit is configured to perform executability judgment on each subtask in the to-be-executed task sequence according to the state difference data, and identify executable subtasks, parameter modification required subtasks and invalid subtasks.

[0079] The local re-planning unit performs local re-planning operation on the to-be-executed task sequence based on a task consistency index, and generates a re-planned task sequence.

[0080] Further, the control execution module comprises a verification unit and an instruction generation unit, comprising:

[0081] The verification unit is configured to perform pre-execution verification on the re-planned task sequence by a visual detection algorithm, and judge whether the constraint conditions are met, wherein the constraint conditions comprise resource conflict detection, mechanical arm posture safety detection and environmental collision detection.

[0082] The instruction generation unit is configured to generate action control instructions according to the re-planned task sequence to control the mechanical arm to return to the execution state before interruption and continue execution when the re-planned task sequence meets the constraint conditions, and generate a re-planning trigger signal to perform re-planning through the task re-planning module when the re-planned task sequence does not meet the constraint conditions.

[0083] Please refer to Figure 2 An embodiment provided by the present application: a configuration strategy of a body-integrated intelligent mechanical arm human-computer interaction system, applied to the body-integrated intelligent mechanical arm human-computer interaction system, and specific steps of the configuration strategy are as follows:

[0084] S1: A visual detection algorithm is used to identify a trigger condition of a temporary task, priority of the temporary task is determined, and priority of the temporary task is generated.

[0085] Specifically, in the human-robot interaction scene where long-time tasks and temporary tasks are mixed, temporary tasks often come from real-time changes in the field environment and the behavior of operating personnel, such as the indicating gestures of operating personnel, the newly appeared workpieces on the workbench, the turning on of the abnormal state indicator light, the falling of objects, the occupation of the channel, etc. If only relying on manual buttons, pre-set scripts or single sensor signals to trigger temporary tasks, it is not only impossible to fully reflect the intention of operating personnel, but also difficult to timely perceive the subtle changes in the working environment, which is easy to cause the response lag of temporary tasks, and even miss the opportunity of safety intervention. Therefore, in this step, the task acquisition module introduces a visual detection algorithm to continuously collect and analyze the scene images in the working space of the robot arm, automatically identify the temporary task triggering conditions that meet the pre-defined rules, and on this basis, determine the priority of the triggered temporary tasks, and generate the temporary task priority that can be directly called by the dynamic task queue and the interruption decision.

[0086] For example, at least one set of visual acquisition devices, such as industrial cameras or depth cameras, are arranged above the working area of the robot arm, which are fixedly installed or linked with the end of the robot arm, to collect the scene image sequence in real time, which includes the workbench, the tool fixture, the workpiece to be processed, the material tray and the upper limb region of the operating personnel. The visual detection algorithm first pre-processes the collected original image, including distortion correction, adaptive adjustment of brightness and contrast, and interested region cropping, etc., to improve the robustness of subsequent feature extraction and target recognition. Then, based on the pre-trained target detection network and gesture recognition network, the key objects and human-robot interaction actions in the image are identified and located, and the multi-class detection results including the workpiece category and position, the operating personnel's hand position and posture, the indicator light state, the warning sign, etc. are obtained, and are uniformly mapped to the spatial representation under the robot arm base coordinate system.

[0087] Further, in order to convert the visual detection results into semantic events that can be used to drive temporary tasks, the embodiment pre-defines a set of temporary task triggering conditions, each of which corresponds to one or more visual pattern combinations. For example: detecting that the operating personnel makes pre-defined gestures such as "waving hands", "pointing to a certain work station", "raising the thumb or making a fist" in a certain area, can be analyzed as different categories of temporary tasks such as "requesting to insert an auxiliary handling task", "requesting to check the specified work station immediately", "confirming that the current process is passed", etc.; detecting that a new workpiece appears in a certain area and the workpiece is in an unprocessed state, can trigger additional feeding or sorting type temporary tasks; detecting that the workpiece falls to the ground or deviates from the fixture working area, can trigger emergency arrangement or abnormal handling temporary tasks. The task acquisition module matches the above visual detection results with the pre-defined triggering conditions, and when a triggering condition is met, the corresponding temporary task description is generated, including the task type, the target position, the target object and the associated safety attributes, etc.

[0088] After the completion of the temporary task trigger, the embodiment also distinguishes the importance and urgency of different temporary tasks, so that the dynamic task queue can make reasonable interruption and scheduling decisions when competing for resources with long-sequencing tasks. To this end, the task acquisition module determines the priority of each triggered temporary task according to the event features detected by vision, environmental context, and current task state. The priority determination considers at least the following factors: first, the task type safety level, for example, temporary tasks involving personnel safety protection, equipment failure warning, dangerous goods falling, etc., have a higher safety level than general carrying or sorting temporary tasks; second, the relationship between spatial position and risk area, for example, temporary tasks triggered at the edge of the robot's motion range, near the personnel activity channel or around high-value equipment have a higher priority; third, time sensitivity, for example, when a workpiece is about to exceed the allowed residence time or a work station is about to block the production line rhythm, the time-sensitive demand of the corresponding temporary task is higher.

[0089] S2: input the initial task sequence into the dynamic task queue, and the dynamic task queue generates task interruption instructions according to the priority of the temporary task and the safety interruption strategy, and records the robot state data and the to-be-executed task sequence at the time of interruption.

[0090] Specifically, since long-sequencing tasks are usually decomposed into multiple sub-tasks with a front-back dependency relationship by a pre-set large language model according to user intent, if only executed in a static list form, it is difficult to flexibly insert temporary tasks during execution and respond to high-priority operation requirements in a timely manner. Therefore, the embodiment introduces a dynamic task queue mechanism in the task queue management module, takes the initial task sequence as the basic input, combines the identified temporary tasks and their priority information, dynamically adjusts the task execution order, and generates task interruption instructions under the premise of meeting the safety interruption strategy constraints. At the same time, at the moment of interruption trigger, the robot state data and the to-be-executed task sequence are structured and recorded, providing a reliable data basis for task recovery and local re-planning.

[0091] S3: After interruption recovery, based on the robot state data at the time of interruption and the current robot state data, the to-be-executed task sequence is locally re-planned, and a re-planned task sequence is output.

[0092] Specifically, the long-time sequence task undergoes multiple interruptions and resumptions during execution, and the environment state, target object position and robot arm posture may change during the interruption. If the task plan and motion path before the interruption are simply followed after the resumption, not only the end effector and target alignment deviation may be caused, but also new collision risks may be introduced. Therefore, in this step, the local re-planning module, after receiving the interruption resumption instruction, does not directly continue from the interruption point, but jointly analyzes the recorded robot arm state data at the interruption time and the currently detected robot arm state data, re-plans the to-be-executed task sequence locally under the premise of ensuring task consistency and safety constraints, generates a re-planned task sequence adapted to the current field state, and enables the long-time sequence task to resume execution.

[0093] S4: performing pre-execution verification on the re-planned task sequence by a visual detection algorithm, and generating motion control instructions of the robot arm according to the verification result to control the robot arm to execute the interactive task.

[0094] Specifically, the re-planned task sequence is regenerated after the interruption resumption in combination with the current robot arm state data and environment state data. If direct execution is performed without unified consistency and safety verification, on the one hand, resource conflicts, posture out-of-bounds or potential collision risks that are missed in the re-planning process may occur, and on the other hand, slight changes in the environment between the re-planning and the actual execution may cause alignment failure or unstable grasping and placing actions. Therefore, in this step, the embodiment introduces a pre-execution verification mechanism based on a visual detection algorithm to centrally verify the executability and safety of the re-planned task sequence in the current real-time environment, and after the verification passes, motion control instructions corresponding to the re-planned task sequence are generated to drive the robot arm to complete the interactive task in a safe and coherent manner; when the verification fails, the task re-planning module is fed back with the verification failure reason, triggering a new round of task re-planning.

[0095] Please refer to Figure 3 The flowchart of the interruption instruction generation provided by the embodiment of the application is shown in FIG. 1, and the specific steps of S2 are as follows:

[0096] S2.1: maintaining the interruptable flag and the safe interruption anchor point of each subtask in the dynamic task queue, wherein the safe interruption anchor point represents the time node allowed to be interrupted in the execution process of the subtask.

[0097] Specifically, since the long-time task is usually composed of multiple sub-tasks with a front-back dependency relationship, and the robot arm involves multiple motion forms such as grabbing, carrying, obstacle avoidance, and human-robot collaboration during execution, if the current sub-task is interrupted at any time, it is easy to cause the end effector to lose stability, the clamped object to fall off, or a safety risk of collision with the environment. Therefore, in this step, each sub-task in the initial task sequence is uniformly accessed into the dynamic task queue, and an interruptable flag and a safe interrupt anchor point description associated with the execution progress are established for each sub-task, thereby explicitly defining the "allowed interruption time" and "prohibited interruption period" in the time dimension and task structure dimension, and providing basic data support for the subsequent task interruption instruction generation of the interruption decision unit.

[0098] For example, the dynamic task queue adopts a data structure of a sequential linked list or a circular queue for maintenance, each sub-task in the queue corresponds to a task record, and the task record at least includes: a sub-task identification ID, a sub-task type, a target position and a target pose, an execution prerequisite, an estimated execution duration, a current execution progress, an interruptable flag, and a safe interrupt anchor point set, etc. fields. Among them, the interruptable flag is used to represent the interruptable state of the current sub-task in the whole execution period, which can be divided into three levels such as completely interruptable, conditionally interruptable and completely interruptable; the safe interrupt anchor point set is used to store the key time nodes allowed to be interrupted in the execution process of the sub-task, and each safe interrupt anchor point corresponds to a specific execution progress, a robot arm pose and an environmental constraint condition. By continuously maintaining the above structured information for the sub-task, the dynamic task queue can reflect the interruption safety boundary of the current system in real time during task execution.

[0099] Further, in this embodiment, the generation of the safe interrupt anchor point is designed by segmenting the motion characteristics and safety attributes of the sub-task. Specifically, for trajectory-type sub-tasks (such as moving from the initial pose to the grabbing pose, moving from the grabbing pose to the placing pose, etc.), based on the offline planning trajectory or online generated trajectory of the robot arm, the complete trajectory is divided into several stages with physical meaning, such as acceleration segment, constant speed segment, deceleration segment, and end pose stabilization segment, etc. The system evaluates the safety indicators such as joint speed, acceleration, distance between end effector and obstacle for each stage, and only sets the safe interrupt anchor point in the stage that meets the conditions of "controllable speed, sufficient spatial redundancy, and end effector not carrying easily falling objects". For state-sensitive sub-tasks (such as grabbing, plugging, docking, etc.), the safe interrupt anchor point is preferentially set after the action is completed and the end effector and the target object are in a stable contact or completely separated state, so as to avoid interruption in the contact process or high collision risk stage, thereby reducing the impact risk on the target object and the robot arm body.

[0100] S2.2: Upon receiving a temporary task, generate a task interruption instruction according to the priority of the temporary task and the interruptable flag of the currently executed subtask.

[0101] Specifically, in the present embodiment, an interruption instruction generation mechanism based on the joint decision of priority, emergency coefficient and interruptable flag is introduced. The fundamental purpose is to enable the response of temporary tasks to reflect the importance of task level while not disrupting the safety and stability of the current action of the robot, while the long-time sequential tasks are continuously executed. By explicitly introducing a safe interruption strategy in the interruption decision process, the interruption instruction is no longer a simple immediate interruption, but a variety of policy-based control instructions such as immediate interruption, delayed interruption or prohibited interruption are dynamically generated according to the interruptable attribute of the currently executed subtask, the distribution of safe interruption anchor points and the emergency of the temporary task, so as to balance between task response efficiency and running safety.

[0102] The specific steps of S2.2 are as follows:

[0103] S2.2.1: Obtain the priority and emergency coefficient of the temporary task;

[0104] S2.2.2: Identify the interruptable flag of the currently executed subtask, if the currently executed subtask is marked as non-interruptable state, search for the nearest safe interruption anchor point in the dynamic task queue from the position of the currently executed subtask;

[0105] S2.2.3: Calculate the interruption risk score according to the priority, emergency coefficient and the nearest safe interruption anchor point of the temporary task;

[0106] S2.2.4: When the interruption risk score is lower than the preset safe interruption threshold, generate an immediate interruption instruction to control the robot to interrupt after the completion of the currently executed subtask;

[0107] S2.2.5: When the interruption risk score is higher than or equal to the preset safe interruption threshold, generate a delayed interruption instruction to control the robot to interrupt after reaching the next safe interruption anchor point.

[0108] Specifically, in the human-robot interaction scene where long-term tasks and temporary tasks are mixed, the embodied intelligent robot arm needs to continuously and stably execute the initial task sequence of long-term planning, and also needs to respond to the instructions issued by the user or the temporary tasks suddenly occurring in the environment in a timely manner. If the current subtask is simply interrupted immediately after receiving the temporary task, it is easy to cause the end effector of the robot arm to be in an unsafe pose, the clamped object to fall off, or interference with the surrounding environment; but if the temporary task is delayed, it will cause the response of high-priority temporary tasks to be delayed, and the human-robot interaction experience of the system will be reduced. Therefore, in this step, the interrupt decision unit comprehensively considers the priority and emergency coefficient of the temporary task, the interruptable flag and the safe interrupt anchor point information of the current execution subtask in the dynamic task queue, quantitatively evaluates the interrupt risk according to the preset safety strategy, and generates an immediate interrupt instruction or a delayed interrupt instruction, so as to dynamically balance between task responsiveness and execution safety.

[0109] For example, the interrupt decision unit first receives the priority and emergency coefficient of the temporary task from the task acquisition module, wherein the priority is used to represent the importance of the temporary task in the overall task sequence, and the emergency coefficient is used to represent the sensitivity of the temporary task to the response time delay. Both the priority and the emergency coefficient can be automatically generated by a rule base or a large language model combined with user intent analysis results. Subsequently, the interrupt decision unit reads the interruptable flag of the current execution subtask and its corresponding safe interrupt anchor point information from the queue maintenance unit: if the interruptable flag of the current subtask is in the interruptable state, the safe interrupt anchor point near the current execution progress is taken as the candidate interrupt point; if the current subtask flag is in the non-interruptable state, the nearest safe interrupt anchor point in the dynamic task queue to the current execution position is searched, and the time distance or trajectory distance on the time axis from the current time to the anchor point is calculated as a quantitative index of the influence of the interrupt behavior on the continuity of the current subtask.

[0110] Further, in the present embodiment, the interrupt decision unit constructs an interrupt risk score function R based on the temporary task priority P, the urgency coefficient E, the time distance T of the nearest safe interrupt anchor point, and the interrupt cost coefficient C corresponding to the interruptable flag of the current executing subtask, and maps the above factors into a risk score value in the interval [0, 1] through, for example, a weighted combination or a normalized linear model. The greater the time distance T and the interrupt cost coefficient C, the more the current subtask deviates from the safe interrupt anchor point or the less suitable the current subtask is for interruption, and the greater the R. The higher the priority P and the urgency coefficient E, the closer the system allows the interruption to the current time, and the smaller the R. By introducing the priority and the urgency coefficient, the interrupt decision unit can distinguish between temporary tasks of “high urgency and high priority” (such as safety protection instructions and direct human-computer interaction instructions) and temporary tasks of “low urgency and ordinary priority” (such as non-critical carrying requests), and adaptively adjust the interrupt behavior in different scenarios.

[0111] Further, the safety strategy of the present embodiment predefines an interrupt risk threshold R_th, and compares the interrupt risk score R with the threshold: when R is lower than the pre-set safe interrupt threshold R_th, it indicates that the impact on system safety and task continuity after interrupting immediately after the completion of the current action is acceptable, and the interrupt decision unit generates an immediate interrupt instruction, instructing the control execution module to stop executing the subsequent trajectory after the end of the current micro-action or the current sub-action atomic unit, and triggering the state recording unit to record the mechanical arm state data and the environment state data at the interrupt time; when R is higher than the pre-set safe interrupt threshold R_th, it indicates that there is a high risk of interrupting near the current time, and the interrupt decision unit generates a delayed interrupt instruction, instructing the system to continue executing along the current subtask trajectory, and triggering the interrupt when the next safe interrupt anchor point or the nearest risk acceptable anchor point is reached, thereby reserving the nearest safe insertion window for the temporary task without disrupting the critical phase of the current subtask.

[0112] Further, in order to reduce the interrupt risk of the embodied intelligent robot arm in complex operations, the present embodiment sets a forced constraint rule in the safety strategy for specific types of subtasks: for subtasks involving grasping, plugging, assembling, collision-sensitive operations, or high-speed motion, even if the interruptable flag is “conditionally interruptable”, the interrupt decision unit still forces the interrupt to be performed only at the pre-labeled safe interrupt anchor point, prohibiting interrupting in the contact establishment phase, high acceleration segment, or unstable clamping phase. When it is detected that the current subtask belongs to the above high-risk category, the system will automatically increase the interrupt risk score R or raise the safe interrupt threshold R_th corresponding to this type of subtask, so that the interrupt decision logic tends to choose delayed interrupt or interrupt after waiting for the specified safe interrupt anchor point, thereby avoiding abnormal stress on the robot arm, unintended collision between the end effector and the environment, or damage to the operated object caused by too aggressive interrupt decision.

[0113] Specifically, when receiving a temporary task, the interrupt decision unit no longer simply interrupts according to the first-come-first-served or fixed rule in a coarse-grained manner, but based on the coupling relationship between the temporary task priority, the emergency coefficient, and the current execution sub-task interruptable flag and the safety interrupt anchor point, combined with the preset safety strategy, the interrupt risk is quantitatively evaluated and processed, and then the task interrupt instruction meeting the safety constraint and the real-time requirement of human-computer interaction is generated. The interrupt decision mechanism based on risk scoring and safety strategy makes the embodied intelligent robot arm in the scene of long-time sequence task and temporary task interlacing not only guarantee the execution continuity of complex tasks, but also make more flexible and reliable response to user's temporary demand and environmental emergencies.

[0114] S2.3: Record the mechanical arm state data, environment state data and to-be-executed task sequence at the interrupt time, and generate a mechanical arm state snapshot.

[0115] The specific steps of S2.3 are as follows:

[0116] S2.3.1: When receiving the task interrupt instruction, the multi-source state data at the interrupt time is collected through the visual sensor and the mechanical arm torque sensor, and the multi-source state data includes:

[0117] Record the joint angle, joint speed, joint torque and end pose of the mechanical arm to generate the mechanical arm motion state data; record the object position, object pose and object category in the workspace to generate the environment state data; record the identification, execution progress and to-be-executed task sequence of the current execution sub-task to generate the task execution state data;

[0118] S2.3.2: The mechanical arm motion state data, environment state data and task execution state data are packaged according to the preset JSON format to generate a mechanical arm state snapshot;

[0119] S2.3.3: Add a timestamp and a unique identifier to the mechanical arm state snapshot, store it to a local database, and establish a mapping index with the to-be-executed task sequence.

[0120] Specifically, during the execution of long-time tasks, the motion state of the body-embodied intelligent robot, the spatial layout of the environmental objects in the workspace, and the sequence of the tasks to be executed in the dynamic task queue are all in a state of continuous change. If only the control instructions are simply paused at the time of interruption without complete recording of the above multi-source states, it is difficult to accurately restore the system context at the time of interruption in the subsequent execution and task re-planning stage, which may lead to the inconsistency between the execution trajectory after recovery and the original planning, the destruction of the task conditions before and after the task, or the neglect of environmental changes. Therefore, the state recording unit synchronously collects the robot state data and environmental state data at the time of task interruption, and combines the current sequence of tasks to be executed in the dynamic task queue to uniformly encapsulate the three types of information as a structured state snapshot, providing a unified data benchmark for subsequent state comparison, executability judgment, and local re-planning.

[0121] For example, after the interruption decision unit generates the task interruption instruction and the control execution module completes the current micro-motion or the action at the current safety interruption anchor point, the state recording unit immediately triggers the interruption state collection process. First, the state recording unit simultaneously collects multi-source state data at the time of interruption through the visual detection module and the robot sensor, wherein the visual detection module is responsible for sensing the environmental object layout information in the workspace, and the robot sensor is responsible for collecting the kinematics and dynamics state information of the robot body. By synchronously collecting visual and sensor data under a unified time reference, the alignment of different types of data in the time dimension in the structured state snapshot is ensured, thereby avoiding the state deviation introduced by inconsistent collection times.

[0122] Further, for the robot body, the state recording unit reads the current joint angle, joint angular velocity, and joint torque parameters of each joint from the joint encoder, speed sensor, and torque sensor, and combines the forward kinematics to obtain the pose information of the end effector, including the position coordinates and attitude Euler angle (or quaternion) representation of the end in the work coordinate system, thereby generating the robot motion state data. In the robot motion state data, the joint angles are uniformly converted into a preset standard unit and joint number sequence, and the end pose adopts the same coordinate system and attitude representation as the task planning module, so as to ensure that the state snapshot can be directly reused by the subsequent task re-planning module and control execution module. For the robot with joint overload protection or collision detection function, the current joint current, contact force estimation value, or collision detection flag can also be recorded for subsequent analysis of whether the interruption occurs in an abnormal stress working condition.

[0123] Further, for the environment state, the state recording unit performs target detection and pose estimation on the image or point cloud data collected at the interruption time through a visual detection algorithm, identifies the object position, object pose and object category information of each target object and obstacle in the workspace, and generates environment state data. In the environment state data, the positions and poses of each object are also represented in a unified coordinate system, and the object category can be encoded using a preset semantic label or category number, so that the subsequent task evaluation unit can judge the executability of the corresponding subtask according to whether the target object has been displaced, whether it has been removed, etc. At the same time, in order to improve the expression ability of the state snapshot for complex scenes, the environment state data can also include auxiliary information such as environmental lighting conditions, camera pose, workbench reference markers, etc., which can be used to assist in restoring the environmental geometric relationship when the interruption time interval is long.

[0124] In addition, for the task execution state, the state recording unit reads the current execution subtask identifier, the current execution progress (such as the completed action stage, the trajectory execution ratio) in the dynamic task queue at the interruption time, and the task sequence still in the execution state, and arranges them into task execution state data. For each to-be-executed subtask, the state recording unit preferably records its target position, target pose, execution prerequisite condition, interruptable flag and associated safe interruption anchor point information, so as to ensure that the task re-planning module can perform fine-grained rearrangement and correction in combination with the task semantics and constraint conditions before interruption during subsequent local re-planning. By including the complete state of the dynamic task queue at the interruption time into the structured state snapshot, time freezing at the task level can be achieved, and the inconsistency problem caused by recording only the current subtask and ignoring the subsequent task chain during recovery can be avoided.

[0125] Referring to Figure 4 The specific steps of the local re-planning work schematic provided by the embodiment of the present application are as follows:

[0126] S3.1: Obtain the joint angle, end pose and environment object position of the current robot arm, and compare and analyze the robot arm state data at the interruption time to generate state difference data.

[0127] Specifically, since the embodied intelligent robot arm resumes to execute the interrupted task after completing the temporary task, the joint posture of the robot arm body, the spatial pose of the end effector, and the layout of each target object and obstacle in the work space may have changed. If the planned trajectory and task parameters before interruption are directly followed without checking the consistency of the current state and the state at the time of interruption, it is easy to cause the robot arm to deviate from the trajectory, fail to grasp, or collide with newly added obstacles during the recovery process. Therefore, in this step, the state comparison unit obtains the joint angle of the robot arm, the end pose, and the position of the environmental object at the current time based on visual detection, and compares and analyzes them one by one with the robot arm state data recorded in the structured state snapshot at the time of interruption, thereby generating state difference data that can quantitatively describe the difference from before interruption to the current time, providing input basis for subsequent executability judgment and local re-planning.

[0128] In one embodiment, the state comparison unit first collects the current work space image or point cloud data through the camera installed near the base or the end of the robot arm, and uses the visual detection module to perform object detection and pose estimation on the data to obtain the current environmental object position, including the position coordinates and attitude information of each target object and obstacle in the unified world coordinate system or the robot arm base coordinate system. At the same time, the state comparison unit communicates with the robot arm controller to read the current joint angle output by each joint encoder in real time, and obtains the current pose of the end effector based on forward kinematics solution. To ensure the consistency of the comparison and analysis, the current joint angle, end pose, and environmental object position are converted to the same coordinate system, attitude representation form, and data precision as the structured state snapshot at the time of interruption in this embodiment.

[0129] Further, for the robot arm body state, the state comparison unit differentially calculates the current joint angle vector and the joint angle vector recorded in the structured state snapshot at the time of interruption to obtain the joint angle difference value, and quantitatively represents the degree of deviation of the robot arm end in the spatial position based on the displacement difference and attitude difference (such as position difference vector and attitude difference quaternion) of the end pose. Preferably, an allowable deviation threshold can be set for each joint and an end pose allowable deviation threshold, and when a joint angle difference value or an end pose deviation exceeds the corresponding threshold, the joint or end state is marked as significantly deviated in the state difference data, otherwise it is marked as acceptable deviation. Through the above processing, the complex joint and pose difference can be abstracted into a difference label with engineering significance while ensuring numerical accuracy, facilitating quick decision-making by subsequent modules.

[0130] Further, for the environmental object state, the state comparison unit establishes data association between the current environmental object set and the object set in the interrupted time environmental state data based on object identification information, category label and spatial proximity relationship. For the object pair successfully establishing association, the position difference vector and the attitude difference are calculated and compared with the preset displacement threshold and rotation threshold: when the object displacement and attitude change are within the threshold range, the object is marked as not significantly moved; when the object displacement or attitude change exceeds the threshold, the object is marked as moved, and the specific displacement and rotation amount are recorded in the state difference data. For the object that only exists in the interrupted time snapshot and is missing in the current detection result, it is marked as removed; for the object that only exists in the current detection result and does not exist in the interrupted time snapshot, it is marked as a new object, and its position, size and category information are recorded. Through the above association and difference process, the change of the object layout in the workspace before and after the interruption can be intuitively reflected.

[0131] S3.2: According to the state difference data, the executability of each subtask in the to-be-executed task sequence is judged, and executable subtasks, parameter modification subtasks and invalid subtasks are identified.

[0132] The specific steps of S3.2 are as follows:

[0133] S3.2.1: Obtain the target position and execution prerequisite condition of each subtask in the to-be-executed task sequence;

[0134] S3.2.2: Match and analyze the environmental object position in the state difference data with the target position of each subtask, and calculate the target position change amount of the subtask;

[0135] S3.2.3: When the target position change amount of the subtask is less than the preset position tolerance threshold, and the execution prerequisite condition meets the execution requirement, the subtask is marked as an executable subtask;

[0136] S3.2.4: When the target position change amount of the subtask is greater than or equal to the preset position tolerance threshold, but the execution prerequisite condition meets the execution requirement, the subtask is marked as a parameter correction subtask, and the target position change amount is taken as the correction parameter;

[0137] S3.2.5: When the execution prerequisite condition of the subtask is no longer met, the subtask is marked as an invalid subtask.

[0138] Specifically, due to the actual running state of the embodied intelligent robot arm in the interruption recovery stage may be different from the state at the interruption moment, for example, the joint angle of the robot arm drifts, the end pose deviates, the target object position moves or part of the objects are removed / added, if the recorded task parameters before interruption are directly executed to execute each sub-task in the to-be-executed task sequence, it may cause safety risks such as target deviation, insertion alignment failure or collision between motion trajectory and environment. Therefore, in this step, the executability judgment unit can perform condition checking and geometric feasibility analysis on each sub-task in the to-be-executed task sequence based on the state difference data output by the state comparison unit, and combine the execution prerequisite conditions and safety constraints of the sub-task to divide each sub-task into an executable sub-task that can be directly executed, a parameter modification sub-task that needs to adjust the target pose / path parameters, and a failed sub-task that cannot continue to execute due to the complete destruction of the prerequisite conditions, thereby realizing automatic hierarchical management of the task sequence after interruption.

[0139] In one embodiment, the executability judgment unit first reads the to-be-executed task sequence and its associated sub-task information from the structured state snapshot, each sub-task at least includes: sub-task type (such as “approach”, “grab”, “move”, “place”, “insert” and the like), target object identifier, target pose, predetermined path or safety workspace constraint, execution prerequisite condition and interruptable flag, etc. Then, the above sub-task information is associated with the state difference data, which includes joint angle deviation, end pose deviation, and displacement, attitude change, and added / removed flags of the target object and obstacles, etc. By matching the difference data with the corresponding sub-task according to the target object identifier and the task stage, a sub-task difference description containing “current environment / attitude change” information is constructed for each sub-task.

[0140] Further, the executability judgment unit in this embodiment preferably adopts a hierarchical threshold judgment strategy to perform prerequisite condition consistency check and geometric feasibility check for each sub-task. In the prerequisite condition consistency check stage, according to the execution prerequisite conditions recorded by the sub-task (such as “target object exists and is not blocked by other objects”, “target object is in the designated area of the workbench”, “a certain upstream sub-task has been completed”, etc.), combined with the object existence flag, object position label and task completion flag in the state difference data, it is judged whether the logical prerequisite of the sub-task still holds; if it is detected that the target object has been removed, a new obstacle completely blocks the target object or a previous key sub-task is judged as failed, etc., then directly mark the sub-task as a failed sub-task.

[0141] Further, in the geometric feasibility checking stage, for the sub-tasks for which the premise conditions are still established, the executability judgment unit constructs a geometric executability index based on the end position deviation, target object position deviation, and environmental obstacle change information. For example, for motion sub-tasks such as "approach", "move", "leave", the executability judgment unit compares the target position and the spatial relationship of the obstacles in the current environment, combines the kinematic constraints and safety distance constraints of the robot arm, and evaluates whether there is still a collision-free trajectory; for fine operation sub-tasks such as "grab", "place", "plug", the focus is on checking whether the offset between the current position of the target object and the original target position is within the preset compensable threshold, and whether the robot arm end can still meet the normal contact, clamping space and other operation constraints under the adjusted pose. If no feasible solution is found in the trajectory planning or constraint verification, the sub-task is determined as a failed sub-task; if there is a feasible solution, but the original target position, grasping pose or intermediate path needs to be adjusted, the sub-task is marked as a parameter modification sub-task.

[0142] Specifically, after obtaining the state difference data, the embodiment performs fine-grained executability judgment and classification management on each sub-task in the to-be-executed task sequence, which not only automatically filters out executable sub-tasks that can be directly recovered and executed, but also identifies parameter modification sub-tasks that need to be locally re-planned or parameter updated, and timely eliminates failed sub-tasks whose logical premise has been destroyed or which are no longer geometrically feasible, thereby providing clear task-level input for subsequent task re-planning and execution sequence reconstruction, and ensuring the safety, continuity and efficiency of the embodied intelligent robot arm in the interruption recovery scenario.

[0143] S3.3: Based on the task consistency index, performing a local re-planning operation on the to-be-executed task sequence to generate a re-planned task sequence.

[0144] The specific steps of S3.3 are as follows:

[0145] S3.3.1: Keep the execution order of all executable sub-tasks unchanged to maintain structural consistency;

[0146] S3.3.2: For parameter modification sub-tasks, update the target position and target pose according to the modified parameters to maintain spatial consistency;

[0147] S3.3.3: Identify all failed sub-tasks, and generate transition sub-tasks through a large language model, wherein the transition sub-tasks are used to link executable sub-tasks before and after the failed sub-tasks to maintain temporal consistency;

[0148] S3.3.4: Construct an optimization objective function based on the task consistency index, maximize the weighted sum of structural consistency, spatial consistency, and temporal consistency as the optimization objective, and perform local adjustment on the to-be-executed task sequence through a task sequence optimization algorithm to generate a re-planned task sequence.

[0149] Specifically, due to the shift in the execution context of the original initial task sequence after temporary task execution and environmental changes, if only the remaining subtasks are executed in the original order after removing the invalid subtasks, it may lead to a broken task semantic chain, a destroyed dependency relationship, and even a repeated operation or a missed key step. Therefore, in this step, the task re-planning module introduces a task consistency index based on the aforementioned state difference data and the executability judgment result, and performs local re-planning operations on the to-be-executed task sequence: under the premise of ensuring that the overall task intention remains unchanged and the key constraint conditions are met, the sequence of the local subtask chain is adjusted, the parameters are replaced, and necessary compensation tasks are inserted, thereby generating a re-planned task sequence that is still semantically consistent, reasonably timed, and motionally feasible under the current environment and robot state.

[0150] In one embodiment, the task consistency index is used to quantify the overall consistency of the re-planned task sequence after interruption recovery with the target task before interruption, which at least includes: semantic consistency index, dependency relationship consistency index, timing continuity index, and motion feasibility index. The semantic consistency index is used to measure the degree of agreement between the re-planned subtask set and the original task intention in terms of target objects, operation types, and result effects, such as whether it still completes the task of "moving a specified object from region A to region B" or "completing a certain assembly step"; the dependency relationship consistency index is used to constrain the causal order between subtasks, such as whether the "approach → grasp → move → place" link is maintained to avoid out-of-order execution or missing critical actions; the timing continuity index is used to evaluate the rationality of the connection between the re-planned subtasks on the time axis, avoiding meaningless back-and-forth movement and frequent switching between long distances; and the motion feasibility index evaluates whether the re-planned subtask set can be generated as a whole without collision trajectory based on the current environmental obstacle layout and the kinematic constraints of the robot. The above indexes can be realized by combining the rule base and the task semantic graph generated by the large language model, mapping the natural language task description into a computable structured constraint.

[0151] Further, after receiving the classification results of the executable subtasks, the parameter modification subtasks and the invalid subtasks output in step S3.2, the task re-planning module in this embodiment first removes all the nodes marked as invalid subtasks in the sequence of tasks to be executed, and sorts the parameter update requirements (such as new target pose, corrected grasping pose, re-planned approach path constraint, etc.) of the parameter modification subtasks into a parameter adjustment list. Then, taking the subtask that was completed most recently before the interruption as an anchor point, the re-planning module uses it as the starting point of re-planning, and performs local rearrangement and parameter injection on the subsequent subtask sequence without changing the task segment that has been completed and the distal task segment that has no direct coupling relationship with the segment, thereby implementing the re-planning strategy of “local modification and global maintenance” and avoiding the computational overhead and logical complexity brought by complete reconstruction of the entire task sequence.

[0152] Further, in the local re-planning process, the task re-planning module preferably uses a graph search or constraint optimization method to combine and sort the executable subtasks and the parameter modification subtasks. Specifically, the subtasks in the sequence of tasks to be executed can be modeled as nodes in a weighted directed graph, the edges between the nodes represent the dependency relationship and switchable relationship, and the edge weights can comprehensively consider the timing continuity and motion feasibility in the task consistency index, such as shorter spatial movement distance and fewer pose switches corresponding to smaller weights. On this basis, by searching for a feasible path with the smallest total weight under the premise of satisfying the dependency consistency constraint, an execution order of the subtasks that takes into account the execution efficiency and action continuity is obtained. For the parameter modification subtasks, the target pose and trajectory constraint after parameter adjustment are introduced in the graph search process, and the influence of the constraint on the motion feasibility index is dynamically evaluated to ensure that the re-planning result is geometrically feasible.

[0153] Specifically, this step performs local re-planning operation on the sequence of tasks to be executed in combination with the task consistency index without destroying the overall task target and key dependency relationship, which not only fully utilizes the value of the executable subtasks and the repairable subtasks, but also effectively eliminates the interference of the invalid subtasks on subsequent execution. The finally generated re-planned task sequence can adapt to the current state of the robot arm and environmental changes, and ensure that the embodied intelligent robot arm can still complete the remaining tasks in a logically coherent, motionally safe and semantically consistent manner after experiencing task interruption and temporary task insertion.

[0154] The specific steps of S3.3.4 are as follows:

[0155] S3.3.4.1: Initialize the re-planning sequence, add all executable subtasks in the original order to the re-planning sequence to generate a candidate re-planning sequence;

[0156] S3.3.4.2: Traverse all failed sub-tasks, analyze the robot pose and environment state before and after the failed sub-task through visual detection, obtain the corresponding transition sub-tasks, and generate a transition sub-task set;

[0157] S3.3.4.3: For each failed sub-task position, select the transition sub-task from the transition sub-task set that maximizes the increment of the optimization objective function and insert it;

[0158] S3.3.4.4: For the parameter modification sub-task, update the sub-task parameters by minimizing the end pose error;

[0159] S3.3.4.5: Adjust the execution order and parameter configuration of the sub-tasks in the to-be-replanned sequence through an iterative optimization algorithm to make the optimization objective function converge to a local optimal solution, and output the final replanned task sequence.

[0160] Specifically, after interruption recovery and temporary task insertion, the order, spatial distribution relationship, and predicted execution time window of each sub-task in the to-be-executed task sequence may change. If only simple sequence rearrangement is performed relying on heuristic rules, it is difficult to ensure task semantic integrity while taking into account motion efficiency and reasonable execution timing. Therefore, in this step, the task replanning module unifies structural consistency, spatial consistency, and temporal consistency into the same optimization framework based on the aforementioned task consistency indicators, converts the replanning problem into a sequence optimization problem with constraints by constructing an optimization objective function in the form of weighted sum, and adjusts the to-be-executed task sequence in the local search space using a task sequence optimization algorithm, thereby obtaining a globally consistent optimal or approximately optimal replanned task sequence.

[0161] In one embodiment, the task replanning module first constructs a task graph model based on the to-be-executed task sequence, represents each sub-task as a node in the graph, and uses the directed edges between the nodes to depict the precedence relationship and switchable relationship between tasks. On this basis, in combination with the executability judgment result and task semantic information, it calculates three types of consistency indicators for each candidate replanned sequence: the structural consistency indicator is used to measure the fidelity of the replanned task sequence in the task chain structure, such as whether to maintain the original key sub-task chain order such as "approach → grasp → move → place", and whether to try to retain the task grouping and stage division in the original sequence; the spatial consistency indicator is used to depict the rationality of the sub-tasks in the spatial execution path, such as whether the spatial distance between adjacent sub-task target poses is small, and whether the overall motion trajectory avoids large invalid back-and-forth motion; and the temporal consistency indicator is used to reflect the timing rationality of the task execution after replanning, such as whether the sub-task time window constraints are met, whether long delays are avoided for high-priority tasks, and whether the time overhead caused by frequent context switching is reduced.

[0162] Further, in the present embodiment, the task re-planning module constructs an optimization objective function based on the structure consistency index, the space consistency index and the time consistency index, and the optimization objective function is The calculation formula is as follows:

[0163]

[0164] Wherein, is the structure consistency score, and the proportion of subtask types reserved in the original task sequence and the re-planned sequence is calculated; is the space consistency score, and the space deviation reciprocal of each subtask target position in the re-planned sequence and the original target position is calculated; is the time consistency score, and the closeness of the expected execution time of the re-planned sequence and the original execution time is calculated; 、 、 is a weight coefficient and satisfies + + =1, which is used to represent the importance of structure consistency, space consistency and time consistency in the current application scenario. For the scene that emphasizes strict adherence to the original process flow, the weight of a can be appropriately increased to make the optimization process prefer to keep the task structure from being destroyed; for the scene that emphasizes motion efficiency and energy consumption, the weight of β can be increased to make the optimization process more inclined to shorten the spatial path and reduce redundant actions; for the scene that contains time window constraints or needs to respond quickly to high-priority tasks, the weight of γ can be increased to ensure the rationality of the overall execution timing after re-planning. In the present embodiment, a, β and γ can be adjusted adaptively according to historical execution data, or by preset configuration or experience rules.

[0165] Further, in order to convert the optimization objective function into a solvable task sequence optimization problem, the present embodiment introduces a set of constraint conditions for the candidate re-planned sequence, including but not limited to: task dependency relationship constraint (downstream subtasks cannot be executed earlier than upstream subtasks), safety constraint (mechanical arm joint limit, speed and acceleration limit and collision avoidance constraint must be met during task execution), executability constraint (only executable subtasks or parameter-modified subtasks are allowed to be used, and the latter must use updated task parameters), interruption recovery constraint (the re-planning starting point should be compatible with the end state of the last completed subtask before interruption). On this basis, all task sequences that meet the constraint conditions are regarded as a feasible solution set, and the optimization objective is to maximize the objective function J, and the optimal or approximately optimal task execution order and parameter combination are selected from the feasible solution set.

[0166] In a preferred embodiment, the task sequence optimization algorithm adopts a heuristic optimization method based on graph search, including but not limited to: A* search algorithm with heuristic cost function, heuristic depth-first / breadth-first search, or genetic algorithm with task sequence as individual encoding, simulated annealing algorithm, etc. Specifically, the task sequence to be executed can be modeled as a starting node, and all legal exchange operations, insertion operations or merging operations that meet the dependency relationship constraints are regarded as state transitions. By exchanging adjacent sub-tasks or adjusting the position of sub-tasks with modified parameters within a local window, a candidate re-planning sequence is gradually generated. During the search process, the above objective function J is used as the evaluation function to comprehensively evaluate the structural consistency, spatial consistency and temporal consistency of the candidate sequence, and the heuristic function is used to guide the search direction, so as to quickly converge to a task sequence with high consistency under limited computing resources.

[0167] Further, to limit the computing overhead, the task sequence optimization in this embodiment preferably adopts a local re-planning strategy: that is, only the local window centered on the breakpoint and covering several sub-tasks after the breakpoint is adjusted, while the remote task sequence outside the window remains unchanged. In this way of "local adjustment and global inheritance", on the one hand, the search space size can be significantly reduced, and the optimization convergence speed can be accelerated; on the other hand, it is also beneficial to maintain the macro stability of the overall task structure and improve the structural consistency index. At the same time, within the local re-planning window, for sub-tasks that need to modify parameters, the "target pose update" or "path constraint adjustment" can be introduced as an additional parameter dimension into the optimization process, so that the spatial consistency index takes into account the task order as well as the trajectory smoothness and energy consumption.

[0168] In addition, in some embodiments, the task re-planning module can also adjust the weight configuration and part of the evaluation items of the optimization objective function based on historical execution data and human-computer interaction feedback. For example, when the system detects that manual intervention to reorder or cancel part of the tasks frequently occurs after re-planning, the contribution weights of different types of sub-tasks in the structural consistency and temporal consistency evaluation can be adjusted through statistical analysis, so that the subsequent re-planning results are more in line with the user's implicit preferences. This optimization target adaptive adjustment mechanism based on task consistency indicators enables the human-computer interaction system of the embodied intelligent robot to have the ability of continuous "self-evolution" of re-planning.

[0169] Through the above design, on the basis of clearly defining the three types of task consistency indexes of structural consistency, spatial consistency and temporal consistency, a unified weighted optimization objective function is constructed, and the task sequence optimization algorithm is combined to locally adjust the to-be-executed task sequence, and finally a re-planned task sequence with better overall consistency under the current mechanical arm state and environment layout is obtained, so as to improve the execution efficiency and interactive experience after interruption and recovery under the premise of ensuring safety and semantic integrity.

[0170] The specific steps of S4 are as follows:

[0171] S4.1: performing pre-execution verification on the re-planned task sequence by a visual detection algorithm to determine whether the constraint conditions are met, the constraint conditions including resource conflict detection, mechanical arm posture safety detection and environment collision detection.

[0172] The specific steps of S4.1 are as follows:

[0173] S4.1.1: kinematically simulating each subtask in the re-planned task sequence to detect whether the mechanical arm execution trajectory collides with the environment obstacles, whether the end effector, sensor and computing resources required by each subtask are in conflict or insufficient in resources, and whether the joint angle and joint speed of each subtask are within the safe operating range of the mechanical arm, and to generate the constraint conditions of the re-planned task sequence;

[0174] S4.1.2: when the constraint conditions are all met, a verification pass flag is generated, and an action control instruction is generated according to the re-planned task sequence;

[0175] S4.1.3: when any constraint condition is not met, a constraint violation report and a re-planning trigger signal are generated, and the constraint violation report is fed back to the task re-planning module for iterative re-planning until all constraint conditions are met.

[0176] Specifically, the re-planned task sequence is regenerated based on the current state difference data after interruption recovery and temporary task insertion. Although task consistency and motion feasibility have been considered in the re-planning process, there is still a possibility that due to further changes in the environment, changes in upper system resource occupation, or deviations caused by re-planning approximation solution, some sub-tasks may have safety hazards such as resource competition, robot pose out-of-limit, or potential environmental collision when actually executed. If the re-planned task sequence is not checked for unified constraint conditions before execution and directly issued to the control execution module, it will significantly increase the risk of robot instability, clamp misuse, or interference with the environment and personnel. Therefore, in this step, the system performs pre-execution checking on the re-planned task sequence based on the latest visual detection results and the current state of the robot, and comprehensively detects whether the constraint conditions are met from the resource layer, the pose layer, and the environment layer. Only when all constraint conditions pass the check, the re-planned task sequence is issued as the final execution plan.

[0177] In one embodiment, the pre-execution checking module first acquires image or point cloud data of the current work space through the camera, identifies the positions and poses of the workbench, tooling fixture, tool, material, and obstacle, etc. environmental resources using the visual detection module, and generates the latest environmental state description. At the same time, it reads the current joint angle, joint speed, and end pose information from the robot controller as the initial pose boundary condition for checking. Subsequently, the pre-execution checking module parses the re-planned task sequence into a series of sub-task execution segments with time sequence marks, each segment including target object identification, target pose, expected occupied tool or tooling resource, and planned trajectory or key path point. Based on the above parsing results, the resource conflict detection module, the pose safety detection module, and the environmental collision detection module are called in turn to perform multi-dimensional constraint checking on the re-planned task sequence.

[0178] Further, in terms of resource conflict detection, the embodiment unifies the resource requests of each subtask in the re-planning task sequence by constructing a resource occupation time axis. Specifically, the system determines the resource set that each subtask needs to occupy during execution according to its semantic information and parameter configuration, including but not limited to: end effector type (such as two-finger gripper, three-finger gripper, suction cup), special tooling fixture, temporary storage tray, area identifier of the work area, and specific target object itself. The pre-execution verification module maps each subtask to an occupation record of "resource-time interval" under a unified time reference, and judges whether there is a situation where the same resource is simultaneously requested by multiple subtasks in the same time interval by scanning the overlapping relationship between resource occupation records in the re-planning task sequence. For example, when a target object has been marked as "picked up but not placed" by a previous subtask in a certain time period, if a subsequent subtask uses this object as a picking object or as a workpiece fixed on a tool, it will be judged as a resource conflict; for fixtures or special tools that need to be exclusively occupied, when multiple subtasks have overlapping usage intervals on the time axis, they are also marked as resource conflicts. If a resource conflict is detected, the pre-execution verification module can return the conflict subtask identifier and conflict resource list, triggering the local re-planning or task adjustment of the previous step.

[0179] Further, in terms of mechanical arm posture safety detection, the embodiment simulates and checks the posture changes of the mechanical arm during execution based on the planning trajectory corresponding to each subtask in the re-planning task sequence. Specifically, the posture safety detection module takes the current joint angle at the start of the re-planning task sequence as the starting point, time-discretely samples each trajectory according to the planning trajectory, calculates the joint angle, joint speed and joint acceleration at the sampling points, and compares them with the pre-set joint angle limit, speed / acceleration upper limit to judge whether there is an out-of-limit situation. At the same time, the system calculates the Jacobian matrix and manipulability index of the mechanical arm at each sampling point to identify dangerous postures that approach singular configurations or have low manipulability; for mechanical arms with self-collision models, the minimum distance between links and between links and the base can also be checked during simulation. If it is lower than the self-collision safety distance threshold, it is considered that the posture safety does not meet the requirements. If there is a continuous or multiple-point violation of posture safety constraints on a trajectory, the pre-execution verification module marks the subtask as "posture infeasible" and feeds back the specific violation position to the re-planning module for trajectory re-planning.

[0180] Further, in terms of environmental collision detection, the embodiment utilizes the latest environment model generated by visual detection (e.g. represented in the form of a point cloud, a voxel grid or a set of bounding boxes) to perform simulation collision detection for each trajectory in the re-planning task sequence. Specifically, the collision detection module calculates the geometric envelope of each link of the robot arm and the end effector in space at the same time discretization step, and performs distance queries with obstacles, tooling and personnel safety areas in the environment model to calculate the minimum safety distance. When the minimum safety distance of any sampling point is less than the preset collision safety distance threshold, it is determined that there is a potential collision risk at the corresponding time, and if the minimum distance is zero or the geometric shapes intersect, it is determined as a certain collision risk. For scenarios involving human bodies or human-robot collaboration areas, the system can also set a higher safety distance threshold around the personnel to achieve a more conservative safety strategy. If an environmental collision risk is detected, the pre-execution verification module will also mark the corresponding sub-task as not meeting the environmental constraints, and feed back the risk position and risk object to the upstream re-planning module.

[0181] Preferably, the embodiment can also comprehensively evaluate the resource conflict detection result, the pose safety detection result and the environmental collision detection result to form an execution pre-verification report. The pre-execution verification module can calculate the overall constraint satisfaction score for the re-planning task sequence, for example, by mapping the resource constraints, pose constraints and environmental constraints into sub-scores in the interval [0, 1] respectively, and obtaining the total score by weighted summation. When the total score is below a preset threshold or any key constraint has a serious violation, it is considered that the re-planning task sequence does not meet the execution conditions, at which time the system can automatically trigger a local re-planning process or request user confirmation or adjustment of the task through a human-machine interaction interface; when all constraint conditions are met and the score is higher than the threshold, the re-planning task sequence is marked as "safe and executable", and the verification pass mark is also stored in the task log as a basis for subsequent tracing and safety audit.

[0182] Through the above design, before the re-planning task sequence is issued for execution, the pre-execution verification mechanism based on visual detection is introduced to comprehensively check the re-planning task sequence from the three dimensions of resource conflict, robot arm pose safety and environmental collision, which can discover potential risks in advance before the control instructions really act on the robot arm, avoid safety accidents caused by resource competition, pose overrun or environmental collision, and thus significantly improve the execution safety and system robustness of the embodied intelligent robot arm in human-robot interaction scenarios.

[0183] S4.2: When the re-planning task sequence meets the constraint conditions, generate action control instructions according to the re-planning task sequence to control the robot arm to restore to the execution state before interruption and continue execution, and when the re-planning task sequence does not meet the constraint conditions, generate a re-planning trigger signal to re-plan through the task re-planning module.

[0184] Specifically, since the embodiment is directed to the human-computer interaction scene of long-time sequence task and temporary task interweaving, the current actual state of the mechanical arm has deviated from the ideal state recorded at the time of interruption after experiencing interruption, state evaluation and task re-planning. If the action instructions corresponding to the re-planned task sequence are directly issued without unified coordination, problems such as "inconsistent execution starting point", "sudden jump of action" or "partial task repetition / omission" are likely to occur, which further leads to task logic confusion or new safety risks. Therefore, based on the execution of the previous verification result, the control execution module uniformly analyzes the re-planned task sequence and the interruption state snapshot, on the one hand, when the verification passes, the action control instructions conforming to the controller interface specification and capable of smoothly connecting the states before and after the interruption are generated; on the other hand, when the verification fails, the reasons why the current constraints cannot be met are fed back to the task re-planning module in the form of re-planning trigger signal, forming a "verification-execution-re-planning" closed-loop control link.

[0185] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those of ordinary skill in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. An embodied intelligent robotic arm human-computer interaction system, applied to human-computer interaction scenarios involving both long-term and temporary tasks, characterized in that: The long-sequence task is decomposed into multiple sub-tasks using a pre-defined large language model to generate an initial task sequence. The system includes: The task acquisition module is used to identify the triggering conditions of temporary tasks through a visual detection algorithm, determine the priority of the temporary tasks, and generate the priority of the temporary tasks. The task queue management module is used to input the initial task sequence into the dynamic task queue. The dynamic task queue generates a task interruption instruction based on the priority of the temporary task and the safety interruption policy, and records the robotic arm status data and the sequence of tasks to be executed at the time of interruption. The task replanning module is used to perform local replanning on the sequence of tasks to be executed based on the robotic arm state data at the time of the interruption and the current robotic arm state data after the interruption is resumed, and output the replanned task sequence. The control execution module is used to perform pre-execution verification of the replanning task sequence through a visual detection algorithm, and generate motion control commands for the robotic arm based on the verification results, thereby controlling the robotic arm to execute interactive tasks.

2. The human-machine interaction system of the embodied intelligent robotic arm according to claim 1, characterized in that, The task queue management module includes a queue maintenance unit, an interruption decision unit, and a status recording unit, including: The queue maintenance unit is used to maintain the interruptible flag and safe interruption anchor point for each subtask in the dynamic task queue. The safe interruption anchor point represents the time node during the execution of the subtask that is allowed to be interrupted. The interruption decision unit is used to generate a task interruption instruction based on the priority of the temporary task and the interruptibility flag of the currently executing subtask when a temporary task is received. The status recording unit is used to record the robotic arm status data, environmental status data, and the sequence of tasks to be executed at the time of interruption, and to generate a snapshot of the robotic arm status.

3. The human-machine interaction system of the embodied intelligent robotic arm according to claim 2, characterized in that, The interrupt decision unit is configured with interrupt point selection logic, which includes: Get the priority and urgency coefficient of temporary tasks; Identify the interruptible flag of the currently executing subtask. If the currently executing subtask is marked as uninterruptible, search the dynamic task queue for the safe interrupt anchor point closest to the current executing subtask. Calculate the interruption risk score based on the priority, urgency coefficient, and most recent security interruption anchor point of the temporary task; When the interruption risk score is lower than the preset safety interruption threshold, an immediate interruption command is generated to control the robotic arm to interrupt after the current subtask is completed. When the interruption risk score is higher than or equal to the preset safety interruption threshold, a delayed interruption command is generated to control the robotic arm to interrupt after reaching the next safety interruption anchor point.

4. The human-machine interaction system of the embodied intelligent robotic arm according to claim 1, characterized in that, The task replanning module includes a state comparison unit, a task evaluation unit, and a local replanning unit, including: The state comparison unit is used to acquire the joint angles, end-effector poses, and positions of environmental objects of the current robotic arm, compare and analyze them with the robotic arm state data at the time of interruption, and generate state difference data. The task evaluation unit is used to determine the executability of each subtask in the sequence of tasks to be executed based on the state difference data, and to identify executable subtasks, subtasks that need to modify parameters, and failed subtasks. The local replanning unit performs a local replanning operation on the sequence of tasks to be executed based on the task consistency index, thereby generating a replanned task sequence.

5. The human-machine interaction system of the embodied intelligent robotic arm according to claim 4, characterized in that, The task evaluation unit is configured with sub-task judgment logic, which includes: Obtain the target location and execution prerequisites for each subtask in the sequence of tasks to be executed; The positions of environmental objects in the state difference data are matched and analyzed with the target positions of each sub-task to calculate the change in the target position of the sub-task. When the change in the target position of a subtask is less than the preset position tolerance threshold and the execution prerequisites are met, the subtask is marked as an executable subtask. When the change in the target position of a subtask is greater than or equal to the preset position tolerance threshold, but the execution prerequisites meet the execution requirements, the subtask is marked as a subtask that needs parameter correction, and the change in the target position is used as the correction parameter. When the preconditions for the execution of a subtask are no longer met, the subtask is marked as an invalid subtask.

6. The human-machine interaction system of the embodied intelligent robotic arm according to claim 5, characterized in that, The local replanning unit is configured with replanning logic, which includes: The execution order of all executable subtasks remains unchanged, maintaining structural consistency; For subtasks that require parameter modification, update the target position and target attitude based on the modified parameters to maintain spatial consistency; Identify all failed subtasks and generate transitional subtasks through a large language model. These transitional subtasks are used to connect executable subtasks before and after the failed subtasks to maintain time consistency. An optimization objective function is constructed based on the task consistency index. The optimization objective is to maximize the weighted sum of structural consistency, spatial consistency and temporal consistency. The task sequence optimization algorithm is used to locally adjust the sequence of tasks to be executed and generate a replanning task sequence.

7. The human-machine interaction system of the embodied intelligent robotic arm according to claim 6, characterized in that, The step of locally adjusting the sequence of tasks to be executed using a task sequence optimization algorithm to generate a replanning task sequence includes: Initialize the replanning sequence by adding all executable subtasks to the replanning sequence in their original order, and generate candidate replanning sequences. Traverse all failed subtasks, analyze the robot arm pose and environmental state before and after the failed subtasks through visual detection, obtain the corresponding transition subtasks, and generate a set of transition subtasks. For each failed subtask location, select the transition subtask from the transition subtask set that maximizes the increment of the optimization objective function and insert it; For subtasks that require parameter modification, update the subtask parameters by minimizing the end-effector pose error; By adjusting the execution order and parameter configuration of subtasks in the replanning sequence through an iterative optimization algorithm, the optimization objective function converges to a local optimum, and the final replanning task sequence is output.

8. The human-machine interaction system of the embodied intelligent robotic arm according to claim 1, characterized in that, The control execution module includes a verification unit and an instruction generation unit, comprising: The verification unit is used to perform pre-execution verification on the replanning task sequence using a visual detection algorithm to determine whether the constraints are met. The constraints include resource conflict detection, robotic arm posture safety detection, and environmental collision detection. The instruction generation unit is used to generate motion control instructions based on the replanning task sequence when the replanning task sequence meets the constraints, control the robotic arm to return to the execution state before the interruption and continue execution, and generate a replanning trigger signal when the replanning task sequence does not meet the constraints, and replanning is performed through the task replanning module.

9. The human-machine interaction system of the embodied intelligent robotic arm according to claim 8, characterized in that, The verification unit is configured with constraint condition judgment logic, which includes: Kinematic simulations are performed on each subtask in the replanning task sequence to detect whether the robotic arm's execution trajectory collides with environmental obstacles, whether there are conflicts or insufficient resources in the end effector, sensors, and computing resources required for each subtask, and whether the joint angles and joint velocities of each subtask are within the safe operating range of the robotic arm. Constraints are then generated for the replanning task sequence. When all the constraints are met, a verification pass flag is generated, and action control instructions are generated based on the replanning task sequence. When any constraint is not met, a constraint violation report and a replanning trigger signal are generated. The constraint violation report is then fed back to the task replanning module for iterative replanning until all constraints are met.

10. The human-machine interaction system of the embodied intelligent robotic arm according to claim 2, characterized in that, The state recording unit is configured with state snapshot generation logic, which includes: Upon receiving a task interruption command, multi-source state data at the moment of interruption is collected using a vision sensor and a robotic arm torque sensor. This multi-source state data includes: Record the joint angles, joint velocities, joint torques, and end-effector pose of the robotic arm to generate motion state data of the robotic arm; Record the position, orientation, and category of objects within the workspace to generate environmental status data; Record the identifier, execution progress, and sequence of tasks to be executed for the currently executing subtask, and generate task execution status data; The robotic arm motion state data, environmental state data, and task execution state data are encapsulated in a preset JSON format to generate a robotic arm state snapshot. Add a timestamp and a unique identifier to the snapshot of the robotic arm's state, store it in a local database, and establish a mapping index with the sequence of tasks to be executed.

Images