Task planning and correcting method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a task planning and correction method, device, equipment and medium, and the method comprises the steps: obtaining a task instruction and environment perception data, and generating instruction features and environment features; retrieving a reference case from an experience case library according to the instruction features and the environment features; decomposing the task instruction into a subtask sequence containing a target object, and generating a task plan based on the reference case; executing the current subtask and detecting whether the target object is missing or not; if the target object is missing, acquiring an environment object set, and determining an alternative object semantically associated with the target object; and updating the current subtask by using the replacement object and executing the correction subtask. According to the method, the task plan is generated by combining the environment perception and the reference case, and the correction sub-task is dynamically replaced and generated when the target is missing, so that the task is executed without interruption, and the robustness and the environment adaptability are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a task planning and correction method, apparatus, device, and storage medium. Background Technology

[0002] Current task planning technologies for robots and embodied intelligent agents suffer from significant limitations in environmental adaptability, particularly when faced with scenarios where the target object is missing during execution. Existing task planning methods often employ static, predefined task execution flows, lacking the dynamic perception and correction capabilities to address situations where the target object is missing midway through task execution. Once a critical operational object is not in its expected location, the system typically enters a waiting or failure state, severely impacting task continuity and completion efficiency.

[0003] In the fintech sector, smart terminals or robots are used to perform tasks based on object-oriented operations, such as invoice delivery, contract archiving, and data collection. If, during execution, the system fails to identify the required object (such as an invoice or voucher box) at a preset location, existing technologies lack a dynamic substitution mechanism. This prevents the system from quickly finding a functionally or semantically relevant substitute based on the environment, thus hindering task completion and impacting overall process efficiency and user experience.

[0004] In the healthcare sector, embodied agents have been used to automate tasks such as drug delivery and sample retrieval. When pre-assigned medicine boxes, test tube trays, or medical recycling containers are temporarily missing or moved during execution, traditional methods often cannot automatically determine if there are alternative objects or adjust the current task plan to continue execution, leading to task interruptions and affecting the stability of hospital workflows and equipment availability. Summary of the Invention

[0005] The main objective of this invention is to provide a task planning and correction method, apparatus, device, and storage medium, which aims to solve the technical problem that the prior art lacks the ability to perform semantic association matching and dynamic correction of task targets by combining environmental perception information, resulting in task execution failure when the target object is missing.

[0006] To achieve the above objectives, the present invention provides a task planning and correction method, comprising:

[0007] Acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and environmental perception data, respectively;

[0008] Based on the instruction characteristics and the environmental characteristics, retrieve reference cases from the experience case library;

[0009] decompose the task instruction into a subtask sequence containing a target object to be operated on;

[0010] generate a task plan based on the reference case and the subtask sequence;

[0011] execute a current subtask in the task plan and detect whether the target object of the current subtask is missing during execution of the current subtask;

[0012] when it is detected that the target object of the current subtask is missing, obtain objects in a current environment to form a current environment object set;

[0013] determine a substitute object associated with the target object in semantics in the current environment object set;

[0014] update the current subtask with the substitute object to generate a corrected subtask, and execute the corrected subtask to complete the task plan.

[0015] Further, to achieve the above object, the present application provides a task planning and correction device, comprising:

[0016] a task analysis module configured to obtain a task instruction and environment perception data, and generate instruction features and environment features based on the task instruction and the environment perception data, respectively;

[0017] a case retrieval module configured to retrieve a reference case from an experience case library according to the instruction features and the environment features;

[0018] a task decomposition module configured to decompose the task instruction into a subtask sequence containing a target object to be operated on;

[0019] a plan generation module configured to generate a task plan based on the reference case and the subtask sequence;

[0020] a target detection module configured to execute a current subtask in the task plan and detect whether the target object of the current subtask is missing during execution of the current subtask;

[0021] an environment construction module configured to, when it is detected that the target object of the current subtask is missing, obtain objects in a current environment to form a current environment object set;

[0022] a substitute reasoning module configured to determine a substitute object associated with the target object in semantics in the current environment object set;

[0023] a task correction module configured to update the current subtask with the substitute object to generate a corrected subtask, and execute the corrected subtask to complete the task plan.

[0024] Further, to achieve the above object, the present application also provides a computer device, which comprises a memory, a processor and a task planning and correction program stored in the memory and executable on the processor, and the task planning and correction program realizes the steps of the task planning and correction method when executed by the processor.

[0025] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a task planning and correction program, and the task planning and correction program realizes the steps of the task planning and correction method when executed by a processor.

[0026] Beneficial effects: The present application relates to the field of artificial intelligence technology, and can be applied to business scenarios such as financial technology and medical health, and discloses a task planning and correction method, device, equipment and medium, which comprises the following steps: obtaining a task instruction and environment perception data, generating instruction features and environment features based on the task instruction and the environment perception data respectively; retrieving reference cases from an experience case library according to the instruction features and the environment features; decomposing the task instruction into a sub-task sequence containing a target object to be operated; generating a task plan based on the reference cases and the sub-task sequence; executing a current sub-task in the task plan, and detecting whether the target object of the current sub-task is missing during execution of the current sub-task; when it is detected that the target object is missing, obtaining objects in a current environment to form a current environment object set; determining a replacement object associated with the target object in the current environment object set; updating the current sub-task by using the replacement object to generate a corrected sub-task, and executing the corrected sub-task to complete the task plan. The present application generates a task plan by combining environment perception data and reference cases, and dynamically obtains a replacement object based on a current environment to construct a corrected sub-task when the target object is missing, so as to realize local adaptive correction and continuous promotion of the task, avoid the task interruption problem caused by target loss in the traditional method, and improve the robustness of task execution and the adaptability of an intelligent agent in an actual environment. BRIEF DESCRIPTION OF DRAWINGS

[0027] The present application will be further described below in combination with the drawings and embodiments, and the drawings are as follows:

[0028] Figure 1 An application environment diagram of the task planning and correction method in an embodiment of the present application;

[0029] Figure 2 A flowchart of the task planning and correction method in an embodiment of the present application;

[0030] Figure 3 A functional module diagram of a preferred embodiment of the task planning and correction device of the present application;

[0031] Figure 4A structural schematic diagram of a computer device in an embodiment of the present application;

[0032] Figure 5 Another structural schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are merely illustrative of the present application and do not limit the present application.

[0034] The task planning and correction method provided by the embodiments of the present application can be applied in an application environment such as Figure 1 , in which a user end communicates with a service end through a network. The service end can obtain task instructions and environment perception data through the user end, generate instruction features and environment features based on the task instructions and the environment perception data respectively, retrieve reference cases from an experience case library according to the instruction features and the environment features, decompose the task instructions into a subtask sequence containing a target object to be operated, generate a task plan based on the reference cases and the subtask sequence, execute a current subtask in the task plan, and detect whether the target object of the current subtask is missing during execution of the current subtask. When it is detected that the target object is missing, the service end obtains objects in a current environment to form a current environment object set, determines a replacement object that is semantically associated with the target object in the current environment object set, updates the current subtask by using the replacement object to generate a corrected subtask, and executes the corrected subtask to complete the task plan. The present application generates a task plan by combining environment perception data and reference cases, and dynamically obtains a replacement object based on a current environment to construct a corrected subtask when a target object is missing, so as to realize local adaptive correction and continuous promotion of a task, avoid task interruption caused by target loss in a traditional method, and improve robustness of task execution and adaptability of an intelligent agent in an actual environment. The user end can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices. The service end can be implemented by an independent server or a server cluster composed of multiple servers. The present application will be described in detail through specific embodiments.

[0035] Please refer to Figure 2 , Figure 2 A flowchart of an embodiment of the task planning and correction method provided by the present application. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that herein.

[0036] As shown in Figure 2 , the task planning and correction method provided by the present application includes the following steps:

[0037] S10, acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and environmental perception data respectively;

[0038] In this embodiment, receiving task instructions typically involves the unified parsing of multimodal inputs, including text instructions in natural language form, voice input, or task definitions in encoded formats. Task instructions represent externally assigned operational goals and are context-dependent and goal-oriented. The process of parsing task instructions can employ language models, intent recognition modules, or structured rule systems to map the raw text into actionable instruction features. Instruction features generally include verb representations (e.g., move, place, open), object recognition (e.g., potato, trash can), and semantic correspondences with the environment, used to describe the task's execution logic and target content.

[0039] Environmental perception data is typically acquired as raw data through sensor devices (such as cameras, radar, and depth sensors). Data types can include image frames, 3D point clouds, infrared thermal images, or spatial mapping results. To extract useful information from this perception data, visual encoders, perceptual models, or semantic segmentation algorithms are used to map the environment into a structured representation, i.e., environmental features. Environmental features contain information about the category, location, spatial relationships, and state of visible objects at the current moment, supporting task judgment and object selection. The feature generation process must ensure temporal synchronization and spatial consistency to avoid erroneous inferences due to data offsets.

[0040] By structuring and managing the features generated from task instructions and environmental perception data respectively, task-environment matching can be performed based on a unified representation space. Collaborative use between the two feature domains can be achieved through vector embedding, attention mechanisms, or cross-modal alignment, supporting subsequent operations such as task retrieval, case matching, or plan generation.

[0041] One implementation involves the system receiving task input in natural language form from an external interface, extracting embedding vectors through a language parser such as BERT or T5, and then mapping them to a set of task labels, represented as structured tuples. For example, the instruction "put the knife in the trash can" can be parsed as the action "Put", the target object "Knife", and the target location "TrashCan".

[0042] When acquiring environmental perception data synchronously, a depth camera mounted on the agent is used to capture image frames. Visual-language alignment models such as CLIP are used to perform semantic recognition on the images, extracting visible objects and their spatial relationships, and generating vector representations. To improve timeliness, a parallel pipeline can be used to perform language processing and visual recognition simultaneously, completing dual-channel feature generation in a very short time after the task arrives.

[0043] After feature generation, a cross-modal attention mechanism is used to jointly model linguistic and visual features to capture the semantic and structural matching relationship between instructions and the current environment, preparing input support for subsequent task planning.

[0044] Example Description: In a healthcare scenario, after receiving the instruction to "put the medicine bottle in the recycling bin," the intelligent robot uses a voice analysis module to extract "medicine bottle" as the operation object and "recycling bin" as the target container. In the environment, the camera identifies similar objects such as "medicine bottle" and "medical waste bin," confirms them as semantically equivalent targets after embedding and matching, and completes the task, thus enhancing the robot's assistance capabilities in healthcare scenarios.

[0045] In the document archiving scenario in the financial field, when the instruction "classify expense reports into the document cabinet" is received, the system parses "expense report" as a document object and "document cabinet" as the target storage location. Through the environment recognition module, elements such as "paper documents" and "storage boxes" are extracted. Based on feature matching, the system completes the fusion processing of instruction features and environmental status to ensure accurate task implementation.

[0046] This embodiment generates structured feature representations for both task instructions and environmental perception data, preserving the integrity of the task objective while accurately capturing the semantic structure of the environmental state. This allows subsequent task planning to be based on a realistic understanding of the current scene. The separate generation and cross-modal fusion of features effectively avoids task failures caused by the disconnect between instructions and the environment in traditional methods, enhancing execution stability and the system's real-time perception capabilities.

[0047] S20, based on the instruction characteristics and the environmental characteristics, retrieve reference cases from the experience case library;

[0048] In this embodiment, retrieving historical cases with reference value during task execution is a key means to improve the efficiency and robustness of task planning. The experience case library is used to store the mapping relationship between the execution trajectory of historical tasks, environmental states, and instruction expressions. Its construction is based on reusable operation templates, environmental adaptation records, and object interaction instances accumulated by the embodied intelligent agent in long-term interactions. Each case can be structured into a triple form, including "instruction characteristics," "environmental characteristics," and "task planning result" or "operation success record."

[0049] Once the system generates instruction and environment features, it uses these two sets of features as joint retrieval conditions to match the most similar reference case in the multimodal embedding space. The retrieval process can employ semantic similarity calculation (such as cosine similarity, Euclidean distance), hash indexing, or attention-based matching networks to ensure that the similarity extends beyond text or object labels, also encompassing transferability in terms of scene structure and target combinations.

[0050] Instruction features typically focus on the target action and operation path, while environmental features encompass the semantic labels, spatial location, and accessibility information of operable objects in the current scene. By jointly modeling these two types of features, the retrieval system can simultaneously consider linguistic intent and environmental context, thereby finding operation examples that perform well in similar situations.

[0051] To enhance the system's adaptability to fuzzy expressions and semantic diversity, case entries in the experience case library can be pre-encoded offline and indexed using a unified multimodal embedding network, improving the speed and accuracy of online retrieval. Simultaneously, the system can dynamically update the case library for newly completed tasks, continuously optimizing its coverage.

[0052] In one implementation, after receiving the instruction feature "put the knife into the trash can" and the environmental feature containing two objects, "Knife" and "GarbageCan", the system encodes this instruction-environment pair into a vector representation. This representation is used as a query vector input to a vector-based retrieval case library. An ANN (Approximate Nearest Neighbor) algorithm, such as FAISS, is used for nearest neighbor search to quickly retrieve historical case samples of "Put into" type operations performed in similar environments.

[0053] Case matching prioritizes operational experience with the same target object and target container. However, when a perfect match is not possible, reference instances containing semantically similar targets can be selected. For example, if the instruction is "put into TrashCan", the case is "GarbageBin". As long as the semantics are consistent and the object functions match, it can be adopted.

[0054] The case matching process can also be incorporated into a learnable retrieval network, for example, by building a cross-modal Transformer model to model bidirectional attention between input instruction-environment pairs and candidate case pairs, making the retrieval results context-sensitive.

[0055] Example Description: In a healthcare application, the system receives the instruction "Place the lancet into the medical waste bin." It identifies "Needle" and "MedicalTrashBox" in the environment and matches them with historical successful cases of similar device placement, such as the task trajectory of "Place the syringe into the SharpsContainer." This case includes information such as item type identification, path selection, and distance assessment, providing a reliable execution template for the current operation.

[0056] In the financial sector, when processing invoices, the instruction is "put the invoice into a folder." The system extracts the characteristics of "Invoice" and "DocumentBox" from the environment, queries the experience case library, finds historical cases of "putting vouchers into a folder" as a reference, and combines this with historical operation sequences and location planning to effectively guide the embodied agent to complete the archiving task. This mechanism significantly reduces the need for manually defining task sequences and improves the efficiency of office automation.

[0057] This embodiment utilizes an experience case library combined with a dual matching process of task instructions and environmental features to significantly reduce the computational overhead required for planning paths from scratch, thereby improving the efficiency and accuracy of task generation. The system uses historically validated task structures and behavioral templates for reasoning, reducing the risk of operational failures due to a lack of prior knowledge in complex environments. This mechanism exhibits good generalization capabilities in dynamic scenarios, effectively supporting the embodied agent's rapid response and adaptive adjustment in practical applications.

[0058] S30, decompose the task instruction into a sequence of subtasks containing the target object to be operated;

[0059] In this embodiment, after the embodied agent receives a natural language task instruction, it needs to perform a structured decomposition of the instruction to generate an executable sequence of subtasks. This task instruction is typically a complex natural language expression issued by a human user, such as "pick the knife off the table and put it in the trash can." Such instructions may implicitly contain multiple action nodes, involve multiple objects of operation, and their interaction relationships at the language level. Therefore, it is necessary to parse the task instruction, extract the target action sequence, interaction objects, and their sequential dependencies, forming a clear sequence of subtasks.

[0060] The decomposition of task instructions can be implemented based on natural language processing models. Common methods include task slot filling models based on sequence labeling, syntactic dependency tree analysis methods, or multi-stage deconstruction models based on Transformers. After completing word segmentation and semantic nesting parsing, the system identifies the operational verbs and their corresponding objects in the task and determines the execution order based on the context. For example, for "pick the knife off the table and put it in the trash can", the system can decompose it into two sub-tasks: one is "pick the knife off the table", and the other is "put the knife in the trash can".

[0061] Each subtask needs to include a clearly defined target object, which is the physical entity that the agent needs to perceive and manipulate. The extraction of the target object depends not only on the linguistic content but also on the alignment with the perception results of the current environment. If the instruction contains abstract objects or pronouns (such as "it" or "this"), the system will perform referential resolution in the current environment based on the instruction context and historical dialogue state to determine the specific object to be manipulated.

[0062] Furthermore, the sequence of subtasks needs to satisfy executability and contextual dependencies, and the system needs to identify the state transition relationships between successive subtasks. For example, an object must be reachable before being "picked up" and should not reappear in the operation list after being "placed." The system needs to ensure logical consistency through operation state modeling.

[0063] In one implementation, the system uses a pre-trained language task decomposition model, such as a task understanding module built on the BERT or T5 architecture, to encode task instructions and output action-object pair sequences for subtasks. For the instruction "Please take the blue folder out of the second drawer and put it on the desk," the system decomposes it into "Take the blue folder out of the second drawer" and "Put the blue folder on the desk."

[0064] The decomposition process can be enhanced with rule templates and intent recognition, especially in fields like healthcare or finance where specific terminology or sentence structures exist. Domain-specific verb dictionaries and target object tables can be defined to make the decomposition process more accurate. For example, "archive" and "submit" can correspond to subtask templates such as "store in the archive area" and "put in an envelope," respectively.

[0065] The system can also integrate graph neural networks for task semantic graph construction, constructing a directed graph for each verb and its associated noun node in the instruction, determining the execution order by the direction of the edges, and automatically generating operation paths based on this graph structure.

[0066] Example Explanation: In a healthcare scenario, an embodied agent receiving the instruction "Remove the medicine box and place it on the dispensing station" breaks down the instruction into two subtasks: "Remove the medicine box" and "Place it on the dispensing station," respectively labeling the target object "medicine box" and the operation location "dispensing station." This type of decomposition facilitates the agent's verification of the state of objects in the environment, such as confirming whether the medicine box exists in the designated location and can be safely retrieved.

[0067] In financial business scenarios, when processing the instruction to "organize invoices and expense reports together and put them into the blue folder", the system can break it down into three sub-tasks: "retrieve invoices", "retrieve expense reports", and "put them into the blue folder". Each sub-task is bound to a specific object, which improves the traceability of the operation and the clarity of the process, and facilitates subsequent task monitoring and problem tracing.

[0068] This embodiment significantly improves the system's ability to parse complex natural language instructions and the controllability of task execution by semantically decomposing task instructions and explicitly generating subtask sequences. Tasks are deconstructed into clear operation chains, which facilitates intermediate state evaluation and fault rollback during task execution. Furthermore, the explicit subtask sequences provide structured input conditions for subsequent reference case matching, target object state monitoring, and dynamic correction, enhancing the robustness of task planning.

[0069] S40, Generate a task plan based on the reference case and the sub-task sequence;

[0070] In this embodiment, the generation of the task plan relies on two core inputs: reference cases and subtask sequences. The reference cases are derived from an experience case library, representing the instantiated operation paths in the historical task execution process, including elements such as operation objectives, environmental context, action sequence, state transitions, and success conditions. The subtask sequences are the action-objective mapping chains parsed in the previous stage, which do not yet have specific execution parameters or time arrangements, and need to be further integrated with contextual experience to construct a complete execution path.

[0071] During task plan generation, the system first performs structural alignment between the reference cases and the current subtask sequence. This alignment goes beyond surface action matching, encompassing semantic role labeling and state transition modeling. For example, if the subtask is "pick up the scissors from the table," then historical fragments with similar operational goals (scissors) and action intentions (grabbing) need to be matched in the reference cases. Even if the reference cases describe it as "pick up the knife" or "grab the tool," as long as the semantic similarity is high and the context is relevant, it can still be considered a transferable path.

[0072] During the alignment process, the system can construct a multimodal embedding space to vectorize action semantics, target object attributes, environmental constraints, and spatial relationships between objects. Similarity scores are calculated within this space, and the reference path that best matches the subtask is selected. Matching considers not only the consistency between action pairs and noun pairs but also the context in which the action occurs, such as whether it is necessary to avoid other objects, whether there is a preceding action dependency, or whether a state transition is involved.

[0073] After the reference case matching is completed, the system enters the planning and combination phase. The core objective of this phase is to generate a suitable action chain based on the sequence requirements of the subtasks and the operation path of the reference case, and to populate the execution parameters. Each subtask will correspond to at least one action module, and each action module includes the operation type (e.g., grasp, move, place), the spatial coordinates of the operation object, operation constraints (e.g., force limits, gripping method), expected state transitions, and failure rollback strategies. These modules are dynamically combined into a task plan tree or task graph structure by the task scheduler, possessing parallel branching capabilities and error node jump mechanisms.

[0074] During this process, the system will also dynamically adjust the path structure of the task plan. For example, when there are path conflicts among multiple reference cases (such as different operation sequences of the same object), a heuristic scheduling mechanism can be introduced to score and rank the conflicting paths, prioritizing the solution that conforms to the constraints of the current environment. For subtasks with strong dependencies, such as "picking up the folder" followed by "opening the folder", the system will nest conditional constraints in the task plan, requiring that the next task can only be scheduled for execution after the previous task is successfully completed.

[0075] If a subtask has multiple candidate paths for reference cases, the system can use a path fusion strategy to construct a generalized plan. This strategy extracts a common structure from multiple execution paths as the main plan, while retaining optional paths as alternative plan branches, supporting dynamic switching based on environmental feedback during task execution.

[0076] In one implementation, the system uses a task plan generator built on a graph neural network to map subtask sequences to task graph nodes. Reference cases are used as the source of graph edge weights, and an edge weight selection algorithm automatically generates the shortest execution path. For example, although "putting the kettle on the table" and "putting the beverage bottle on the tray" involve different objects, if their historical operation paths are similar, the system can learn their commonalities: acquiring the object, moving it to the target area, and placing it. It can then construct a standard plan template and map it to the current task.

[0077] In another implementation, the task plan is generated by a module based on language model fine-tuning. This module takes into input subtask text descriptions and reference case text summaries, and outputs a structured sequence of actions, specifically including: object identifier, action type, target location, environmental conditions, expected result, and alternatives. For example, given a nested description of "taking scissors out of the drawer" and the reference case "taking knives out of the cabinet," the model generates a plan: locate the drawer → identify the scissors → grasp → pull out → confirm location.

[0078] Alternatively, a task combination system based on reinforcement learning can be used to learn the temporal combination and state evolution patterns of subtasks from historical task execution trajectories, constructing a task plan with the objective function of minimizing cost and maximizing success rate. The system repeatedly tries multiple planned paths in a simulated environment and uses actual execution feedback as a reward signal to optimize the action chain combination strategy.

[0079] In addition, to adapt to dynamic changes in complex scenarios, the system can integrate a plan interruptibility mechanism and a plan correction interface. If environmental interference or execution failure signals are encountered during task execution, the plan can be reconfigured according to the failure recovery path recorded in the original reference case, ensuring the flexibility and recoverability of the task plan.

[0080] Example Explanation: In a healthcare scenario, for the task instruction "deliver the blood collection needle to the recycling bin," the system extracts a similar path based on historical case execution records of "dispose of the syringe in the medical waste bin": identify the target object → determine if it can be grasped → move it to the recycling area → accurately locate the disposal port → release the object. Based on this path, the current subtask sequence is mapped to a homogeneous task plan, automatically filling in the recycling bin location information and release distance, completing the operation path construction and scheduling execution.

[0081] In the financial sector, when faced with the task instruction "file invoices and expense reports into a folder", the system aligns its subtask sequence "retrieve invoices", "retrieve expense reports", "open folder", and "put into folder" with the historical case of "organizing and archiving ledgers", constructing a task path that includes steps such as fetching, classifying, and archiving. At the same time, a failure recovery strategy is added. If the target file location changes or the folder is closed, the system can execute a detour or alternative deployment plan based on the historical path to ensure that the archiving operation is completed stably.

[0082] This embodiment significantly enhances the rationality and environmental adaptability of task path generation by fusing historical reference cases with the current subtask sequence to generate a structured task plan. Unlike static template matching methods, fusing case experience allows the generated path to automatically incorporate previous successful strategies and failure avoidance experiences, achieving cross-task transfer and structural generalization. The generated task plan includes not only action sequences but also parameter configurations, operation order, environmental dependencies, failure jumps, and expected results, which is beneficial for highly reliable execution of complex tasks in embodied environments. Through modular generation and dynamic scheduling mechanisms, it also supports real-time adjustments and local repairs, thereby significantly improving the robustness of task execution and the agent's adaptability.

[0083] S50, execute the current subtask in the task plan, and detect whether the target object of the current subtask is missing during the execution of the current subtask;

[0084] In this embodiment, the execution of the current subtask depends on the task plan generated by the preceding operation process. This plan clearly defines the target object, operation parameters, and environmental conditions of the current action. The execution process needs to access these parameters in real time and drive the embodied control module to perform physical operations. The current subtask is an operation node in the task plan, and its definition typically includes the operation type (such as grab, move, place, etc.), target object identifier (such as entity name, category, attribute characteristics), operation position coordinates, and timing constraints.

[0085] When scheduling the current subtask, the system needs to call functional modules such as visual recognition, spatial perception, robotic arm control, and navigation decision-making through the embodied control interface. First, it acquires the spatial location and state information of the target object corresponding to the current task in the environment, possibly using methods such as RGB image analysis, depth perception, point cloud clustering, or multimodal recognition to identify the target object's appearance, shape, and spatial location. If the target object is a specific tool, implement, or marker, multimodal fusion matching can be performed by combining predefined labels, visual maps, or language annotations.

[0086] Subsequently, the system enters the execution phase, invoking lower-level execution modules to issue specific action commands, such as initiating the movement of the gripper arm, path planning and navigation to the target location, alignment correction, and gripping execution. During this phase, the state of the target object needs continuous monitoring to determine whether it exists at the expected location, whether it is operable, or whether its state has changed (e.g., obscured by other objects, moved, occupied, or no longer reachable).

[0087] The process of detecting whether a target object is missing is not limited to judging its presence or absence, but requires establishing a target state model. This model includes indicators such as the spatial reachability, semantic consistency, and operational feasibility of the target object. Specifically, a target state scoring mechanism can be used, such as calculating the confidence of the object in the current vision, the semantic similarity between the predicted target and the planned target, and determining the degree of occlusion or the feasibility of the operation path. When any key indicator falls below the tolerance threshold, the system can determine that the object is in a "missing" state, meaning that the current task cannot be successfully executed.

[0088] Furthermore, this detection operation has real-time requirements, typically requiring multiple status updates and checks before and during execution. The system can be designed with triggering mechanisms, such as performing a final check before approaching the target, or conducting emergency checks after execution feedback such as capture failure or path obstruction to determine if the issue is due to target missingness.

[0089] The above operations are achieved by establishing a multi-level perception and feedback loop, including high-frequency image recognition updates, action result monitoring, pose tracking, and tactile or force feedback as supplementary perceptions, thereby enabling timely identification of target anomalies without increasing global replanning.

[0090] In one implementation, the system constructs a target recognition module using a deep visual neural network to classify and locate targets in environmental images captured before task execution, obtaining the confidence score and bounding box of the target object. Subsequently, a 3D reconstruction module constructs a physical scene point cloud map and performs spatial fitting on the target object to determine if it falls within the planned spatial area. If the positional deviation exceeds the allowable range, or the recognition confidence score is below a threshold, the target is determined to be missing.

[0091] In another approach, the system performs a front-view capture operation before each subtask is executed, and uses semantic segmentation and multi-target tracking to determine whether the target is located at the expected point. If the target is not at its original point but appears in an adjacent area, the system records it as an offset state and allows execution; if the target is completely lost, it jumps to the target loss handling process.

[0092] Alternatively, a grasping control arm with embedded tactile sensors can be used to detect whether standard force feedback is triggered during the operation attempt. If there is no feedback or abnormal feedback (such as no target contact or misoperation during empty grasp), the system determines that the target is missing and generates a relevant interrupt signal for use by the subsequent task correction mechanism.

[0093] To address the problem of target interference in highly dynamic environments, a redundancy mechanism based on multimodal fusion detection can be implemented. For example, target verification logic can be constructed by combining multiple dimensions such as language guidance (e.g., recognizing "red round cup"), object attributes (e.g., plastic material, height range) and spatial context (e.g., at the corner of a table) to improve the robustness of missing data determination.

[0094] Example Explanation: In a healthcare scenario, when the embodied system performs the subtask of "placing the lancet into the recycling bin," the target object is the "recycling bin." If, before execution, the identification module detects that the recycling bin has been moved to another corner or is obstructed by debris, the system can mark the missing target status in real time through a detection mechanism, avoiding directly moving to the original location to perform the placement operation, thereby preventing misplacement or collision issues. The system can then proceed to the alternative object matching stage to correct the plan.

[0095] In a financial transaction processing scenario, when executing the task of "putting invoices into a folder," the original target was planned to be a "blue folder," but due to desktop organization, this folder has been replaced with a "file folder." The visual recognition system detects the missing target before the operation, and through semantic matching, finds a new object that meets the operation requirements, thereby triggering subsequent correction and replacement logic to ensure that the action path is not interrupted.

[0096] This embodiment effectively avoids issues such as target inoperability or execution failure by detecting whether the target object is missing in real time during subtask execution, preventing erroneous actions or execution interruptions. Compared to static strategies that only perform a one-time detection during task initialization, this method dynamically refreshes the target state before or during each execution, exhibiting high adaptability and robustness. The multimodal state judgment mechanism can respond quickly to situations such as object occlusion, name discrepancies, or slight movements, improving the agent's responsiveness to complex environments and providing stable support for subsequent local corrections and plan adjustments.

[0097] S60, when it is detected that the target object of the current subtask is missing, the objects in the current environment are obtained to form a current environment object set;

[0098] In this embodiment, if the target object of the current subtask is missing, it is necessary to re-identify the set of objects in the current environment that can be used to continue the task. This process takes environmental perception data as input and combines visual images, depth maps, 3D point clouds, or semantic annotation information to extract all identifiable objects from the environment and construct the current environmental object set. The definition of objects in the environment typically includes object category, attribute features, spatial location, operability, etc., aiming to construct a multi-dimensional representation of task-related candidate entities.

[0099] The operation of acquiring a set of objects is generally based on a local perception area, which can be limited by the camera's field of view, the reachable space of the robotic arm, and the current pose perception radius. In terms of implementation, a forward-looking vision recognition module can capture a sequence of images of the current scene, and a multi-object detection network (such as YOLO, Mask R-CNN, etc.) can be applied to identify all target objects in the scene that have entity boundaries and semantic labels. Each recognition result needs to be bound to the entity category (such as "folder," "document box," "trash can"), confidence score, physical size, spatial coordinates, and contextual attributes (such as color, material, state, etc.).

[0100] Furthermore, to enhance the usability of the object set, the system needs to further filter out objects that are irrelevant to the current task or inoperable. An operational scoring mechanism can be introduced to evaluate parameters such as stability (whether it is fixed), reachability (whether it is occluded), and consistency (whether it matches the current context) for each identified object, eliminating unusable entities and retaining only candidate objects into the set. Each object in the set should also support multimodal descriptions to support subsequent semantic substitution matching. Typical data structures may include visual embedding vectors, text label descriptions, attribute vectors, etc.

[0101] In some implementations, the construction of the object set needs to consider the impact of scene changes, such as movement and occlusion, lighting changes, and camera blur. In such cases, combining temporal image frames and employing strategies such as dynamic background modeling, target tracking fusion, and temporal stability filtering can improve object recognition accuracy. If necessary, auxiliary sensing devices (such as RGB-D cameras, LiDAR, and inertial navigation) can be used to enhance perceptual redundancy and data accuracy.

[0102] The core objective of this process is to reconstruct the structure of environmental objects based on the current actual perception, and to provide an effective candidate resource pool for subsequent target semantic matching and task path correction.

[0103] One approach is based on a deep learning detection network. It acquires RGB images from the robot's current perspective and performs object recognition. A trained multi-class object detection model outputs multiple bounding boxes and class labels, while simultaneously obtaining the object's relative position coordinates. Then, a 3D reconstruction module is invoked to reconstruct the object's position and size based on the depth map or point cloud. Finally, a confidence score is used to select structurally complete and highly identifiable objects as candidate objects.

[0104] Another approach involves continuously acquiring multiple frames of images after the robot moves or rotates its viewpoint. These frames are then used to track the trajectories of potential targets through temporal image fusion and target tracking modules (such as SORT and DeepSORT), forming a stable set of candidate targets. This method can eliminate target recognition distortion caused by short-term occlusion or detection omissions.

[0105] Multimodal fusion can also be used, combining key descriptive information from voice input or task context to filter objects identified in the environment by attribute. For example, when the task description is "put the red file into the storage box", the system extracts all "storage box" category objects in the current environment and further filters out those with the red attribute or similar appearance, adding them to the environmental object set.

[0106] Another approach is to introduce graph structure modeling, which constructs all objects identified in the current environment as graph nodes, establishes edge weights through dimensions such as spatial distance, attribute similarity, and interaction frequency, executes a clustering algorithm based on specific semantic weights in the graph structure, and extracts the candidate object subgraph most relevant to the target task context as the core part of the environmental object set.

[0107] Example Explanation: In a financial business scenario, if the original task objective is "to put the checks into the blue storage box," but the storage box is not in its original position during execution, the system rescans the desktop and surrounding area using a camera, identifying several box-shaped objects with different shapes, colors, and labels. Through detection and filtering, the system groups objects labeled "storage box," "voucher folder," or "document box" into a set of current environment objects, providing candidate resources for subsequent replacement object determination.

[0108] In a healthcare scenario, if the subtask objective is to "place the medicine box in the left-hand medicine cabinet", and it is discovered during execution that the target medicine cabinet has been removed due to on-site disinfection, the system immediately calls the environmental recognition module in the current view to identify the "spare cabinet" and the cabinet "marked as temporary medicine storage" on the right wall, and extracts them as a set of objects. This allows the task to be completed subsequently using semantic substitution without affecting the continuity and security of the medicine management process.

[0109] This embodiment reconstructs the current set of environmental objects after the target object is lost. The system can proactively explore existing environmental resources, providing data support for selecting alternative objects and correcting task paths. This mechanism enhances the embodied agent's environmental understanding and adaptive behavior in complex and dynamic environments, avoiding a passive waiting or global replanning situation after the target is lost. The process of building the object set using semantic multimodal expression also improves the semantic generalization ability for similar entities, enhancing the transferability and execution continuity of operational tasks.

[0110] S70, determine a substitute object semantically associated with the target object in the current set of environment objects;

[0111] In this embodiment, during the execution of a subtask, when the original target object is detected as being unable to be perceived or identified in the current environment, the system needs to actively search for a semantically related alternative object from the current environment object set to maintain the logical consistency and executability of the task. The current environment object set is typically generated by the pre-sensing module, and this set contains all identifiable physical objects and their related attribute information, such as name, category, spatial location, geometric features, historical usage frequency, and task context labels.

[0112] Semantic relevance is defined as the degree of similarity in semantic space between the desired object in the current subtask and existing objects in the environment. To achieve this recognition process, the system needs to construct a unified semantic embedding model that can map object names, attributes, and task context to a unified semantic vector space. This model can be trained based on pre-trained language models (such as BERT, RoBERTa) combined with domain knowledge graphs to ensure that semantic relationships cover task-related semantic variations and entity references.

[0113] The matching process consists of a multi-level filtering and ranking mechanism. The first level uses the cosine similarity between semantic vectors to select several candidate objects. The second level combines operation history and task context information for scoring, giving priority to objects that have been used for similar operations or semantic instructions. The third level considers the accessibility and physical adaptability of objects in the current environment, such as spatial accessibility, the clamping surface required for the operation, stability, and other conditions, to form the final candidate set.

[0114] Furthermore, this recognition process also needs to be fault-tolerant and interpretable. When there is a discrepancy between the target object's name and the actual annotation of the scene (such as "trash can" and "waste bin"), the system can perform semantic generalization reasoning based on hierarchical or synonym substitution models. For example, if the target object is a "folder" and objects such as "credential bag" and "document box" appear in the current set of objects, the system can recognize that they have equivalent semantic roles in terms of task intent.

[0115] The final identified replacement object not only meets the semantic similarity requirement, but also needs to pass multiple constraint tests such as environmental feasibility judgment, task consistency verification and operation cost analysis to ensure that the replacement object can actually participate in task execution and will not cause functional errors or semantic deviations.

[0116] In one implementation, the system uses a knowledge graph to construct semantic graph edges between target objects and environment objects, and sets a semantic jump distance threshold, determining a substitutable relationship only when two objects belong to the same functional class (such as the container class for "containing items"). Each object is represented as a composite vector containing semantic labels, functional descriptions, operation records, and task context labels.

[0117] Another approach is based on a language-vision joint embedding model, which performs cross-modal matching between the target object described in natural language (such as "folder" in "put the invoice in the folder") and the environment object recognized visually. After the visual object is converted into a descriptive label, it enters the embedding space for semantic distance measurement.

[0118] Alternatively, a lightweight vector lookup mechanism can be used to maintain a local embedding cache, which is used to quickly and approximately search for entities that are closest to the target semantics, greatly improving matching efficiency. This is suitable for edge devices or business scenarios with strict response time limits.

[0119] In tasks with high security requirements, a set of manually defined semantic substitution rules can be introduced, such as: "If a 'needle collection box' cannot be found, try a 'medical waste bin'." These rules are triggered as a fallback strategy when the semantic matching score does not exceed a threshold, improving the system's stability in unstructured environments.

[0120] Example Explanation: In a financial business scenario, when the system executes the task of "putting invoices into a folder," since no objects labeled "folder" are detected in the actual workplace, but there is an entity labeled "voucher bag," the system identifies it as a replaceable object based on the high semantic similarity between the embedded vectors. Combining its location, reachability, and operation history, the system determines it as the replacement target and proceeds to the subsequent correction task execution process.

[0121] In a medical setting, a nurse inputs the command "Put the test tubes into the recycling bin" via voice or text. However, no object labeled "recycling bin" is detected in the current ward environment. The system identifies an entity named "medical waste bin" from the environmental objects and, based on the equivalent entity definition of "recycling bin" in the medical knowledge graph, automatically determines it as a suitable replacement. The task can then continue without manual intervention, effectively avoiding response delays and potential errors caused by manually re-entering commands.

[0122] This embodiment identifies alternative objects semantically related to the original target object from the current set of environmental objects, enabling the system to maintain task execution continuity even when key objects are missing. This mechanism effectively alleviates the reliance on precise matching of single instruction terms for task execution, enhances the embodied agent's ability to cope with non-standard naming, missing objects, or dynamic changes in reality, and prevents the linguistic ambiguity of task instructions from becoming a source of task interruption. It significantly improves the usability and robustness of the intelligent system, especially demonstrating stronger self-recovery capabilities in unsupervised or few-sample scenarios.

[0123] S80, update the current subtask using the replacement object to generate a corrected subtask, and execute the corrected subtask to complete the task plan.

[0124] In this embodiment, after identifying a substitute object semantically associated with the missing target object from the current set of environment objects, the system needs to introduce the substitute object into the current subtask and generate a new executable subtask, i.e., a correction subtask, while maintaining the semantic integrity of the task. This process involves updating operation parameters, replacing instruction targets, rebinding dependent resources, and adapting execution paths.

[0125] The correction process first requires the system to have a structured and parsable representation of subtasks. Typically, the current subtask is represented as a structure or action tree containing fields such as action type, target object, spatial location, and operation tool. Upon discovering that the target object is missing and a replacement object has been identified, the system replaces the original target field in the structure with the replacement object identifier. Simultaneously, it evaluates the spatial reachability, operational feasibility, and semantic consistency of the replacement object to ensure the integrity of the task logic.

[0126] When generating corrective subtasks, the system also needs to consider whether the replacement object differs from the original object in terms of operational conditions, such as geometric dimensions, gripping points, load-bearing capacity, and material compatibility, which may affect the generation of operation paths, motion parameters, or control commands. To address these potential differences, the system can combine the perception module to obtain the geometric and physical parameters of the replacement object, and recalculate the path or motion posture parameters through the motion planning module to ensure that the corrective subtask has physical executability.

[0127] The process of executing the correction subtask is consistent with the execution flow of the atomic task. That is, the execution engine schedules the execution unit to complete the operation, monitors the execution status in real time, and determines whether the operation plan needs to be adjusted again, supporting multi-level correction.

[0128] The entire process implements a dynamic repair mechanism for task plans, enabling tasks to remain flexible in the face of environmental fluctuations or resource unavailability, thereby improving the agent's task continuity and environmental adaptability.

[0129] One implementation approach is to use a composable task representation language (such as PDDL or a custom DSL) to represent subtasks as structures containing operation targets. When the target object is replaced, the system rewrites the parameter fields in the task structure and triggers the downstream path replanning module to update the control flow graph and timing dependencies.

[0130] Another approach is to use an operation graph model based on graph neural networks, representing subtasks as a graph structure between operation nodes and target object nodes. In this case, after the target node is replaced, only the features of the relevant nodes in the graph need to be updated and locally backpropagated to generate the corrected operation flow.

[0131] A correction strategy model can also be constructed, which takes the semantic vector of the target missing event and the substitute object as input, and outputs correction actions or new execution strategies to achieve a more abstract level of operation repair.

[0132] In certain task scenarios with strict constraints, conditional rule templates can be used for correction and judgment. For example, the semantic type of the replacement object must be "file container class" and the spatial location must be reachable before the update operation can be performed, thereby ensuring that the task logic constraints are not violated.

[0133] Example description: In a medical scenario, when the embodied agent receives the subtask "put the blood collection needle into the recycling bin" and identifies that the "recycling bin" is missing, and the previous step identified "medical waste bin" as a semantically related substitute object, the system updates the target object field in the subtask structure from "recycling bin" to "medical waste bin", recalculates the physical path to avoid the current device layout, and executes the corrected subtask to complete the item placement operation.

[0134] In a financial scenario, if the original task is to "file invoices to a folder," but the folder location is lost, the system has identified "voucher bag" as a semantic substitute. The subtask execution target is then updated to "voucher bag," and the gripping force parameters and placement height in the grabbing and placement actions are adjusted. This ensures the corrected task is completed smoothly without affecting the integrity of the documents, guaranteeing the stability of the process and the accuracy of document processing.

[0135] This embodiment effectively solves the problem of task interruption due to missing target objects during task execution by using identified replacement objects to update the current subtask and generating corrected subtask execution operations. This mechanism allows the system to dynamically adapt to environmental changes during operation, achieving the original task objective without relying on human re-entry of instructions, thereby improving the continuity of task execution and the system's intelligent autonomy. Furthermore, combining structured task representation with automated correction mechanisms can shorten response time and reduce resource waste caused by environmental disturbances.

[0136] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as fintech and healthcare. It discloses a task planning and correction method, apparatus, device, and medium, comprising: acquiring task instructions and environmental perception data; generating instruction features and environmental features based on the task instructions and environmental perception data, respectively; retrieving reference cases from an experience case library based on the instruction features and environmental features; decomposing the task instructions into a sequence of subtasks containing a target object to be operated on; generating a task plan based on the reference cases and the subtask sequence; executing the current subtask in the task plan, and detecting whether the target object of the current subtask is missing during the execution of the current subtask; when the target object is detected to be missing, acquiring objects in the current environment to form a current environment object set; determining a substitute object semantically related to the target object in the current environment object set; updating the current subtask using the substitute object to generate a corrected subtask; and executing the corrected subtask to complete the task plan. This invention generates a task plan jointly using environmental perception data and reference cases, and dynamically acquires substitute objects based on the current environment to construct corrected subtasks when the target object is missing. This achieves local adaptive correction and continuous progress of the task, avoiding the task interruption problem caused by target loss in traditional methods, and improving the robustness of task execution and the adaptability of the agent in the real environment.

[0137] In one embodiment, step S10 above includes:

[0138] S101 receives input natural language instructions as task instructions and acquires environmental perception data through image acquisition devices;

[0139] S102, perform semantic parsing on the task instructions to extract task action elements and task object elements;

[0140] S103, input the task action elements and task object elements into the language processing model to generate text embedding vectors, and use the text embedding vectors as instruction features;

[0141] S104, perform image preprocessing on the environmental perception data to extract key visual regions;

[0142] S105, input the key visual region into the visual processing model to generate a visual embedding vector, and use the visual embedding vector as an environmental feature.

[0143] In this embodiment, the system first receives natural language task instructions initiated by the user through a task input channel. These instructions can originate from speech-to-text, text input, or language task templates generated by a third-party task scheduling platform. The natural language task instructions need to undergo structured processing to identify operational and target information that can be used for downstream task planning. To this end, the system employs a semantic parsing module based on language modeling to perform part-of-speech analysis, syntactic structure modeling, and semantic role labeling on the original instructions, thereby extracting operational task action elements and entity object elements related to the task target. Task action elements are typically represented as verb phrases or combinations of verbs and their modifiers, indicating the actual operational behavior in the instruction; task object elements encompass entity identifiers such as the object of the action, required tools, and operation location.

[0144] The extracted task action elements and task object elements are uniformly converted into text input and fed into a language processing model to obtain their corresponding semantic embedding representations. The language processing model can employ a structure with strong bidirectional context modeling capabilities, such as BERT, RoBERTa, or T5 variants. By mapping the text to a high-dimensional semantic space, it generates embedding vectors that maintain semantic similarity. These vectors not only preserve the semantic information of the task intent but also possess cross-task universality, facilitating matching, reasoning, and fusion with subsequent modules. The generated text embedding vectors are recorded as instruction features, serving as the core input in the entire system for describing language task objectives and operations.

[0145] In terms of environmental data acquisition, the system synchronously or asynchronously triggers the environmental perception module to acquire visual information of the current execution scene using image acquisition devices (such as wide-angle RGB cameras, depth cameras, or multimodal fusion devices). Image acquisition can employ high frame rates and short exposure times to reduce motion blur, and combines automatic exposure and white balance adjustment to optimize image quality. The acquired environmental images undergo an image preprocessing process, which includes geometric correction, noise reduction, edge enhancement, and color space transformation, aiming to improve the recognizability of key areas and reduce visual redundancy interference.

[0146] After image preprocessing, key visual regions are extracted using a region proposal mechanism. This mechanism can generate a set of regions of interest based on region generation networks (such as Selective Search and RPN) or target candidate region clustering algorithms (such as EdgeBoxes). Each region is bound to its corresponding location, size, and initial visual feature values. The selected key visual regions are then input into a visual processing model for further encoding. The visual processing model can be a convolutional neural network (such as ResNet and EfficientNet) or a Transformer structure (such as ViT and Swin Transformer). It uses a spatial attention mechanism to focus on the image representation of salient or task-related targets in the scene, and finally outputs a visual embedding vector for each region.

[0147] After aggregating the embedding vectors of all key visual regions, a scene feature compression layer is used for dimensionality reduction and normalization to generate a unified environment representation vector, i.e., environment features. These environment features preserve the spatial relationships, appearance features, and contextual information of operable objects in the scene. They will then be jointly analyzed with instruction features for use in task planning, target matching, and anomaly detection.

[0148] The entire processing flow emphasizes the decoupling and re-integration of language commands and environmental awareness, ensuring that the system can not only independently understand the task intent, but also concretely match the task execution elements in a specific environment, and has cross-scenario execution capability and dynamic adaptability.

[0149] This embodiment achieves dual-structured modeling of task instructions and environmental states through the simultaneous execution of language parsing and visual perception. This generates semantically consistent and logically self-consistent instruction and environmental features, serving as the foundational expression for subsequent task reasoning and planning. This approach effectively avoids the reliance on rule template matching in traditional task scheduling systems, improving the robustness of instruction parsing under linguistic diversity and semantic ambiguity. Simultaneously, by extracting key regions and employing deep embedding encoding based on a visual attention mechanism, the system ensures accurate identification of environmental entities related to the operational target, enhancing the contextual adaptability and environmental understanding capabilities of task execution. Ultimately, this processing mechanism significantly improves the agent's ability to identify and manipulate task targets in complex dynamic environments, supporting more flexible and robust reasoning and execution in the task planning module.

[0150] In one embodiment, step S20 above includes:

[0151] S201, extract the text embedding vector of the instruction features and extract the visual embedding vector of the environment features;

[0152] S202, Obtain the textual and visual features of historical cases in the experience case library;

[0153] S203, determine the text similarity between the text embedding vector and the case text features;

[0154] S204, determine the visual similarity between the visual embedding vector and the case visual features;

[0155] S205, The text similarity and visual similarity are combined to generate a comprehensive similarity;

[0156] S206, Sort historical cases according to comprehensive similarity, and select historical cases whose comprehensive similarity ranking is above the preset ranking threshold as reference cases.

[0157] In this embodiment, after processing the task instructions and environmental perception data, the system obtains instruction features and environmental features, namely text embedding vectors and visual embedding vectors, respectively. It then proceeds to a reference case retrieval process based on multimodal feature matching. This process first extracts the aforementioned generated text embedding vectors and visual embedding vectors. These two types of vectors respectively express the semantic content of the task and the information of environmental entities, and are both represented in the form of high-dimensional floating-point vectors, possessing semantic structural features in the embedding space.

[0158] The experience case library stores a large number of historical task execution records, each containing task description text and corresponding environmental scene data. During the construction of the case library, all cases are processed using a model consistent with the current instruction and environment processing flow to generate case text features and case visual features isomorphic to the current task. Case text features are typically embedded vector representations of static text descriptions (e.g., "Place the medicine bottle on the tray"), while case visual features are sets of key region embedded vectors from images or video frames of the task execution environment at the time. These features are stored uniformly and support efficient retrieval.

[0159] The matching phase first calculates the text similarity between the current task's text embedding vector and the text features of each case. The calculation method can be vector dot product, cosine similarity, Mahalanobis distance, or the negative of Euclidean distance, measuring semantic overlap through geometric relationships in the vector space. Simultaneously, the visual similarity between the current task's visual embedding vector and the visual features of the cases is calculated. This process can employ visual cluster center comparison, cross-attention mechanisms to construct region alignment scores, or calculate the cosine distance between features after mapping through a specific projection space, thereby reflecting the similarity of entity placement and target overlap between the two environmental states.

[0160] To achieve multimodal consistent retrieval, the system integrates textual and visual similarity to form a comprehensive similarity score. Fusion methods include weighted linear combination, adaptive gating mechanisms, and joint probability scoring based on mutual information. Weighting factors can be set based on historical statistical data or dynamically adjusted to adapt to the modality dependencies of the current task.

[0161] After the comprehensive similarity score is constructed, the system sorts all candidate cases in descending order according to their comprehensive similarity scores, resulting in a ranked list. Then, based on a preset ranking threshold, the system selects the top few cases from the ranked list as reference cases for the current task. The ranking threshold can be a fixed number (e.g., Top-3) or an adaptive score threshold (e.g., similarity exceeding a certain dynamic threshold), ensuring that the selected reference cases are highly relevant to the current task both semantically and in terms of context.

[0162] This process achieves joint matching of instruction semantics and environment state, ensuring that the reference case has structural alignment and behavioral transferability at the multimodal level, and providing accurate and structured prior knowledge support for subsequent task decomposition and task plan generation.

[0163] This embodiment employs a dual-modal similarity retrieval mechanism combining text and vision. This mechanism not only accurately captures the correspondence between instruction semantics and environmental entities but also integrates semantic consistency and perceptual overlap between the two, improving the accuracy and practicality of reference case selection. This mechanism effectively overcomes the modal bias problem caused by single semantic or single visual retrieval in traditional task planning, enhancing cross-scenario behavior transfer capabilities and task matching robustness. In dynamic and complex interactive environments, this mechanism can quickly locate historical experiences consistent with the current task context, thereby reducing the complexity of plan generation, shortening inference paths, and improving the reliability and efficiency of task execution.

[0164] In one embodiment, step S30 above includes:

[0165] S301, parse the semantic structure of the task instruction and identify the core action elements in the semantic structure;

[0166] S302, extract the object acting on the core action element as the target object to be operated on;

[0167] S303, determine the associated container of the target object to be operated on based on environmental constraints;

[0168] S304, combine the core action elements, the target object to be operated on, and the associated container to form a sub-task unit;

[0169] S305, the subtask units are arranged in the execution logic order to form a subtask sequence.

[0170] In this embodiment, decomposing the task instructions in the early stages of task execution is a crucial step in enabling the system to understand and execute complex tasks. Task instructions are typically expressed in natural language, and their structure is highly flexible and ambiguous. Therefore, a systematic semantic structure parsing mechanism is needed to transform them into a sequence of operable subtasks.

[0171] First, the system employs a language parsing module to perform semantic structure analysis on task instructions. This module uses a semantic dependency analysis model, a Transformer-like language model, or a parsing engine based on syntax graphs to structurally model natural language instructions. By constructing syntactic dependency graphs, event trigger graphs, or semantic role annotation graphs, it identifies the core action elements in the sentence that carry the main operational intent. Core action elements include verb phrases or compound action structures, such as "place," "move," "load," and "check." The system determines the main action and its semantic association boundaries through rule matching and model inference.

[0172] After the core action elements are located, the system continues to extract their corresponding objects, i.e., the physical entities or abstract objects to be manipulated in the task, referred to as the target objects. Target objects can be identified through the object position in semantic roles, semantic context entity recognition, and referential resolution mechanisms. For example, in the action "put the medicine bottle into the tray," the "medicine bottle" is the target object of the action "put into." The system performs unified naming and identification of target objects to ensure consistent referencing in subsequent processing.

[0173] After identifying the target object, the system further introduces environmental constraint factors to determine its associated containers. Environmental constraints include spatial location constraints, reachability constraints, task context restrictions, and interaction relationships between object categories. The system analyzes spatial distribution maps, object semantic tags, and container structure hierarchy diagrams within the environmental features to determine the environmental interaction relationships involved when the target object performs the task. For example, in the "placing on a tray" scenario mentioned above, the "tray" is the target container for the "medicine bottle." If multiple containers of the same type exist in the environment, the system determines the best match based on distance, reachability, and task context information.

[0174] Subsequently, the system combines the aforementioned three elements—core action element, target object, and its associated container—to form a structurally complete subtask unit. This subtask unit can be represented as a triple structure or an instruction structure with operational semantics, such as <Action: Place, Object: Medicine Bottle, Location: Tray>, which supports modular execution and state tracking.

[0175] After generating all subtask units, the system sorts these subtasks based on semantic dependencies and execution order logic to form the final subtask sequence. The sorting method incorporates information such as semantic causal dependencies, spatial flowcharts, goal achievement order, and temporal templates from historical task samples. For example, if a task includes "opening the cabinet door" and "taking out the medicine bottle," then the "open" operation must precede the "take out" operation. The sorting result constitutes a clear task decomposition path, supporting downstream task planning and execution scheduling.

[0176] Throughout the process, the transformation from a natural language task to a structured, executable sequence of subtasks is achieved through structured processing of the task language, alignment of semantic backbone and entities, injection of environmental constraints, and construction of logical order, effectively improving the determinism and adaptability of task processing.

[0177] This embodiment utilizes a task instruction decomposition mechanism to transform complex operational expressions in natural language into a sequence of subtasks with clear semantic boundaries and environmental entity constraints, achieving a closed-loop structure from instruction to action execution. By incorporating environmental factors as constraints through semantic structure analysis and target container association, the accuracy and operability of subtask decomposition are improved, avoiding task parsing errors caused by semantic ambiguity or environmental bias. Simultaneously, the generated subtask sequence possesses logical consistency in execution order, providing structured input for subsequent task plan generation and execution scheduling, significantly enhancing the system's parsing capabilities and task-driven stability when dealing with multi-step, multi-entity tasks.

[0178] In one embodiment, step S50 above includes:

[0179] S501, parse the action instructions and target object identifiers of the current subtask in the task plan;

[0180] S502, initiate the execution of the action command, and acquire the environmental perception data stream in real time during the execution process;

[0181] S503, segment the candidate object region in the environment perception data stream and extract the feature vector of the candidate object region;

[0182] S504, in the semantic feature space, determine the similarity between the feature vector and the semantic description vector of the target object identifier;

[0183] S505, when the similarity exceeds the matching threshold, it is determined that the target object is not missing;

[0184] S506, when the similarity does not exceed the matching threshold, it is determined that the target object is missing.

[0185] In this embodiment, during the execution of the task plan, the agent needs to perform operations one by one according to the predetermined sequence of subtasks. Each subtask typically contains a specific action instruction and its corresponding target object. To ensure that the execution result is consistent with the expected plan, the existence of the target object on which the subtask depends must be continuously detected in the current environment during the execution of each subtask.

[0186] The system first parses the structure of the current subtask, identifying the action instructions and target object identifiers contained within it. Action instructions typically originate from operation items generated during the task decomposition phase, including operation type, operation method, and operation constraints. The system then triggers the action. Target object identifiers are provided by the preceding language analysis and object recognition modules, including object name, category attributes, location information, or semantically relevant embeddings. This identifier serves as a reference standard for subsequent semantic matching.

[0187] After the activation command is executed, the system needs to collect environmental perception data streams in real time, typically image sequences captured by visual sensors (such as RGB cameras, depth cameras, etc.). To reduce processing redundancy, the system uses a sliding window strategy to select representative frames from consecutive frames, avoiding repeated analysis of redundant data.

[0188] The system performs candidate region segmentation on each frame of the perceived image, using a region proposal network (such as RPN) or saliency detection network to identify a set of regions that may contain objects, and extracts features from these candidate regions. The feature extraction module can call a pre-trained convolutional neural network (such as ResNet or EfficientNet) to generate fixed-dimensional visual embedding vectors, or combine a Transformer structure to obtain region semantic vectors with contextual information.

[0189] After acquiring candidate region features, the system needs to establish a semantic space matching relationship with the current target object identifier. The semantic description vector of the target object is usually obtained by encoding the target name or description using a language model (such as BERT or T5), and is pre-generated during the initial task instruction decomposition stage. The system adopts a unified semantic embedding space, enabling direct similarity calculation between visual features and text descriptions, for example, through visual-language fusion using the CLIP model or a joint embedding network.

[0190] For each candidate region's feature vector, the system calculates its similarity score with the target object's description vector, often using cosine similarity or bilinear attention mechanisms to measure the relationship between vectors. To improve discrimination robustness, similarity calculation can incorporate multi-head attention mechanisms or context-aware biases to strengthen semantic relevance rather than pure vector distance.

[0191] Finally, the system evaluates the similarity scores of all candidate regions. If the highest score exceeds a preset matching threshold, it indicates that an object entity with semantically consistent with the target object exists in the environment, meaning the target object is not missing, and the current subtask can continue. If the similarity between all candidate regions and the target object description does not reach the threshold, it means that the target object has not been detected in the current environment, and the system will determine that the object is missing, triggering subsequent alternative object analysis and task correction mechanisms.

[0192] Through this processing flow, the system achieves closed-loop control from task execution to dynamic perception of the target object, ensuring that any potential target loss during execution can be identified and responded to in a timely manner.

[0193] This embodiment achieves linkage between the task execution state and the target perception state by dynamically judging the target object of the current subtask by combining visual perception and semantic matching during task execution. When the target object is lost or unrecognizable, the system can immediately respond and enter a correction process to avoid task execution failure or abnormal state. By using cross-modal similarity calculation between visual features and semantic vectors, the robustness and generalization ability of target recognition are effectively improved, enabling the system to adapt to problems such as changes in object pose, occlusion, or differences in name expression in visual scenes, significantly enhancing the execution stability and fault handling capability of the task system in real-world environments.

[0194] In one embodiment, step S60 above includes:

[0195] S601, when the target object of the current subtask is detected to be missing, an environment rescan instruction is triggered;

[0196] S602, real-time environmental perception data is reacquired based on the environmental rescan instruction;

[0197] S603, Identify all visible objects in the real-time environmental perception data;

[0198] S604, Extract the object category identifier and location information for each visible object;

[0199] S605, combine the object category identifier and location information to form an object description item;

[0200] S606, aggregate all object descriptions to generate a collection of current environment objects.

[0201] In this embodiment, in order to achieve dynamic task correction in scenarios where the target object is missing, the system needs to determine a candidate object that can semantically replace the original target object from the constructed current environment object set. The key is to establish a stable, scalable object semantic association mechanism with semantic generalization capabilities.

[0202] First, the system extracts the semantic description representation of the target object based on the identifier defined in the task. This representation should not be limited to simple label names or IDs, but should include richer contextual information, such as the object's functional attributes, category semantics, typical uses, and semantic coupling relationships with other entities in the task context. This is usually achieved by using language models (such as BERT, RoBERTa) to transform text identifiers into high-dimensional semantic embedding vectors, so that the vectors themselves carry a structured semantic hierarchy and task context.

[0203] Simultaneously, each candidate object in the current set of environmental objects needs to have a corresponding semantic representation constructed. In addition to embedding category identifiers, the contextual relationship features of objects within the environment can be considered, such as the relative spatial relationship between objects and other objects, the functional label of the area where the object is located, and the frequency of use or replacement records of this type of object in historical tasks. To this end, the system can employ a joint semantic encoding mechanism or graph neural network to model the relationship graph between objects in the current environment and assign a semantic feature vector to each node.

[0204] The system then proceeds to the semantic similarity calculation stage, the core of which lies in measuring the semantic relevance between the target object representation and each candidate object representation in a unified vector space. Here, basic similarity metrics such as cosine similarity, Euclidean distance, or Mahalanobis distance in the vector space can be used. Alternatively, a weighted similarity strategy can be introduced, integrating composite weights from multiple dimensions (such as functional attribute similarity, category semantic similarity, and task suitability) to reflect a more complex matching mechanism. Especially when object categories differ significantly, establishing a connection based solely on surface semantics is insufficient. The system can introduce a knowledge graph-based semantic path reasoning mechanism, leveraging external knowledge bases (such as ConceptNet, WordNet, and industry ontology) to analyze whether there are transformable logical paths between the substitute object and the target object, such as "similar usage," "hierarchical inheritance," or "functional substitution," to support cross-class substitution.

[0205] In addition to static semantic similarity calculation, the system should also consider the immediate needs in the task execution context and dynamically evaluate whether each candidate object is operable and replaceable in the current scenario. For example, in the task of "getting a cup", if the cup is missing, and the system identifies a candidate environment where a water bottle and a bowl coexist, it needs to further determine whether the "water bottle" is closer to the current task goal in terms of semantics and physical reachability. It can even generate a replacement priority ranking by incorporating user preference and historical behavior data.

[0206] Ultimately, the system will select several objects with the strongest semantic association from the candidate set by setting a similarity threshold or ranking mechanism. These objects will serve as potential replacements for the target object, and the one with the highest priority will be selected to perform the task replacement operation. This process supports a redundancy mechanism, meaning that if the first replacement object fails again during execution, the next replacement object can be quickly scheduled to ensure that the task is not interrupted.

[0207] This embodiment constructs a multi-dimensional embedded representation system oriented towards semantic understanding and integrates semantic similarity calculation, semantic path reasoning, and context adjustment mechanisms to achieve object-level dynamic semantic substitution reasoning. This enables the system to make substitution judgments and selections based on the task context and the actual composition of the environment when the target object is missing. This mechanism does not rely on strict matching of object categories and supports flexible substitution capabilities across categories and contexts, effectively avoiding the static failure problem of traditional systems when objects are missing. By introducing task context and historical task behavior data to enhance the practicality and relevance of semantic judgment, the system not only improves the continuity and self-healing ability of tasks but also possesses broad adaptability to complex environments and uncertain scenarios, laying a solid foundation for building intelligent agent systems with generalized execution capabilities.

[0208] In one embodiment, step S80 above includes:

[0209] S801, Locate the target object element in the current subtask;

[0210] S802, replace the target object element with the substitute object, and adjust the association parameters of the current subtask to adapt to the substitute object, and generate a corrected subtask containing the substitute object;

[0211] S803, Load the execution instructions for the modified subtask;

[0212] S804, control the actuator to execute the correction subtask;

[0213] S805, record the execution result of the modified subtask to advance the task plan until the task plan is completed.

[0214] In this embodiment, after detecting that the target object in the original subtask is missing and a replacement object has been successfully matched, the system needs to dynamically update the current subtask to ensure the consistency of the task logic and the continuity of the execution process. This step generates an executable new subtask through replacement object injection and parameter self-adaptation, and fully integrates it with the original task plan.

[0215] First, the system needs to explicitly define the location and binding relationships of the target object elements within the current subtask. This definition includes not only explicit fields (such as object ID or name) but also implicit dependencies such as operation command parameters, path configurations, grab points, and pose angles associated with the object. All parameter dimensions directly bound to the target object are extracted by parsing the execution semantic structure of the current subtask or using parameter parsing templates.

[0216] Subsequently, the system replaces the target object elements with previously determined semantically related substitute objects and performs adaptive adjustments to the original parameter set. This adjustment includes not only entity replacement but also compensation for differences between the substitute object and the original object in terms of geometric dimensions, material properties, grasping feasible areas, and dynamic interaction constraints. For example, if the original task is "grab a ceramic cup," and the substitute object is a "plastic kettle," the system needs to reset execution parameters such as gripping force, pose angle, and path speed to ensure that the actuator behavior is consistent with the physical characteristics of the new object. This adaptation process can be automatically configured using an object knowledge base or a dynamic parameter mapping model.

[0217] After the replacement and parameter adjustment are completed, the system constructs a new subtask instance as the "corrected subtask" and loads it into the task control module. The corrected subtask must retain the context information and dependency parameters related to subsequent processes in the original task to ensure that it can be naturally connected to the original task chain after execution, forming logical continuity.

[0218] Next, the control system schedules the actuators to load the execution instructions for the corrected sub-tasks and performs motion control, such as executing robotic arm path planning, gripper motion control, and visual closed-loop feedback, to ensure that the corrected sub-tasks are actually executed in the current environment.

[0219] After execution, the system records the execution status of the correction subtask, including success or failure, time taken, and environmental feedback information. This information is written to the task execution record database as a task log and used to update the current status index of the task plan. If the completion flag of the correction subtask matches the plan's expectations, the task plan proceeds to the next subtask until the final task is completed.

[0220] Example Description: A 3D surgical navigation system receives a natural language instruction from a surgeon via voice: "Please place the excised tissue sample into a Class B refrigerated container and place it on the right-hand operating table." The system receives this instruction through an embedded microphone and simultaneously acquires real-time visual data of the operating table and its surroundings through binocular cameras mounted on the robot platform.

[0221] The system first parses the voice commands into structured semantic information, identifying the action intent as "placement," the target object as a "Class B refrigerated container," and the operating area as the "right-side operating table." These semantic elements are then encoded into command vectors using a language coding model. Simultaneously, images captured by the camera are pre-processed to identify objects in the environment, such as instrument trays, sample containers, refrigerated boxes, and surgical trays, encoding key areas into visual vectors. The command vectors and visual vectors together constitute a pair of task execution features, used for subsequent case comparison and task decomposition.

[0222] The system retrieves historical task records from the surgical robot's experience case library, including previously executed sample placement and refrigerated container recognition tasks. By calculating the linguistic and semantic similarity and visual scene similarity between the current instruction and historical cases, the system selects the most matching set of operation procedures as a reference case to guide the current execution strategy.

[0223] Next, the system automatically breaks down the instructions into multiple subtasks: Subtask 1 is "locating the Class B refrigerated container", Subtask 2 is "grabbing the target sample", Subtask 3 is "placing the sample into the container", and Subtask 4 is "placing the container on the right-hand operating table". During the decomposition process, the system clearly defines the physical target corresponding to each action and marks its relative position and grasping parameters in the current environment, ensuring that the execution conditions of each subtask can be perceived and planned.

[0224] When executing subtask 1, namely "locating Class B refrigerated containers," the system scans the environment with a camera and extracts the visual regions of all candidate refrigerated containers. The extracted image features are then matched with the semantic embedding vector of the original target "Class B refrigerated container." The system finds that the object has been temporarily removed from the environment by a nurse, and the similarity is below a set threshold, leading the system to determine that the target object is missing.

[0225] At this point, the system automatically triggers an environment rescan process to acquire the current image data and extract the category and location information of all container objects in the scene, constructing an environment object set. Through semantic similarity calculation, the system identifies a "Class B refrigerated container substitute"—a refrigerated container with similar labels, the same functional attributes, but slightly different shapes.

[0226] The system updates the replacement object as the new execution target of subtask 1 and adjusts path planning parameters, such as gripping point position, container grasping posture, and surface anti-slip force settings, to adapt to the object's size and material properties, and regenerates the corrected subtask 1. The robot loads the corrected subtask execution instructions and successfully completes the operation.

[0227] Subsequent subtasks are executed sequentially, and the system records the execution results of each subtask, ultimately completing the entire sample collection and placement process. Throughout the task, the robot maintains a closed-loop interaction with environmental perception data, dynamically adapting to changes in the objects to ensure the continuous, stable, and accurate completion of the medical task.

[0228] In the business processing area of ​​a bank branch, the intelligent counter collaboration robot receives a natural language task instruction from the back-end: "Deposit today's loan approval documents into the Class B voucher cabinet." The robot receives the task instruction through a voice analysis module, and at the same time uses a front-end camera to perform visual perception and collection of the current working environment, covering visual elements such as the counter, desktop documents, and surrounding voucher cabinets.

[0229] First, the system performs semantic parsing on the task instruction, identifying the core action as "deposit," the target object as "Class B voucher cabinet," and the operation object as "today's loan approval documents." The parsed linguistic information is embedded into a language encoding model to generate an instruction embedding vector. Simultaneously, the environmental images captured by the camera system are processed to extract key areas on the desktop, such as paper documents, folders, and voucher cabinets, and encoded into an environment embedding vector using a visual model. These instruction and environmental features together constitute multimodal input features, used to drive subsequent task retrieval and decomposition.

[0230] The system accesses the banking business robot task library, retrieves historical experience cases such as "document archiving", "counter recognition" and "multi-target sorting", and calculates the comprehensive similarity by comparing the language and visual features of the current task with those of the historical cases. Several highly matched reference task processes are then selected as action templates for this execution.

[0231] Next, the system breaks down the task instructions into a sequence of subtasks: Subtask 1 is "Identify today's loan approval documents", Subtask 2 is "Find the Class B voucher cabinet", Subtask 3 is "Retrieve the target file", and Subtask 4 is "Archive to the target cabinet". Each subtask specifies the operation, target object, and constraint environment variables, such as the relative position of the target file, the open / closed state of the cabinet, and the file placement method.

[0232] When executing subtask 2, "Find the Class B voucher cabinet," the robot performs a visual scan of the current environment and attempts to match the visual features of the voucher cabinet. Due to personnel changes, the cabinet is not currently placed in its original designated area. The system does not find an object with a matching degree exceeding the threshold in the visual semantic space, and therefore determines that "the target object is missing."

[0233] The system then triggers a rescan command to re-acquire current visual data, identify all cabinet-type objects in the scene, extract their semantic tags (such as "Class A voucher cabinet" and "Class B file box") and location information, and construct a set of objects in the current environment. Then, based on the similarity with "Class B voucher cabinet" in semantic fields such as cabinet number, purpose description, and color code, "Class B file box" is selected from the set as the optimal replacement object.

[0234] The system then updates the current subtask 2 with the replacement object, replacing the original target object field with "Class B File Box", and updating motion parameters such as the grasping path, placement depth, and cabinet door opening angle, generating a corrected subtask. The robot loads the corrected subtask, successfully places the target file into the replacement container, and records the task completion status.

[0235] Subsequent subtasks continued to be executed in logical order until completion. The entire task process was successfully completed despite missing objectives, dynamic corrections, and path adjustments, ensuring the continuous progress of the task plan.

[0236] This embodiment generates corrected subtasks by semantically associating substitute objects and performs adaptive parameter adjustment and instruction-level dynamic loading. This enables the system to maintain the continuity of the task plan and the integrity of the execution path even when the target object is missing, avoiding task failure due to abnormal interruptions. Compared to traditional methods that rely on static plan execution, this method has stronger real-time adaptability and object-level flexible control capabilities, improving the agent's ability to continuously complete tasks in complex dynamic environments and enhancing its robustness and environmental adaptability. By constructing a local self-healing structure for the task plan, the system can achieve dynamic repair and re-execution of the task plan, improving the stability and success rate of the overall task flow.

[0237] In one embodiment, a task planning and correction apparatus is provided, which corresponds one-to-one with the task planning and correction methods described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the task planning and correction device of the present invention. The modules include a task parsing module 10, a case retrieval module 20, a task decomposition module 30, a plan generation module 40, a target detection module 50, an environment construction module 60, an alternative reasoning module 70, and a task correction module 80. Detailed descriptions of each functional module are as follows:

[0238] The task parsing module 10 is used to acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and the environmental perception data, respectively.

[0239] Case retrieval module 20 is used to retrieve reference cases from the experience case library based on the instruction features and the environmental features;

[0240] Task decomposition module 30 is used to decompose the task instruction into a sequence of subtasks containing target objects to be operated;

[0241] The plan generation module 40 is used to generate a task plan based on the reference case and the sub-task sequence;

[0242] The target detection module 50 is used to execute the current subtask in the task plan and detect whether the target object of the current subtask is missing during the execution of the current subtask.

[0243] Environment building module 60 is used to obtain objects in the current environment to form a current environment object set when the target object of the current subtask is detected to be missing.

[0244] Alternative reasoning module 70 is used to determine alternative objects semantically associated with the target object in the current set of environment objects;

[0245] The task correction module 80 is used to update the current subtask using the replacement object to generate a corrected subtask, and execute the corrected subtask to complete the task plan.

[0246] In one embodiment, the task parsing module 10 is specifically used for:

[0247] It receives input natural language instructions as task instructions and acquires environmental perception data through image acquisition devices;

[0248] The task instructions are semantically parsed to extract task action elements and task object elements;

[0249] The task action elements and task object elements are input into the language processing model to generate text embedding vectors, and the text embedding vectors are used as instruction features.

[0250] The environmental perception data is preprocessed to extract key visual regions;

[0251] The key visual regions are input into the visual processing model to generate visual embedding vectors, and these visual embedding vectors are used as environmental features.

[0252] In one embodiment, the case retrieval module 20 is specifically used for:

[0253] Extract the text embedding vector of the instruction features and extract the visual embedding vector of the environment features;

[0254] Obtain the textual and visual features of historical cases from the experience case library;

[0255] Determine the text similarity between the text embedding vector and the case text features;

[0256] Determine the visual similarity between the visual embedding vector and the case visual features;

[0257] The textual similarity and visual similarity are combined to generate a comprehensive similarity score;

[0258] Historical cases are sorted based on their overall similarity, and historical cases whose overall similarity ranking is above a preset threshold are selected as reference cases.

[0259] In one embodiment, the task decomposition module 30 is specifically used for:

[0260] Parse the semantic structure of the task instruction and identify the core action elements in the semantic structure;

[0261] Extract the objects that act on the core action elements as the target objects to be operated on;

[0262] The associated container of the target object to be operated on is determined based on environmental constraints;

[0263] The core action elements, the target object to be operated on, and the associated containers are combined to form a sub-task unit;

[0264] The subtask units are arranged in the logical order of execution to form a subtask sequence.

[0265] In one embodiment, the target detection module 50 is specifically used for:

[0266] Parse the action instructions and target object identifiers of the current subtask in the task plan;

[0267] The execution of the action command is initiated, and environmental awareness data streams are acquired in real time during the execution process;

[0268] Segment the candidate object regions in the environmental perception data stream and extract the feature vectors of the candidate object regions;

[0269] In the semantic feature space, the similarity between the feature vector and the semantic description vector of the target object identifier is determined;

[0270] When the similarity exceeds the matching threshold, it is determined that the target object is not missing;

[0271] If the similarity does not exceed the matching threshold, the target object is determined to be missing.

[0272] In one embodiment, the environment building module 60 is specifically used for:

[0273] When the target object of the current subtask is detected to be missing, an environment rescan instruction is triggered;

[0274] Real-time environmental perception data is reacquired based on the aforementioned environmental rescan instruction;

[0275] Identify all visible objects in the real-time environmental perception data;

[0276] Extract the object category identifier and location information for each visible object;

[0277] The object category identifier and location information are combined to form an object description item;

[0278] Aggregate all object descriptions to generate a collection of current environment objects.

[0279] In one embodiment, the task correction module 80 is specifically used for:

[0280] Locate the target object element in the current subtask;

[0281] Replace the target object element with the substitute object, and adjust the association parameters of the current subtask to adapt to the substitute object, thereby generating a corrected subtask containing the substitute object;

[0282] Load the execution instructions for the modified subtask;

[0283] The actuator is controlled to execute the modified subtask;

[0284] Record the execution results of the modified subtasks to advance the task plan until the task plan is completed.

[0285] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements a task planning and correction method on the server side, performing server-side functions or steps.

[0286] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements a task planning and correction method, or related functions or steps on the user side.

[0287] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0288] Acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and environmental perception data, respectively;

[0289] Based on the instruction characteristics and the environmental characteristics, retrieve reference cases from the experience case library;

[0290] The task instruction is decomposed into a sequence of subtasks containing the target objects to be operated on;

[0291] Based on the reference case and the sub-task sequence, a task plan is generated;

[0292] Execute the current subtask in the task plan, and during the execution of the current subtask, detect whether the target object of the current subtask is missing;

[0293] When the target object of the current subtask is detected to be missing, the objects in the current environment are obtained to form a current environment object set.

[0294] Determine a substitute object semantically associated with the target object from the current set of environment objects;

[0295] The current subtask is updated using the replacement object to generate a corrected subtask, and the corrected subtask is executed to complete the task plan.

[0296] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0297] Acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and environmental perception data, respectively;

[0298] Based on the instruction characteristics and the environmental characteristics, retrieve reference cases from the experience case library;

[0299] The task instruction is decomposed into a sequence of subtasks containing the target objects to be operated on;

[0300] Based on the reference case and the sub-task sequence, a task plan is generated;

[0301] Execute the current subtask in the task plan, and during the execution of the current subtask, detect whether the target object of the current subtask is missing;

[0302] When the target object of the current subtask is detected to be missing, the objects in the current environment are obtained to form a current environment object set.

[0303] Determine a substitute object semantically associated with the target object from the current set of environment objects;

[0304] The current subtask is updated using the replacement object to generate a corrected subtask, and the corrected subtask is executed to complete the task plan.

[0305] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0306] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0307] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0308] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A task planning and correction method, characterized in that, Includes the following steps: Acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and environmental perception data, respectively; Based on the instruction characteristics and the environmental characteristics, retrieve reference cases from the experience case library; The task instruction is decomposed into a sequence of subtasks containing the target objects to be operated on; Based on the reference case and the sub-task sequence, a task plan is generated; Execute the current subtask in the task plan, and during the execution of the current subtask, detect whether the target object of the current subtask is missing; When the target object of the current subtask is detected to be missing, the objects in the current environment are obtained to form a current environment object set. Determine a substitute object semantically associated with the target object from the current set of environment objects; The current subtask is updated using the replacement object to generate a corrected subtask, and the corrected subtask is executed to complete the task plan.

2. The task planning and correction method as described in claim 1, characterized in that, Acquire task instructions and environmental awareness data, and generate instruction features and environmental features based on the task instructions and environmental awareness data, respectively, including: It receives input natural language instructions as task instructions and acquires environmental perception data through image acquisition devices; The task instructions are semantically parsed to extract task action elements and task object elements; The task action elements and task object elements are input into the language processing model to generate text embedding vectors, and the text embedding vectors are used as instruction features. The environmental perception data is preprocessed to extract key visual regions; The key visual regions are input into the visual processing model to generate visual embedding vectors, and these visual embedding vectors are used as environmental features.

3. The task planning and correction method as described in claim 1, characterized in that, Based on the instruction characteristics and the environmental characteristics, reference cases are retrieved from the experience case library, including: Extract the text embedding vector of the instruction features and extract the visual embedding vector of the environment features; Obtain the textual and visual features of historical cases from the experience case library; Determine the text similarity between the text embedding vector and the case text features; Determine the visual similarity between the visual embedding vector and the case visual features; The textual similarity and visual similarity are combined to generate a comprehensive similarity score; Historical cases are sorted based on their overall similarity, and historical cases whose overall similarity ranking is above a preset threshold are selected as reference cases.

4. The task planning and correction method as described in claim 1, characterized in that, The task instruction is decomposed into a sequence of subtasks containing the target object to be operated on, including: Parse the semantic structure of the task instruction and identify the core action elements in the semantic structure; Extract the objects that act on the core action elements as the target objects to be operated on; The associated container of the target object to be operated on is determined based on environmental constraints; The core action elements, the target object to be operated on, and the associated containers are combined to form a sub-task unit; The subtask units are arranged in the logical order of execution to form a subtask sequence.

5. The task planning and correction method as described in claim 1, characterized in that, Execute the current subtask in the task plan, and during the execution of the current subtask, detect whether the target object of the current subtask is missing, including: Parse the action instructions and target object identifiers of the current subtask in the task plan; The execution of the action command is initiated, and environmental awareness data streams are acquired in real time during the execution process; Segment the candidate object regions in the environment-aware data stream and extract the feature vectors of the candidate object regions; In the semantic feature space, the similarity between the feature vector and the semantic description vector of the target object identifier is determined; When the similarity exceeds the matching threshold, it is determined that the target object is not missing; If the similarity does not exceed the matching threshold, the target object is determined to be missing.

6. The task planning and correction method as described in claim 1, characterized in that, When the target object of the current subtask is detected to be missing, the objects in the current environment are obtained to form a current environment object set, including: When the target object of the current subtask is detected to be missing, an environment rescan instruction is triggered; Real-time environmental perception data is reacquired based on the aforementioned environmental rescan instruction; Identify all visible objects in the real-time environmental perception data; Extract the object category identifier and location information for each visible object; The object category identifier and location information are combined to form an object description item; Aggregate all object descriptions to generate a collection of current environment objects.

7. The task planning and correction method as described in claim 1, characterized in that, Updating the current subtask using the replacement object to generate a corrected subtask, and executing the corrected subtask to complete the task plan, includes: Locate the target object element in the current subtask; Replace the target object element with the substitute object, and adjust the association parameters of the current subtask to adapt to the substitute object, thereby generating a corrected subtask containing the substitute object; Load the execution instructions for the modified subtask; The actuator is controlled to execute the modified subtask; Record the execution results of the modified subtasks to advance the task plan until the task plan is completed.

8. A task planning and correction device, characterized in that, The task planning and correction device includes: The task parsing module is used to acquire task instructions and environmental perception data, and generate instruction features and environmental features based on the task instructions and environmental perception data, respectively. The case retrieval module is used to retrieve reference cases from the experience case library based on the instruction features and the environmental features. The task decomposition module is used to decompose the task instruction into a sequence of subtasks containing target objects to be operated on; The plan generation module is used to generate a task plan based on the reference case and the sub-task sequence; The target detection module is used to execute the current subtask in the task plan and detect whether the target object of the current subtask is missing during the execution of the current subtask; The environment building module is used to obtain objects in the current environment to form a current environment object set when the target object of the current subtask is detected to be missing. An alternative reasoning module is used to determine, within the current set of environment objects, an alternative object semantically associated with the target object; The task correction module is used to update the current subtask using the replacement object to generate a corrected subtask, and execute the corrected subtask to complete the task plan.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a task planning and correction program stored in the memory and executable on the processor, wherein the task planning and correction program, when executed by the processor, implements the steps of the task planning and correction method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a task planning and correction program, which, when executed by a processor, implements the steps of the task planning and correction method as described in any one of claims 1-7.

Images