Control method and device based on task understanding representation, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as personal intelligence, financial science and technology and medical health, and discloses a control method, device and equipment based on task understanding characterization and a medium, and the method comprises the steps: obtaining an environment image of a to-be-processed scene and a task instruction for specifying an operation task, a visual encoder is utilized to process an environment image to generate a visual feature vector, a language encoder is utilized to process a task instruction to generate a semantic representation, the visual feature vector and the semantic representation are fused to obtain a fused feature, and the fused feature is input into a pre-training model to generate a task understanding representation. And generating an action sequence by using an action decoder based on a diffusion model and a flow matching technology, and controlling an execution device to execute an operation according to the action sequence. According to the method, multi-modal information is fused, a task understanding mechanism is introduced, a high-reliability action sequence is generated in combination with a diffusion model and a flow matching technology, and the response ability to variable task instructions and the operation accuracy can be improved in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a control method, apparatus, device, and storage medium based on task understanding representation. Background Technology

[0002] In existing technologies, research on robotic arm control and object grasping has been widely applied in various fields such as industrial production, healthcare, and fintech. However, the operational capabilities of fixed robotic arms in complex environments still have many limitations, particularly in terms of generalization ability, visual cognition, and multi-tasking adaptability, making it difficult to meet the needs of practical applications.

[0003] In the field of embodied intelligence, although robotic arms have achieved a certain degree of autonomous operation in constrained environments through methods such as deep learning and imitation learning, the generalization and task understanding capabilities of traditional methods still have significant shortcomings when facing unknown tasks or entirely new environments. Existing strategies often require a large amount of targeted training data and are prone to combinatorial explosion, failing to effectively cope with the complex operational needs of multiple tasks, multiple objects, and multiple states. At the same time, visual perception modules mostly rely on simplified background environments and lack the ability to acquire structured visual information in complex and dynamic environments, further restricting the adaptability of robotic arms to complex task scenarios.

[0004] In the fintech sector, intelligent robotic arms are increasingly being applied to scenarios such as physical teller counters and automated equipment maintenance, including the automated operation of components within cash deposit and withdrawal equipment and the intelligent sorting and handling of valuables. However, most existing robotic arm operation strategies are designed based on closed, structured working environments, making them unsuitable for real-world applications with varying physical layouts and frequently changing task requirements. Especially when faced with dynamic teller layouts or unexpected interference, traditional strategies suffer from insufficient generalization capabilities, resulting in poor system stability and weak task adaptability in new environments, hindering their ability to flexibly respond to the high-frequency, customized operational needs of financial services.

[0005] In the healthcare field, robotic arms are widely used in tasks such as surgical assistance, medical equipment operation, and rehabilitation training. These scenarios typically involve unstructured environmental contexts and diverse individual patient differences. Existing technologies lack effective generalization control capabilities when dealing with different patient body shapes, postures, or sudden disturbances, resulting in poor operational stability and weak risk control capabilities. Especially in complex surgical assistance or personalized rehabilitation training, robotic arms struggle to accurately understand and execute changing task instructions, limiting their safety and application efficiency in healthcare scenarios. Summary of the Invention

[0006] The main objective of this invention is to provide a control method, device, equipment, and storage medium based on task understanding representation, aiming to solve the technical problems of poor generalization ability, limited visual cognition, and inaccurate action generation in existing machine control technologies when facing new scenarios, new objects, or new task instructions.

[0007] To achieve the above objectives, the present invention provides a control method based on task understanding representation, comprising:

[0008] Acquire environmental images of the scene to be processed and task instructions for specifying the operation tasks;

[0009] The environment image is processed by a visual encoder to generate a visual feature vector, and the task instruction is processed by a language encoder to generate a semantic representation.

[0010] The visual feature vector and the semantic representation are fused to obtain fused features, and the fused features are input into a pre-trained model for processing to generate task understanding representations;

[0011] The task understanding representation is processed using an action decoder based on a diffusion model and flow matching technology to generate an action sequence;

[0012] The actuator is controlled to perform operations according to the action sequence.

[0013] Furthermore, to achieve the above objectives, the present invention provides a control device based on task understanding representation, comprising:

[0014] The perception input module is used to acquire environmental images of the scene to be processed and task instructions for specifying the operation task.

[0015] The feature extraction module is used to process the environmental image through a visual encoder to generate a visual feature vector, and to process the task instruction through a language encoder to generate a semantic representation.

[0016] The multimodal fusion module is used to fuse the visual feature vector and the semantic representation to obtain fused features, and input the fused features into a pre-trained model for processing to generate task understanding representations;

[0017] The action generation module is used to process the task understanding representation using an action decoder based on a diffusion model and flow matching technology to generate action sequences;

[0018] The execution control module is used to control the execution device to perform operations according to the action sequence.

[0019] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a task-understanding representation-based control program stored in the memory and executable on the processor, wherein when the task-understanding representation-based control program is executed by the processor, it implements the steps of the task-understanding representation-based control method as described above.

[0020] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a control program based on task understanding representation, wherein the control program based on task understanding representation, when executed by a processor, implements the steps of the control method based on task understanding representation as described above.

[0021] Beneficial Effects: This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as embodied intelligence, fintech, and healthcare. It discloses a control method, device, equipment, and medium based on task understanding representation, comprising: acquiring an environmental image of the scene to be processed and a task instruction for specifying the operation task; processing the environmental image through a visual encoder to generate a visual feature vector; processing the task instruction through a language encoder to generate a semantic representation; fusing the visual feature vector and the semantic representation to obtain a fused feature; inputting the fused feature into a pre-trained model for processing to generate a task understanding representation; using an action decoder based on a diffusion model and flow matching technology to process the task understanding representation to generate an action sequence; and controlling an execution device to perform operations based on the action sequence. This invention, by fusing multimodal information and introducing a task understanding mechanism, combines a diffusion model and flow matching technology to generate a highly reliable action sequence, effectively improving the robotic arm's responsiveness and operational accuracy to variable task instructions in complex environments. Attached Figure Description

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0023] Figure 1 This is a schematic diagram of an application environment for a control method based on task understanding and representation according to an embodiment of the present invention;

[0024] Figure 2 This is a flowchart illustrating an embodiment of the control method based on task understanding and representation of the present invention;

[0025] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the control device based on task understanding and representation of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0027] Figure 5This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0028] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0029] The control method based on task understanding representation provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can obtain environmental images of the scene to be processed and task instructions for the specified operation task from the user terminal. It processes the environmental images using a visual encoder to generate visual feature vectors, and processes the task instructions using a language encoder to generate semantic representations. The visual feature vectors and semantic representations are fused to obtain fused features. These fused features are input into a pre-trained model for processing to generate task understanding representations. An action decoder based on a diffusion model and flow matching technology processes the task understanding representations to generate action sequences. The execution device is then controlled to perform operations based on these action sequences. This invention, by fusing multimodal information and introducing a task understanding mechanism, combines a diffusion model and flow matching technology to generate highly reliable action sequences, effectively improving the robotic arm's responsiveness and operational accuracy to variable task instructions in complex environments. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this invention.

[0030] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the control method based on task understanding representation provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0031] like Figure 2 As shown, the control method based on task understanding representation proposed in this invention includes the following steps:

[0032] S10, acquire the environmental image of the scene to be processed and the task instructions for specifying the operation task;

[0033] In this embodiment, acquiring environmental images of the scene to be processed refers to using an imaging device to collect information about the physical space where the robotic arm will perform its operation, forming image data that can be analyzed by the subsequent vision processing system. Environmental images include, but are not limited to, visible light images, infrared images, depth images, multispectral images, or composite images acquired through multimodal imaging devices. The type of image can be flexibly selected based on different application environments, lighting conditions, and the physical properties of the target object. During image acquisition, the device can be fixedly installed at the end of the robotic arm, around the workspace, or used to cover the space via a mobile platform. The image data should include spatial structure information, background environment information, operation object information, and information on potential interference factors in the scene, facilitating the subsequent processing module to extract effective information for task understanding.

[0034] Obtaining task instructions for a specified operation refers to generating instruction information describing the task objectives that the robotic arm is about to perform, based on the human-machine interface, system scheduling module, or a pre-set task library. Task instructions can be presented in the form of natural language text, voice instructions, graphical instructions, coded instructions, or structured instruction sets. They can be obtained through voice recognition modules, text input modules, gesture recognition modules, image recognition modules, or other sensor signal acquisition devices. Task instructions should include information such as the operation objective, operation method, expected result, or spatial location to ensure that the subsequent language processing module can accurately extract the task semantics. The generation of task instructions can be dynamically adjusted based on the real-time status of the operation scenario, historical task records, or system inference results to ensure the matching and applicability of the instruction content to the scenario environment.

[0035] In the process of acquiring environmental images and task instructions, it is usually necessary to correlate their temporal dimensions to ensure that the image data and task instructions correspond to the same spatial state, avoiding misunderstandings caused by changes in scene state. To this end, the system can generate timestamps for both image data and task instructions, and establish a correspondence between them through synchronization or buffering mechanisms. Synchronization mechanisms can be implemented using timestamp matching, frame rate adjustment, or buffer alignment to ensure coordinated input of visual and instruction information.

[0036] Depending on the application environment, environmental images can be acquired in various ways. For example, under low-light or complex lighting conditions, multispectral imaging equipment can be used to acquire images containing infrared channel information, enhancing the ability to identify objects with weak features or low-contrast backgrounds. Depth cameras can also be combined to acquire spatial structure information, aiding in the understanding of three-dimensional spatial relationships. Before image acquisition, image preprocessing operations such as automatic exposure adjustment, white balance calibration, and distortion correction can be performed to improve image data quality.

[0037] During the acquisition of task instructions, voice commands can be received through a natural language interface and converted into text by a speech recognition module; alternatively, structured task parameters can be input through a graphical user interface, or the intention to operate can be obtained through a gesture recognition device. The expression structure of task instructions can adopt standardized grammatical templates to reduce the difficulty of language understanding. For example, task instructions can include elements such as operation verbs, target objects, and target locations, forming a complete and parsable instruction expression.

[0038] During the synchronized processing of environmental images and task commands, a unified time base can be set. Image acquisition devices and command acquisition devices record data according to a unified timestamp. The system matches the corresponding image and command pairs based on the timestamps to ensure consistency in the scene states reflected by both. For multi-frame image data or continuous task commands, the system can dynamically maintain the effective matching relationship of the data through a cache queue or sliding window mechanism.

[0039] Example: In the healthcare field, multispectral imaging equipment can acquire images of the operating room environment. The image data includes information on the operating table surface, the location of medical instruments, and the patient's body parts. Combined with the doctor's voice input of operation task instructions, such as "hand the instrument to the assistant's position," the robotic arm can assist in delivering and operating instruments during the operation. The synchronous acquisition of environmental images and task instructions ensures that the operation behavior is consistent with the actual scene, reduces operational risks, improves operational efficiency and accuracy, and meets the high precision and high safety requirements of medical operations.

[0040] In the fintech business, the system can acquire environmental images of the financial counter area through camera equipment. The images include the teller's location, document location, and customer dynamic information. Combined with task instructions input through a graphical interface, such as "transfer contract documents to the customer's signing area", the system enables the robotic arm to deliver intelligent documents, ensuring the accuracy of delivery of sensitive documents and the security of the process. The real-time synchronization of environmental images and task instructions helps the robotic arm flexibly adjust its operating strategies according to changes in the scenario, improving the efficiency and standardization of financial services and meeting the requirements for controllability and compliance in the transmission of confidential information.

[0041] In the field of embodied intelligence, the system acquires environmental images of the operating space through multimodal imaging devices installed at the end of a robotic arm or in the environment. The image data covers three-dimensional structure, object position, and environmental dynamic information. Combined with the user's operation task instructions input by voice, gesture, or natural language, such as "move the object on the table to the designated position," the system enables the robotic arm to operate flexibly in open scenarios. The environmental images and task instructions are synchronized in time, ensuring that the scene state understood by the robotic arm is highly consistent with the actual environment. This improves the adaptability to dynamic environments and the accuracy of executing unknown tasks, and promotes the development of embodied intelligence systems towards multi-tasking and high-complexity operations.

[0042] This embodiment achieves accurate synchronization of operational scene information and operational requirements by acquiring environmental images and task instructions, avoiding misunderstandings caused by information mismatch and improving the adaptability and accuracy of the robotic arm in complex environments. Diverse image acquisition methods enhance the richness and robustness of visual information, while multiple task instruction input methods improve the system's ability to express different operational needs. Time synchronization between environmental images and task instructions ensures consistency of spatial states in the information processing flow, comprehensively improving the stability and reliability of the robotic arm in performing operational tasks.

[0043] S20, the environment image is processed by a visual encoder to generate a visual feature vector, and the task instruction is processed by a language encoder to generate a semantic representation;

[0044] In this embodiment, after acquiring the environmental image, feature extraction processing is first performed on the environmental image using a visual encoder. The visual encoder may include a convolutional neural network structure, a Transformer structure, or other neural network modules with spatial information processing capabilities. The environmental image is typically two-dimensional or three-dimensional image data, and its source can be visible light imaging equipment, infrared imaging equipment, depth cameras, or LiDAR systems. The specific image type and device form selected are flexibly determined according to the application scenario. The visual encoder gradually extracts visual information, including scene structure, object contours, color information, and spatial positional relationships, through multi-level image processing, and encodes this information into fixed-dimensional visual feature vectors. Visual feature vectors are high-dimensional vectors expressed in numerical form, possessing the ability to structurally and compactly represent environmental scene information, facilitating subsequent fusion and task understanding processing.

[0045] Simultaneously, a language encoder extracts semantic information from task instructions. Task instructions can originate from user-input natural language commands, structured text, speech recognition transcription results, or other text data. The language encoder's structure can employ word embedding layers, sequence encoding networks, and contextual semantic modeling modules. The language encoder first maps words or phrases in the task instructions into numerical word vectors. Through the encoding process, it captures semantic information such as the grammatical structure, contextual relationships, operational goals, and action logic within the instructions, and outputs a semantic representation in a unified format. This semantic representation, as structured vector data, preserves key intent information from the task instructions, facilitating subsequent fusion and reasoning with visual information.

[0046] Both visual feature vectors and semantic representations adopt standardized data formats to ensure that they maintain structural consistency and clear information expression during subsequent fusion, avoiding the impact of format incompatibility on the fusion effect. The entire processing flow strictly guarantees the independence and collaboration of the visual encoder and the language encoder. The visual encoder focuses on acquiring and expressing environmental scene information, while the language encoder focuses on understanding task logic and intent information. Together, they provide the system with multimodal perceptual input capabilities.

[0047] In practical applications, visual encoders can choose deep convolutional networks, such as ResNet and EfficientNet, or they can use a visual Transformer architecture to extract local and global visual features layer by layer, generating visual feature vectors with dimensions of 512, 1024, or 2048. Environmental images can originate from fixed surveillance cameras, vision modules on mobile robots, or vision systems mounted on the end effector of robotic arms. The image type can be RGB images, RGB-D images, or 3D point cloud data, specifically matched to the requirements of the scenario.

[0048] The language encoder can use Transformer-based natural language understanding models, such as BERT, RoBERTa, and the GPT series, or it can adopt a lightweight semantic encoding structure combined with an attention mechanism, flexibly adapting to the length of instructions and expression habits. Task instructions can be obtained through keyboard input, speech-to-text conversion, graphical interface selection, etc. The semantic representation dimension output by the language encoder corresponds to the visual feature vector for subsequent fusion.

[0049] The granularity of visual feature extraction can also be adjusted according to different task complexities. For example, in environments with high operational precision requirements, the ability to express visual features can be enhanced through multi-scale feature fusion technology. At the language understanding level, a context caching mechanism can be introduced to improve the semantic coherence and stability in the execution of continuous tasks.

[0050] Example: In the healthcare business, visual encoders can process real-time images in the operating room and extract environmental information such as medical instruments, operating tables, and patient body positions. Language encoders, on the other hand, parse doctors' natural language instructions, such as "pass the scalpel to the assistant on the left," converting the operational intent into a computable semantic expression. This ensures that the robotic arm accurately performs delivery and operation tasks according to the environmental conditions during assisted operations, reducing the risks of medical operations and improving the intelligence level of the auxiliary system.

[0051] In the fintech business, visual encoders acquire environmental images of financial service outlets, including teller locations, customer dynamics, and document placement information, while language encoders parse operation instructions generated by the business system, such as "transfer the document to the customer's signature area," enabling robotic arms to operate accurately in sensitive scenarios, ensuring the compliance and efficiency of service processes, and meeting the stringent requirements of financial businesses for operational security and information confidentiality.

[0052] In the field of embodied intelligence, visual encoders extract complex visual information from open environments, such as object categories, spatial layouts, and the positions of dynamic obstacles, while language encoders parse natural language commands input by users through voice, text, or multimodal interaction, such as "move the water glass on the table to the right side of the bookshelf." Multimodal fusion ensures that the robotic arm accurately understands task requirements in unstructured environments, enhances the system's generalization capabilities in complex dynamic environments, and promotes the development of embodied intelligence systems towards high autonomy and multi-task execution.

[0053] This embodiment transforms environmental images into visual feature vectors and task instructions into semantic representations, enabling the system to simultaneously perceive the operating environment and understand the user's task intent. The combined use of a visual encoder and a language encoder solves the problem of incomplete task understanding under a single perception mode, improving the accuracy and consistency of multimodal information expression and providing reliable basic data input for subsequent task reasoning and operational control. The overall process ensures the synergistic effect of environmental perception and task understanding, effectively enhancing the system's stability and adaptability in performing complex tasks in dynamic environments.

[0054] S30, the visual feature vector and the semantic representation are fused to obtain fused features, and the fused features are input into a pre-trained model for processing to generate a task understanding representation;

[0055] In this embodiment, after obtaining the visual feature vector and semantic representation, an information fusion operation is first performed on the two. The visual feature vector is extracted by a visual encoder and is typically a fixed-length high-dimensional vector that expresses comprehensive information such as spatial structure, object category, positional relationship, and visual semantics in the environmental image. The semantic representation is a vector structure generated by processing task instructions through a language encoder, containing key semantic information such as action requirements, operation goals, and intent logic in the user instructions. Both are structured numerical vectors, possessing the basic conditions for direct calculation and fusion.

[0056] Information fusion can be based on feature concatenation, attention mechanisms, interactive networks, or other multimodal fusion methods. By concatenating visual feature vectors and semantic representations along the feature dimension, a unified format of fused features is formed. These fused features retain both environmental and task intent information, possessing the ability to simultaneously express visual and linguistic information. Furthermore, cross-modal interaction structures can be introduced, using bidirectional attention mechanisms or gating control modules to further strengthen the correlation between visual and linguistic information, thereby improving the expressive effect of the fused features and the collaborative understanding of multimodal information.

[0057] After the fused features are formed, they are input into a pre-trained model for comprehensive processing. Pre-trained models typically possess a multi-layered deep neural network structure, including embedding layers, cross-attention layers, feedforward neural networks, and normalization layers, exhibiting powerful feature representation and reasoning capabilities. These pre-trained models can be trained on large-scale multimodal data, possessing rich visual, linguistic, and task logic modeling capabilities. After inputting fused features, they can further extract hidden information, enhance semantic associations, and reason about task logic relationships, ultimately outputting a structured task understanding representation.

[0058] Task understanding representations are high-dimensional vectors output by pre-trained models, expressing the system's comprehensive understanding of the current operating environment and task instructions. They are characterized by clear semantics, complete structure, and sufficient information expression, facilitating their use in subsequent action generation and operation control. The overall process ensures that visual information, linguistic information, and logical relationships are fully integrated during multimodal fusion, enhancing the system's ability to perceive and understand complex tasks.

[0059] In practical applications, the fusion process can generate fused features by directly concatenating visual feature vectors and semantic representations according to their dimensions, or by using a cross-modal attention structure to calculate the influence of visual information on linguistic information and the regulation of visual information by linguistic information, thus forming more expressive fused features.

[0060] The pre-trained model can be a visual-language fusion model, such as a multimodal Transformer structure. Internally, the model incorporates multi-head cross-attention layers to enhance the deep fusion and interactive expression of visual and linguistic information. The embedding layer performs initial mapping of the fused features, adjusting feature dimensions and distribution. The cross-attention layers capture long-distance dependencies and contextual semantic logic within the multimodal information. A feedforward neural network further enhances feature representation capabilities and non-linear modeling effects. The final output task understanding representation exhibits good semantic clarity and multimodal consistency.

[0061] The complexity and model size of the fusion structure can also be adjusted according to different scenario requirements. For example, in resource-constrained environments, a lightweight multimodal fusion structure can be selected to ensure a balance between system response speed and resource consumption. In high-precision tasks, a deep multimodal interaction module can be configured to improve the system's understanding accuracy and operational reliability.

[0062] Example: In the healthcare business field, visual feature vectors can express information about the operating room environment, the location of medical devices, and the patient's physical condition. Semantic representations reflect the operational instructions and task intentions given by doctors. After the two are fused, they are input into a pre-trained model, and the output task understanding representation covers the environmental state and operational logic. This helps the system to efficiently and accurately understand the task requirements in complex medical scenarios, and improve the intelligence level and operational safety of robotic arms in surgical assistance, medical supply delivery, and other aspects.

[0063] In the field of fintech business, visual feature vectors express information such as the environment of financial service outlets, the layout of teller windows, and customer dynamics. Semantic representations convey system instructions and operational goals. After being processed by a pre-trained model with integrated feature inputs, the task understanding representation has a clear ability to express operational logic, ensuring that the robotic arm can accurately understand the task content and execution process in sensitive financial service scenarios, thereby improving service efficiency and information security.

[0064] In the field of embodied intelligence, visual feature vectors reflect the spatial structure, object attributes, and dynamic changes in the robot's environment. Semantic representations express complex task instructions given by users through voice, text, and other means. By integrating feature inputs into pre-trained models, task understanding representations comprehensively express environmental and task information, enabling embodied intelligence systems to perform multi-task operations in unstructured environments. This enhances the system's generalization ability and adaptability to complex environments, achieving a higher level of autonomous operation and human-machine collaboration.

[0065] This embodiment effectively integrates environmental and task intent information by fusing visual feature vectors and semantic representations, forming structured fused features that ensure the synergistic effect of multimodal information under a unified expression. The fused features are then input into a pre-trained model for further processing, enhancing the deep understanding and logical reasoning capabilities of multimodal information. The resulting task understanding representation accurately expresses the environmental state, task requirements, and operational logic, providing a complete, clear, and reliable data foundation for subsequent action generation. This solves the problems of incomplete task understanding and information gaps in traditional single-perception modes, improving the system's task adaptability and operational efficiency in complex dynamic environments.

[0066] S40, The task understanding representation is processed using an action decoder based on a diffusion model and flow matching technology to generate an action sequence;

[0067] In this embodiment, after obtaining the task understanding representation, it is used as input data and passed to the action decoder for processing. The action decoder is based on a diffusion model and flow matching technology, possessing powerful capabilities for generating complex action sequences and representing high-dimensional action spaces. The task understanding representation is the result of fusing multimodal information and deep processing by a pre-trained model, accurately including environmental information, task intent, and operational logic, providing the basic data conditions to drive high-precision action generation.

[0068] The diffusion model refers to a probabilistic modeling structure that simulates high-dimensional complex data distributions and gradually generates structured outputs by introducing noise and progressively denoising it. This model first initializes the noise distribution parameters based on task understanding and representation, constructing a noise space representation that meets the current task requirements. Then, it randomly samples from this noise distribution to generate an initial noise sequence. This initial noise sequence does not have a clear structure and expresses random state information.

[0069] The current noise sequence is then set to the initial noise sequence described above, and the current iteration count is initialized to zero. The noise prediction network of the diffusion model receives the current noise sequence, performs multi-layer feature extraction and prediction operations through its internal neural network structure, and outputs the prediction result of the current noise component. The predicted values ​​of the noise components are removed to obtain the updated noise sequence, reflecting the intermediate results after denoising and structural reconstruction. Each time a noise prediction and denoising process is completed, the current iteration count is incremented to record the progress of the denoising process.

[0070] The system continuously performs denoising processing and iteration counting in the noise prediction network until the current iteration count reaches the maximum number of iterations preset by the action decoder, ensuring that the denoising process has a clear termination mechanism and output time control. When the iteration ends, the system regards the updated noise sequence as the preliminary action sequence. The preliminary action sequence expresses the structural information after multiple rounds of denoising optimization, but it does not yet fully conform to the target action distribution and dynamic constraints.

[0071] To further optimize the motion generation effect, the system introduces flow matching technology. Flow matching technology constructs a probability flow from the initial motion sequence to the target motion distribution, calculates the data distribution transfer path, and optimizes the initial motion sequence along the direction of the probability flow, gradually making the motion expression structure approximate the target motion space to form an optimized motion sequence. Finally, the optimized motion sequence undergoes dynamic feasibility verification to check whether it meets the kinematic constraints, joint limits, execution safety, and operational continuity requirements of the mechanical system. The optimized motion sequence that passes verification is the final motion sequence, possessing the structural expression to directly drive the execution device to complete the task operation.

[0072] In practical applications, the diffusion model within the action decoder can be based on a conditional diffusion generative network, fusing task understanding representations and noise distribution information to ensure the denoising process is controlled by task requirements and improve the matching degree between the generated action sequence and the task intent. The noise prediction network can employ deep convolutional structures, Transformer structures, or other neural network architectures with strong expressive capabilities to improve the multi-round denoising effect.

[0073] In the flow matching technology section, designs based on continuous probabilistic flow networks or optimal transmission theory can ensure the optimal path from the initial action sequence to the target action distribution, improving the rationality of the action sequence structure and execution efficiency. During the dynamic feasibility verification process, the system can combine multi-dimensional constraints such as the robotic arm dynamics model, joint constraint model, and environmental safety model to rigorously examine the key parameters, trajectory planning, and action structure in the action sequence, eliminating action schemes that do not conform to physical laws and operational safety requirements, and ensuring the controllability and safety of the final action sequence.

[0074] The number of iterations, noise distribution parameters, and denoising intensity of the diffusion model can be flexibly adjusted according to different task requirements to optimize generation efficiency and the expressive accuracy of action sequence structures. In high-precision tasks, the number of iterations and optimization processes can be increased to ensure that the action sequences meet the requirements of complex operations in terms of structural accuracy and expressive detail.

[0075] Example description: In the field of healthcare, the motion decoder generates surgical operation motion sequences based on a diffusion model. Through flow matching optimization, the generated motions accurately meet surgical requirements and medical operation standards. The final output motion sequence is verified by dynamics to ensure that the surgical robotic arm moves accurately, safely, and reliably, assisting doctors in completing high-risk, high-precision operation tasks such as minimally invasive surgery and medical supply delivery.

[0076] In the fintech business, motion decoders generate sequences of actions for intelligent teller robotic arms, ensuring that the system accurately performs tasks such as item delivery and voucher processing in the financial service environment. Flow matching technology optimizes the coherence and security of the motion sequences, and dynamics verification ensures that the actions comply with the safety standards of the office environment, thereby improving the level of automation and service experience of financial services.

[0077] In the field of embodied intelligence, the action decoder generates a sequence of autonomous robot actions. The system uses a diffusion model for multi-round optimization and flow matching guidance to form a highly robust action expression in complex dynamic environments. Dynamic feasibility verification ensures the safe and stable execution of the robot action sequence, enhances the robot system's generalization ability and autonomous operation level in unseen scenarios and new task conditions, and realizes intelligent operation and human-machine collaboration in multiple tasks and environments.

[0078] This embodiment combines a diffusion model with flow matching technology, enabling the action decoder to generate high-dimensional complex action sequences. This effectively overcomes the limitations of traditional action generation methods in terms of the diversity, complexity, and high precision of action representation. The diffusion model's multi-round denoising mechanism ensures the gradual optimization of the action sequence structure and the stability of its representation. Flow matching technology guides the action sequence to approximate the target action distribution, improving task adaptability and execution accuracy. The entire process undergoes dynamic feasibility verification to ensure that the final output action sequence meets the physical constraints of the mechanical system and environmental safety requirements, enhancing the operational stability and safety of the robotic arm in complex dynamic environments and meeting the execution needs of demanding tasks.

[0079] S50, the execution device is controlled to perform operations according to the action sequence.

[0080] In this embodiment, based on the action sequence generated in the preceding steps, the system controls the execution device to carry out specific operational tasks. The action sequence, as the source data for control commands, expresses the motion requirements of each component of the mechanical system in terms of temporal and spatial structure. The action sequence specifically includes the expected state change information of each joint and end effector within multiple time periods, and is usually expressed using a parametric structure, such as joint angle, joint angular velocity, end effector pose, and end effector velocity.

[0081] The system first analyzes the motion sequence and extracts the specific control parameters for each component of the actuator, mainly including multiple joint angle commands and end effector position commands. The joint angle commands are used to control the rotational position or angular velocity of each joint drive unit of the robotic arm, and the end effector position commands express the spatial pose requirements of the end effector. Together, they determine the specific motion path and operational accuracy of the robotic arm in three-dimensional space.

[0082] Based on the aforementioned instruction information, the system executes motion trajectory planning, constructing a continuous trajectory path for the robotic arm to transition from its current state to the target state. The trajectory planning process combines task requirements, spatial environment information, and the structural characteristics of the robotic arm to calculate a continuous sequence of state changes, ensuring the smoothness, coherence, and task adaptability of the robotic arm's motion. Commonly used trajectory planning methods include polynomial interpolation, spline curve planning, and optimal control path calculation. The specific method is flexibly selected based on task complexity, environmental dynamics, and system performance parameters.

[0083] After generating the motion trajectory, the system performs collision detection analysis, combining environmental models, obstacle information, and the dynamic range of the robotic arm to detect potential collision risks in real time. During collision detection, geometric calculations, environmental simulation, or sensor data fusion technologies are used to accurately determine the relative position of the robotic arm with its surrounding environment during trajectory execution. If a potential collision risk is found, the system automatically adjusts the motion trajectory planning, recalculates a motion path that meets safety requirements, and generates a safe motion trajectory to ensure the safe operation of the robotic arm in complex environments.

[0084] The final safe motion trajectory is converted into drive control signals. These signals specifically include the desired torque, current, or speed commands for each joint motor, and the desired position and attitude adjustment information for the end effector. The control signals are output in a standardized format to adapt to the control system architecture of the actuator. The system sends drive control signals to the actuator, which then drives the mechanical structure to perform specific actions based on the received signals, completing the task.

[0085] In different implementation environments, the motion sequence analysis process can be combined with inverse kinematics algorithms to calculate joint angles and end effector positions in real time, improving the accuracy and real-time performance of control parameters. During motion trajectory planning, fixed-sequence trajectories, variable-speed trajectories, or dynamic obstacle avoidance trajectories can be selected according to task requirements to adapt to operational tasks of varying complexity.

[0086] In the collision detection and analysis phase, the accuracy and response speed of collision detection can be improved through sensor-based real-time environmental perception, simulation-based virtual environment detection, or a combination of multi-source information fusion methods. For highly complex dynamic environments, the system can introduce dynamic obstacle prediction models to anticipate environmental change trends and optimize trajectory adjustment strategies.

[0087] During the control signal output stage, the drive control signal format can be flexibly defined according to the type of actuator, such as using the standard CAN bus protocol, EtherCAT industrial Ethernet protocol, or a custom serial communication format, to ensure the stability of signal transmission and system compatibility.

[0088] Example Description: In the healthcare business field, the system controls a medical robotic arm to perform surgical operations by parsing motion sequences. The motion trajectory planning combines the patient's body structure and surgical needs to ensure the accuracy and safety of the surgical path. The collision detection and adjustment process prevents the robotic arm from contacting non-surgical areas, ensuring high safety standards for medical operations and improving the reliability of minimally invasive surgery and telemedicine.

[0089] In the field of fintech business, the system analyzes and executes action sequences to control the intelligent teller robotic arm to complete tasks such as ticket delivery and item placement. Trajectory planning and collision detection improve the safety and accuracy of operations in the office environment, and drive control signals to adapt to different financial terminal systems, thereby improving the level of automation of financial services and customer experience.

[0090] In the field of embodied intelligence, the system controls embodied robots to perform complex operations based on action sequences. Trajectory planning, collision detection, and dynamic adjustment ensure the safety and stability of the robot in unknown environments and changing tasks. The drive control signals efficiently drive the execution structure, enhancing the robot's autonomous operation capabilities and generalization adaptability in multiple tasks and scenarios, and promoting the practical application of embodied intelligence technology in complex real-world environments.

[0091] This embodiment achieves seamless integration of task information and physical control links by controlling the actuators based on action sequences. The complete action sequence structure fully expresses the operational intent, and the parsing and trajectory planning process ensures the accuracy of control parameters and task adaptability. Collision detection and trajectory adjustment mechanisms enhance the system's safety and operational stability in complex environments. Drive control signal conversion and standardized output ensure control consistency and real-time response capabilities for different types of actuators, thereby improving the overall operational efficiency and task completion accuracy of the mechanical system in dynamic, complex, and multi-tasking environments.

[0092] This invention relates to the field of artificial intelligence technology and can be applied to business scenarios such as embodied intelligence, fintech, and healthcare. It discloses a control method, device, equipment, and medium based on task understanding representation, comprising: acquiring an environmental image of the scene to be processed and a task instruction for specifying the operation task; processing the environmental image through a visual encoder to generate a visual feature vector; processing the task instruction through a language encoder to generate a semantic representation; fusing the visual feature vector and the semantic representation to obtain a fused feature; inputting the fused feature into a pre-trained model for processing to generate a task understanding representation; using an action decoder based on a diffusion model and flow matching technology to process the task understanding representation to generate an action sequence; and controlling an execution device to perform an operation based on the action sequence. This invention, by fusing multimodal information and introducing a task understanding mechanism, combines a diffusion model and flow matching technology to generate a highly reliable action sequence, effectively improving the robotic arm's responsiveness and operational accuracy to variable task instructions in complex environments.

[0093] In one embodiment, step S10 above includes:

[0094] S101 acquires raw environmental images of the scene to be processed using a multispectral imaging device;

[0095] S102, perform illumination normalization and geometric correction processing on the original environmental image to generate a corrected environmental image;

[0096] S103, The corrected environment image is subjected to resolution normalization processing to obtain the environment image;

[0097] S104, Receive raw operation commands from the multimodal interface;

[0098] S105, perform semantic disambiguation and intent parsing on the original operation command to generate intermediate operation instructions;

[0099] S106, The intermediate operation instructions are processed to standardize their syntax structure to generate the task instructions;

[0100] S107, perform timestamp synchronization and association processing on the environmental image and the task instruction.

[0101] In this embodiment, during the acquisition of environmental images of the scene to be processed, image data acquisition is first performed on the scene to be processed using a multispectral imaging device. Multispectral imaging devices are capable of sensing information from different electromagnetic bands, and can simultaneously capture image information from visible light, near-infrared, far-infrared, or other specific bands. Compared to single-band imaging devices, the image data acquired by multispectral imaging devices contains more structural information, material characteristics, and environmental details, making them suitable for high-precision image acquisition needs in complex environments, and ensuring the reliability and comprehensiveness of subsequent image analysis and task understanding.

[0102] The raw environmental images acquired after processing often suffer from uneven lighting conditions, image distortion, or geometric shift. Therefore, it is necessary to perform illumination normalization and geometric correction on the raw environmental images. Illumination normalization processing unifies the overall brightness level of the image through image brightness remapping, color balancing algorithms, or local contrast enhancement methods, eliminating image differences caused by changes in ambient lighting and improving the stability and consistency of image representation. Geometric correction processing addresses distortion problems during the imaging process by applying camera intrinsic parameter calibration, projection transformation, and image resampling algorithms to correct the image's geometric structure, ensuring that the spatial representation of the image is consistent with the geometric relationships of the real world and guaranteeing the spatial accuracy of image information.

[0103] After completing illumination normalization and geometric correction, the system performs resolution standardization processing on the corrected environment image. Resolution standardization adjusts the image to the system's preset standard resolution range through pixel resampling and scale unification algorithms. This ensures the spatial scale uniformity of image data collected from different devices and scenes, avoids the impact of image resolution differences on subsequent visual feature extraction and information fusion effects, and improves the consistency and algorithm adaptability of the overall system data processing.

[0104] During task instruction acquisition, the system receives raw operation commands through a multimodal interface. This multimodal interface includes a speech recognition module, a gesture recognition module, and a text input module, covering language, visual, and text information sources, thus expanding the diversity and environmental adaptability of task instruction input. The speech recognition module, based on joint inference using acoustic and language models, converts speech signals into structured text information. The gesture recognition module, combined with a depth camera or image sensor, utilizes temporal motion analysis and pattern recognition techniques to extract dynamic hand features and interpret the meaning of user gestures. The text input module directly receives structured text commands through standard human-computer interaction devices, ensuring the integrity and reliability of the task instruction acquisition process.

[0105] The received original operation commands contain uncertainties and ambiguities in their content and semantic structure. The system performs semantic disambiguation and intent parsing on the original operation commands. The semantic disambiguation process combines contextual information, predefined semantic rules, or deep learning-based language model reasoning to accurately parse polysemous words and ambiguous expressions in the operation commands and determine a unique and clear semantic meaning. Intent parsing identifies the user's operation intent through task template matching, semantic structure analysis, and behavioral goal reasoning, extracts core information such as operation category, target object, and expected result, and generates intermediate operation instructions to ensure the accuracy of operation information expression and the integrity of task logic.

[0106] To eliminate structural inconsistencies caused by differences in expression, the system further standardizes the syntax of intermediate operation instructions. Based on the system's preset syntax rules and task model structure, the syntax of the operation instructions is reorganized to generate standardized and uniformly formatted task instructions, ensuring the compatibility and stability of subsequent processing modules in information parsing and instruction transmission.

[0107] After both the environmental image and the task instructions have been processed, the system performs timestamp synchronization and association processing on the environmental image and the task instructions. By unifying the time base and aligning the data acquisition sequence, the system binds the environmental image and the task instructions into a joint data structure at the same moment, ensuring the synchronization of task instructions and scene environment information in the time dimension. This avoids operational misunderstandings or task execution errors caused by data timing misalignment, and improves the system's task adaptability and operational accuracy in dynamic and real-time environments.

[0108] This embodiment enhances the multi-dimensional representation of image information and improves the system's image perception accuracy under complex lighting and material variations by acquiring environmental images through multispectral imaging equipment. Illumination normalization and geometric correction eliminate unstable factors during imaging, ensuring the spatial structure accuracy and brightness consistency of environmental images. Resolution standardization unifies data scale and improves the fusion effect between different data sources. Multimodal interfaces expand the acquisition of task instructions, improving the flexibility and applicability of human-computer interaction. Semantic disambiguation and intent parsing improve the accuracy and comprehension of task information expression, while syntax standardization enhances the consistency and compatibility of internal data processing. Timestamp synchronization ensures a strict temporal correspondence between environmental information and task instructions, avoiding operational errors caused by information misalignment. Overall, this improves the system's adaptability to complex environments, the accuracy of information fusion, and the reliability of task execution.

[0109] In one embodiment, step S20 above includes:

[0110] S201, the environmental image is segmented into multiple image regions;

[0111] S202, The image region is processed through the spatial feature extraction path of the visual encoder to generate spatial features;

[0112] S203, The image region is processed through the semantic feature extraction path of the visual encoder to generate semantic features;

[0113] S204, fused spatial features and semantic features to generate fused visual features;

[0114] S205, perform dimensionality reduction processing on the fused visual features to generate the visual feature vector;

[0115] S206, The task instructions are processed through the word embedding layer of the language encoder to generate a word vector sequence;

[0116] S207, The word vector sequence is processed by the encoder layer of the language encoder to generate contextual features;

[0117] S208, aggregate the context features to generate an initial semantic representation;

[0118] S209, The initial semantic representation is standardized to generate the semantic representation;

[0119] S210, establish the attention association mapping between the visual feature vector and the semantic representation.

[0120] In this embodiment, to efficiently represent and semantically understand environmental image information, the system first divides the environmental image into regions, dividing the overall image into multiple image regions to improve the local representation capability and spatial structure perception accuracy of subsequent feature extraction. The division of image regions can be based on a fixed grid, image segmentation algorithms, or an adaptive division method based on visual interest points, ensuring a balance between spatial coverage and semantic integrity in the division results, facilitating the system's capture of both local and overall information.

[0121] For each image region, the spatial feature extraction path of the visual encoder performs image feature processing, and applies convolutional neural networks, graph convolutional structures or Transformer-based spatial feature modeling methods to extract spatial structure information, edge features, shape contours and local textures of the image region, forming spatial features that express spatial location and structural relationships, thereby improving the system's ability to model the spatial structure and positional relationships of objects.

[0122] Simultaneously, the semantic feature extraction path of the visual encoder performs semantic information extraction on the image region. Combining deep vision models, semantic segmentation networks, or visual semantic pre-trained models, it identifies object categories, semantic attributes, and high-level visual concepts within the image region, generates semantic features that describe the semantic content of the image region, and enhances the system's ability to express high-dimensional semantics of visual information.

[0123] Spatial features and semantic features are fused to generate fused visual features. During the fusion process, various information integration methods such as feature splicing, weighted superposition, and interactive attention mechanisms can be used to ensure that spatial structural information and semantic conceptual information are synergistically expressed in the same feature space, forming high-dimensional fused visual features that combine spatial structure and semantic content.

[0124] Visual feature vectors are generated by fusing visual features and then performing dimensionality reduction. The dimensionality reduction process can apply fully connected layers, projection transformations, or feature compression methods based on principal component analysis to reduce feature dimensions while preserving the integrity and discriminativeness of information expression, thereby improving the system's computational efficiency and adaptability to subsequent information fusion.

[0125] In the processing of task instruction information, the system maps textual task instructions into word vector sequences through the word embedding layer of the language encoder. The word embedding process can use a pre-trained language model, a lexical semantic mapping table, or an adaptive word embedding mechanism to ensure that the semantic structure of the task instructions in the vector space is accurately expressed.

[0126] The word vector sequence is processed by the encoder layer of the language encoder. Combined with self-attention mechanism, position encoding and context information modeling, context features that reflect the semantic dependencies and structural relationships within the task instructions are generated, thereby improving the system's ability to understand the overall semantics of the task instructions.

[0127] Contextual features are aggregated to generate an initial semantic representation. Aggregation methods may include sequence pooling, global attention weighting, or feature fusion strategies, resulting in a structurally complete and semantically clear semantic expression.

[0128] To improve the consistency of semantic representation and system compatibility, the system performs standardization processing on the initial semantic representation. The standardization operation ensures the stability of the semantic representation under different data inputs through feature reparameterization, scale normalization or structural unification strategies, and finally generates the semantic representation.

[0129] The system establishes an attention-related mapping between visual feature vectors and semantic representations based on an attention mechanism. The mapping process captures the correlation, semantic correspondence and complementary features between visual and linguistic information through cross-modal attention structures, joint representation modeling or multimodal feature interaction mechanisms, forming an efficient foundation for the fusion and expression of multimodal information, and providing joint feature support with a clear structure and comprehensive expression for subsequent information processing.

[0130] This embodiment enhances the system's ability to express the structural and semantic information of environmental images through image region segmentation and joint extraction of spatial and semantic paths. The fusion of visual features ensures the integrity of information and multi-dimensional collaboration. Dimensionality reduction optimizes information transmission efficiency and guarantees the compactness of feature representation. Task instructions are embedded in words and modeled in context by the language encoder to form a semantic representation that accurately expresses the semantic structure. Standardization improves the consistency of data representation and system adaptability. Through attention-based association mapping, the system effectively captures the high-level correlation between visual and linguistic information, improving the multimodal information fusion effect and enhancing the system's overall ability to understand task semantics and the accuracy of information expression in complex scenarios.

[0131] In one embodiment, step S30 above includes:

[0132] S301, Based on attention association mapping, feature alignment is performed on the visual feature vector and the semantic representation to generate aligned visual features and aligned semantic representation;

[0133] S302, The alignment visual features and the alignment semantic representation are processed by the cross-modal interaction module to generate an interaction feature vector;

[0134] S303, concatenate the alignment visual features, the alignment semantic representation, and the interaction feature vector to generate a primary fusion feature;

[0135] S304, The initial fusion feature is processed by the feature compression layer to generate the fusion feature;

[0136] S305, The fused features are input into the embedding layer of the pre-trained model to generate embedded feature representations;

[0137] S306, The embedded feature representation is processed by the multi-head cross-attention layer of the pre-trained model to generate attention-weighted features;

[0138] S307, The attention-weighted features are processed by the feedforward neural network of the pre-trained model to generate the task understanding representation.

[0139] In this embodiment, the system first performs feature alignment operation on visual feature vectors and semantic representations based on the established attention association mapping to ensure the consistency of semantic structure and coordination of expression dimension between visual information and linguistic information in the same feature space. During the feature alignment process, spatial mapping, linear transformation or semantic space projection can be combined to solve the problem of inconsistent expression of multimodal information and improve the accuracy of cross-modal information fusion.

[0140] After feature alignment is completed, the system processes the aligned visual features and semantic representations through a cross-modal interaction module. The interaction module can adopt a dual-stream neural structure, a joint attention mechanism, or a multi-layer information interaction network to capture the high-dimensional dependencies and interaction patterns between visual and linguistic information, and generate interactive feature vectors that express deep collaborative features between vision and language, thereby enhancing the integrity of information fusion and the semantic richness of expression.

[0141] The system further performs feature concatenation operations on aligned visual features, aligned semantic representations, and interactive feature vectors. The feature concatenation method can be based on vector concatenation, dimensional expansion, or structural reorganization strategies to construct primary fusion features containing multi-source information, ensuring the structural integrity and information diversity of multimodal information in joint expression.

[0142] To optimize information expression efficiency and reduce interference from redundant information, the system processes primary fusion features through a feature compression layer. The feature compression layer can compress feature dimensions based on dimensionality reduction transformation, feature filtering, or parameter sharing mechanisms, while retaining the main components and discriminative information of the expressed information, thereby generating fusion features that are compact in expression and highly integrated in information.

[0143] As the result of the joint expression of multimodal information, the fused features are input into the embedding layer of the pre-trained model. The embedding layer transforms the fused features into embedded feature representations that adapt to the structure of the pre-trained model through parameter mapping, feature reconstruction or spatial transformation, ensuring the expression standard and data structure compatibility of information in the internal processing of the model.

[0144] The system utilizes a multi-head cross-attention layer in the pre-trained model to process embedded feature representations. The multi-head cross-attention mechanism captures global dependencies and multi-source associations under different information dimensions through parallel multiple attention head structures, thereby improving the system's ability to express long-distance dependencies of cross-modal information and generating attention-weighted features that integrate multi-level semantic information and global structural information.

[0145] Finally, the attention-weighted features are processed by a feedforward neural network. The feedforward structure extracts deep features and abstract concepts from the information expression through nonlinear mapping, feature transformation and information recombination, and generates a task understanding representation that expresses the results of comprehensive semantic understanding and multimodal information fusion. The task understanding representation has high-dimensional expression capability and task adaptability, and provides structurally complete and semantically clear joint feature support for downstream action generation and control tasks.

[0146] This embodiment unifies the representational spatial structure of visual and linguistic information through feature alignment, improving the fundamental accuracy of multimodal information fusion. The introduction of a cross-modal interaction module enhances the deep dependency expression between visual and linguistic information, enriching the collaborative expression capability of multi-source information. The synergistic effect of feature concatenation and compression layers achieves the integrity of the multimodal information structure and the compactness of its expression, optimizing information transmission efficiency. The hierarchical processing of the embedding layer, cross-attention layer, and feedforward neural network comprehensively improves the system's ability to understand the semantics of complex tasks and the fusion expression effect of multimodal information, enhancing the overall accuracy of task understanding representation and information integration capability, and improving the system's adaptability and execution performance in diverse scenarios.

[0147] In one embodiment, step S40 above includes:

[0148] S401, based on the task understanding characterization, initialize noise distribution parameters, and sample from the noise distribution parameters to generate an initial noise sequence;

[0149] S402, set the current noise sequence to the initial noise sequence, and set the current iteration count to zero;

[0150] S403, The current noise sequence is processed by the noise prediction network of the diffusion model of the action decoder to generate noise component predictions;

[0151] S404, Remove the predicted noise components from the current noise sequence to generate an updated noise sequence;

[0152] S405, increment the current iteration count by one;

[0153] S406, determine whether the current iteration count has reached the preset maximum number of iteration steps of the action decoder;

[0154] S407, when the current iteration count has not reached the maximum iteration step, the current noise sequence is set to the updated noise sequence, and the noise prediction network processing is repeated until the iteration count judgment step is reached;

[0155] S408, when the current iteration count reaches the maximum iteration step count, the updated noise sequence is used as the initial action sequence;

[0156] S409, Construct a probability flow from the initial action sequence to the target action distribution using flow matching technology;

[0157] S410, optimize the initial action sequence along the direction of the probability flow to generate an optimized action sequence;

[0158] S411, Perform dynamic feasibility verification on the optimized action sequence, and take the verified optimized action sequence as the final action sequence.

[0159] In this embodiment, the system initializes noise distribution parameters based on task understanding representation to establish a random noise model expressing the uncertainty of the action space. The noise distribution parameters can be set using Gaussian, uniform, or adaptive distribution strategies to ensure the diversity and coverage of the initial data in the action decoder. The system performs random sampling based on the initialized noise distribution parameters to generate an initial noise sequence expressing the potential action space. This initial noise sequence serves as the starting input for the action decoding process, providing a basis for diverse and random action generation.

[0160] The system sets the current noise sequence as the initial noise sequence to ensure that the iterative decoding process unfolds based on a unified input state. Simultaneously, it sets the current iteration count to zero to record the number of iterations during the decoding process, controlling the computational resources and number of steps in the overall decoding process. The diffusion model within the action decoder processes the current noise sequence through a noise prediction network. Based on the network's feature extraction and mapping capabilities, it outputs predicted noise components in the current state. These noise component predictions reflect the information expressed in the portion of the current noise sequence that deviates from the target action sequence.

[0161] The system removes the predicted noise components from the current noise sequence, generating an updated noise sequence. This updated sequence retains the information structure while gradually reducing noise interference, driving the decoding process towards the target action distribution. After each noise update, the system increments the current iteration count by one, tracking the iteration progress of the decoding process in real time. The system determines whether the current iteration count has reached the maximum number of iterations preset by the action decoder. The maximum number of iterations serves as a parameter controlling decoding accuracy and computational efficiency, ensuring the decoding process is completed within a limited number of steps.

[0162] If the current iteration count has not reached the maximum number of iterations, the system will set the updated noise sequence as the new current noise sequence, and repeatedly execute the noise prediction network processing, noise removal and count update operations to iteratively promote the gradual denoising and information optimization of the noise sequence, ensuring that the final generated data has clear structural information and action expression capabilities.

[0163] When the current iteration count reaches the maximum number of iterations, the system uses the updated noise sequence as the preliminary action sequence. The preliminary action sequence expresses the preliminary action information after multiple noise predictions and optimizations. The system further constructs a probability flow from the preliminary action sequence to the target action distribution based on flow matching technology. The generation process of the probability flow combines optimal transport theory, probability mapping strategies, or distribution reconstruction mechanisms to ensure a balance between the accuracy of the action sequence and the rationality of its distribution.

[0164] The system optimizes the initial action sequence along the direction of probability flow. Through gradient descent, flow-guided optimization, or probability density-based adjustment, it generates an optimized action sequence that expresses structural stability and action logic coherence. The optimized action sequence undergoes rigorous structural adjustment and information supplementation, thereby improving the expressive completeness and target adaptability of the action sequence.

[0165] To ensure the executability and safety of the motion sequences in real-world environments, the system verifies the dynamic feasibility of optimizing the motion sequences. During the verification process, the system combines the robotic arm's dynamics model, kinematic constraints, and physical boundary conditions to evaluate the joint limitations, path accessibility, and stability of the motion sequences during execution. The system selects the optimized motion sequences that pass the dynamics verification as the final output. The final motion sequences possess high stability, executability, and complete motion logic, making them suitable for subsequent robotic arm control and manipulation tasks.

[0166] This embodiment initializes the noisy sequence based on task understanding and iteratively optimizes it using a diffusion model to gradually remove information redundancy and noise interference, thereby improving the accuracy and clarity of action sequence generation. The introduction of stream matching technology ensures the consistency between the action sequence distribution and the target action distribution, avoiding distribution offset and representation errors present in traditional action decoding. Combined with dynamic feasibility verification, action sequences with physical executableness are selected, effectively ensuring the safety and stability of action execution and comprehensively improving the representational quality of the system's generated action sequences and its adaptability in complex scenarios.

[0167] In one embodiment, step S50 above includes:

[0168] S501, parse the action sequence and extract multiple joint angle commands and end effector position commands from the action sequence;

[0169] S502, generate motion trajectory planning based on the joint angle command and the end effector position command;

[0170] S503, perform collision detection analysis on the motion trajectory planning;

[0171] S504, When a potential collision is detected, adjust the motion trajectory planning to generate a safe motion trajectory;

[0172] S505, convert the safe motion trajectory into a drive control signal;

[0173] S506, send the drive control signal to the actuator to control the actuator to complete the operation task.

[0174] In this embodiment, the system first parses the action sequence. The action sequence, as a structured data set generated in the preceding steps, expresses the action information, spatial trajectory, and control parameters of the robotic arm at each moment during the execution of a specific operation task. During parsing, the system extracts the joint angle commands and end effector position commands corresponding to each action node based on the multi-dimensional data structure contained in the action sequence, according to predefined semantic specifications, data structure rules, and action expression standards. The joint angle commands specifically indicate the spatial angle information of each joint of the robotic arm, reflecting the local motion state and posture changes of the robotic arm; the end effector position commands express the spatial position and posture parameters of the end-effector, serving as a direct representation of the overall motion path and end-effector operation effect of the robotic arm.

[0175] Based on the extracted joint angle commands and end effector position commands, the system generates corresponding motion trajectory plans. During the motion trajectory planning process, the system combines the robot arm's structural parameters, link lengths, joint rotation constraints, and end effector pose requirements, employing methods such as inverse kinematics, path interpolation, trajectory smoothing, and dynamic constraint optimization to calculate the complete motion path of the robot arm from its current state to the target state. The motion trajectory planning results not only encompass spatial path information but also consider velocity, acceleration, joint synchronization, and system stability, ensuring the smoothness and controllability of the overall movement.

[0176] The system performs collision detection analysis on the generated motion trajectory plan. The collision detection process is based on the environment model, obstacle information, robotic arm dimensions, and dynamic safety distance. Utilizing geometric analysis, spatial mapping, and real-time environmental perception data, it predicts the potential physical collisions or interference risks between the robotic arm's joints, links, and the environment during movement. Based on the detection results, the system dynamically adjusts the motion trajectory plan, optimizing path nodes, spatial trajectories, and movement rhythm to generate a safe motion trajectory. This ensures that obstacles are avoided, interference areas are reduced, and the utilization efficiency of the operating space is improved during execution.

[0177] The system converts the generated safe motion trajectory into drive control signals. The drive control signals specifically include control commands for the servo motors of each joint, drive parameters for the end effector, and real-time feedback interface settings. During the generation of drive control signals, the system calculates the corresponding electrical signals, torque outputs, or actuator action parameters based on the spatial position, motion speed, and dynamic change parameters defined in the motion trajectory, ensuring that the robotic arm accurately executes the preset motion path at the physical level.

[0178] The system sends the generated drive control signals to the actuator, which includes the robotic arm body, joint drive unit, end effector, and control interface module. The system efficiently transmits the drive control signals to each functional component via control bus, real-time control protocol, or wireless transmission. The drive actuator strictly executes the operation task according to the planned action sequence, realizing a complete closed-loop control process from data generation and path planning to physical execution.

[0179] This embodiment analyzes the motion sequence and extracts key motion control parameters to ensure accurate transmission of motion information between each joint and the end effector. Combined with motion trajectory planning, the system generates a smooth, stable spatial path that conforms to dynamic constraints based on the robotic arm's structure and operational requirements, improving the overall motion continuity and execution efficiency. Collision detection and trajectory adjustment effectively avoid physical interference and potential risks in the environment, ensuring the safety and environmental adaptability of the operation. The conversion and transmission of drive control signals achieve high-precision, high-reliability motion control, ensuring the controllability and operational stability of the robotic arm in complex task scenarios, and overall improving the system's task execution capability and intelligent operation level in uncertain environments.

[0180] In one embodiment, step S506 includes:

[0181] S5061, Send the drive control signal to the actuator to control the actuator to perform the operation corresponding to the drive control signal;

[0182] S5062, continuously monitor the actual motion trajectory of the actuator in real time;

[0183] S5063, when the deviation between the actual motion trajectory and the planned motion trajectory is greater than or equal to a preset threshold, a trajectory compensation signal is generated;

[0184] S5064, Adjust the drive control signal based on the trajectory compensation signal to generate an adjusted drive control signal;

[0185] S5065, The adjusted drive control signal is sent to the actuator to control the actuator to perform the operation corresponding to the adjusted drive control signal;

[0186] S5066, when the deviation between the actual motion trajectory and the planned motion trajectory is less than a preset threshold, the control execution device completes the operation task.

[0187] In this embodiment, the system first sends a drive control signal to the actuator. The drive control signal is a low-level execution command generated based on the action sequence, motion trajectory planning, and environmental constraints. Specifically, it includes joint angle setpoints, end effector position parameters, and necessary dynamic compensation information. The signal is transmitted in real time to the actuator's servo system, joint control unit, and end effector module via a wired or wireless data transmission link, ensuring that all components of the actuator execute operations synchronously based on unified control commands.

[0188] The system controls the actuator to perform the operation corresponding to the action parameters set in the drive control signal. During the operation, the system continuously monitors the actual motion trajectory of the actuator in real time using built-in encoders, position sensors, and external visual monitoring equipment. The actual motion trajectory includes the spatial position of each joint, rotation angle, and spatial position and attitude changes of the end effector. Through high-frequency data acquisition, real-time data fusion, and trajectory reconstruction algorithms, the system accurately obtains the dynamic changes of the actuator in the spatial environment.

[0189] The system compares the positional deviation between the actual motion trajectory and the planned trajectory. The planned trajectory is an ideal path generated by previous steps based on task requirements, spatial constraints, and safety boundaries. The positional deviation reflects the spatial error between the actual execution state and the theoretical expectation. The system's preset threshold is an upper limit of error set according to the structural characteristics of the execution device, task accuracy requirements, and environmental stability, typically quantified in the form of millimeter-level spatial positional error or angular deviation. If a positional deviation greater than or equal to the preset threshold is detected, it indicates that there is an accuracy deviation or environmental interference in the execution process, and the system immediately generates a trajectory compensation signal.

[0190] The trajectory compensation signal dynamically adjusts the specific parameters of the drive control signal based on the type, magnitude, and direction of the deviation. The adjustment process comprehensively considers the current execution state, real-time feedback information, and path optimization strategies to ensure that the compensation action quickly and effectively eliminates spatial position errors while maintaining system stability. An adjusted drive control signal is then generated, which, based on the original drive control signal, incorporates compensation correction parameters to adapt to current environmental changes and real-time execution requirements.

[0191] The system sends the adjusted drive control signal to the actuator, re-controlling the actuator to perform the operation corresponding to that signal. During the operation, the system continues to monitor the actual motion trajectory in real time, forming a dynamic closed-loop control. If the deviation between the actual motion trajectory and the planned motion trajectory is less than a preset threshold, it indicates that the execution process meets the spatial accuracy and task requirements. The system then controls the actuator to complete the operation task, terminates the compensation loop, and ensures the accuracy and stability of the overall task execution.

[0192] Example description: In the field of embodied intelligence, for scenarios where multifunctional robotic arms autonomously perform complex tasks, the system is applied in a complex desktop environment in a laboratory. It has multimodal perception and language command understanding capabilities, and its goal is to autonomously complete multi-step operation tasks specified by natural language commands.

[0193] The system first acquires raw environmental images of the desktop environment to be processed using a multispectral imaging device. This environment contains objects of different materials, colors, and reflective properties, and suffers from complex lighting interference and spatial occlusion. To improve the spatial stability and visual clarity of the image data, the system performs illumination normalization and geometric correction processing to eliminate uneven lighting and perspective distortion in the image, generating a corrected environmental image with accurate geometric structure and balanced colors. Subsequently, the system uniformly adjusts the corrected environmental image to a standard resolution to ensure consistency of image data across different devices and tasks, forming the final environmental image input.

[0194] The operation task is input through a multimodal interface, including a speech recognition module, a gesture recognition module, and a text input module. The user verbally issues an instruction in natural language: "Please move the red bottle from the left side of the table to the right side, and ensure that it does not collide with other objects." After receiving the original operation command, the system uses the semantic disambiguation and intent parsing modules to standardize the parsing of ambiguous words, spatial location descriptions, and action requirements, and outputs structured intermediate operation instructions. To facilitate further system processing, the system performs syntactic structure standardization on the intermediate operation instructions, unifying the language format and generating standardized task instructions. To ensure the temporal consistency of visual and linguistic information, the system synchronously associates the timestamps of the environmental image and the task instructions, ensuring the accurate timing of multi-source information fusion.

[0195] Subsequently, the system segments the environmental image into multiple image regions, representing different spatial areas such as the desktop, objects, and background. Using the spatial feature extraction path of the visual encoder, the geometric structure, spatial layout, and boundary information of each image region are extracted to form a spatial feature representation. Simultaneously, the system uses the semantic feature extraction path of the visual encoder to identify high-order semantic information such as object category, color, and material, generating semantic features for the image regions. The system fuses spatial and semantic features to generate fused visual features, and through dimensionality reduction processing, compresses the high-dimensional features into low-dimensional visual feature vectors.

[0196] The task instructions are converted into a sequence of word vectors through the word embedding layer of the language encoder. The system further extracts the contextual relationships between words through the encoder layer, generating contextual features that contain the overall meaning of the language, and aggregating them to generate an initial semantic representation. To adapt to multimodal fusion, the system standardizes the initial semantic representation and outputs the final semantic representation. Based on attention-based association mapping, the system dynamically captures the relationship between visual feature vectors and semantic representations, ensuring the consistency and semantic accuracy of information fusion across different modalities.

[0197] The system aligns visual feature vectors and semantic representations based on attention association mapping to obtain aligned visual features and aligned semantic representations. Subsequently, the cross-modal interaction module jointly processes the aligned features to generate interaction feature vectors, capturing the deep-seated correlation between visual and linguistic information. The system concatenates the aligned visual features, aligned semantic representations, and interaction feature vectors to form a primary fused feature, and then compresses the fused information through a feature compression layer to generate the final fused feature.

[0198] The system integrates the feature inputs into the embedding layer of the pre-trained model, generates embedded feature representations, strengthens the contextual dependencies of different feature channels through a multi-head cross-attention layer, outputs attention-weighted features, and finally processes them through a feedforward neural network to generate task-oriented task understanding representations that comprehensively represent scene information and operational intentions.

[0199] The system invokes an action decoder based on a diffusion model and flow matching technology. It initializes the noise distribution parameters based on the task understanding representation and samples them to generate an initial noise sequence. Setting the current noise sequence and initial iteration count, the system predicts and removes noise components using the noise prediction network in the diffusion model, updating the noise sequence. With each iteration, the count increments, and the system repeats the prediction and update process until the maximum number of iterations is reached, generating the initial action sequence.

[0200] The system employs flow matching technology to construct a probability flow from the initial action sequence to the target action distribution. The initial action sequence is then optimized along this probability flow to obtain an optimized action sequence. To ensure physical feasibility, the system performs a dynamic feasibility verification on the optimized action sequence, retaining only the action sequences that meet the structural limitations, kinematic constraints, and stability requirements of the actuator as the final action sequence.

[0201] Based on the final motion sequence, the system parses multiple joint angle commands and end effector position commands, and comprehensively generates a motion trajectory plan. The plan includes obstacle avoidance paths, dynamic constraints, and accuracy constraints. The system performs collision detection analysis, and if potential collision risks are detected, it adjusts the motion trajectory in real time and outputs a safe motion trajectory.

[0202] The system converts the safe movement trajectory into specific drive control signals and sends them to the actuator, controlling the robotic arm to perform the transport operation according to the predetermined trajectory. During execution, the system continuously monitors the actual movement trajectory in real time, comparing it with the planned trajectory. If the deviation is greater than or equal to a preset threshold, the system generates a trajectory compensation signal, dynamically adjusts the drive control signal, resends it, and controls the robotic arm to perform corrective actions to ensure operational accuracy. If the deviation is less than the threshold, the system determines that the operation task is completed, the robotic arm successfully transports the red bottle to the designated location, and the task ends.

[0203] In the healthcare field, targeting complex operating environments such as hospital emergency rooms and operating rooms, the system is applied to embodied intelligent robotic arms to autonomously perform designated medical assistance tasks. The goal is to safely and accurately complete tasks such as delivering items, operating equipment, or assisting in examinations based on doctors' verbal instructions and changes in the environment.

[0204] The system acquires raw environmental images of the operating room or treatment area using multispectral imaging equipment. These images encompass a variety of complex elements, including medical instruments, hospital beds, personnel activities, and surgical instruments, and are subject to interference from reflections, occlusions, and variations in lighting. The system performs illumination normalization processing to compensate for the effects of ambient light changes, while simultaneously performing geometric correction to calibrate the camera's perspective and spatial structure, ensuring stable spatial representation of the environmental images. After standardized resolution adjustment, the system generates environmental image data with clear structure and accurate spatial relationships.

[0205] Doctors issue natural language commands via voice recognition, gesture recognition, or text input modules, such as "Please deliver the sterile surgical forceps to the worktable at the foot of the bed." The system receives the original command, performs semantic disambiguation and intent parsing to eliminate semantic ambiguity, accurately extract the item name, spatial location, and operational requirements, and form intermediate operation instructions. The system further unifies the structure of these intermediate operation instructions, outputs standardized task instructions, and timestamps the environmental images and task instructions to ensure temporal alignment between perception and language information.

[0206] Subsequently, the system divides the environmental image into multiple image regions, each containing information such as the operating table, medical supplies, and personnel distribution. Using the spatial feature extraction path of the visual encoder, the system extracts the spatial layout, positional relationships, and boundary features of each region, outputting a spatial feature representation. Simultaneously, the system uses a semantic feature extraction path to identify semantic information such as item categories, equipment attributes, and personnel characteristics, forming semantic features for each image region. After fusing the spatial and semantic features, the system generates fused visual features and outputs a low-dimensional visual feature vector through dimensionality reduction processing.

[0207] The task instructions are input into the language encoder. The system generates a sequence of word vectors through a word embedding layer, extracts lexical contextual information using the encoder layer, generates contextual features, and further aggregates them to form an initial semantic representation. After standardization, the system outputs the final semantic representation, ensuring accurate and consistent expression of linguistic information. Through an attention-based association mapping mechanism, the system dynamically captures the correspondence between visual features and semantic representations, achieving effective association of cross-modal information.

[0208] The system, based on attention-based association mapping, performs feature alignment between visual feature vectors and semantic representations, outputting aligned visual features and aligned semantic representations. The system processes aligned features through a cross-modal interaction module, generating interactive feature vectors to enhance the depth of multimodal information fusion. The system concatenates aligned visual features, aligned semantic representations, and interactive feature vectors to form primary fused features, and then compresses the fused information through a feature compression layer to generate the final fused features. These fused features are input into the embedding layer of a pre-trained model, and the system outputs embedded feature representations. These representations are then reinforced with information dependencies through a multi-head cross-attention layer, and a feedforward neural network generates task-understanding representations that fully express operational intentions and environmental information.

[0209] The system initializes noise distribution parameters based on task understanding and representation, samples to generate an initial noise sequence, and sets the current noise sequence and initial iteration count. Through a noise prediction network in the diffusion model, the system predicts and removes noise components, iteratively updating the noise sequence until the maximum number of iterations is reached, outputting a preliminary action sequence. The system applies flow matching technology to construct a probability flow from the preliminary action sequence to the target action distribution, optimizes the action sequence along the direction of the probability flow, and outputs an optimized action sequence. The system performs dynamic feasibility verification on the optimized action sequence to ensure that the robotic arm movement meets physical constraints and medical operation safety requirements, generating the final action sequence.

[0210] The final motion sequence parsing generates multiple joint angle commands and end effector position commands. Based on this command information, the system generates a complete motion trajectory plan, including obstacle avoidance paths and accuracy limits to ensure operational safety. The system performs collision detection analysis; if a potential collision risk exists, the motion trajectory is dynamically adjusted, outputting a safe motion trajectory. The system converts the safe motion trajectory into drive control signals and sends them to the actuators to control the robotic arm to perform the corresponding actions, completing the medical supply delivery task.

[0211] During delivery, the system continuously monitors the actual movement trajectory of the robotic arm in real time, comparing the actual trajectory with the planned trajectory. If the deviation is greater than or equal to a threshold, the system generates a trajectory compensation signal, adjusts the drive control signal, and resends it to ensure accurate positioning and safe path during delivery. If the deviation is less than the threshold, the system confirms the operation is complete, the robotic arm accurately delivers the sterile surgical forceps to the doctor's designated position, assisting in the smooth completion of the medical operation, and the system exits the task flow.

[0212] In the fintech field, the system is applied in bank branches, smart teller terminals, or automated financial service environments to assist in completing intelligent customer service tasks and improve the efficiency and accuracy of business processing. Based on multimodal perception and language understanding, the system drives an embodied intelligent robotic arm to autonomously perform counter assistance operations or document transfer tasks in complex environments.

[0213] The system first uses multispectral imaging equipment to acquire raw environmental images of the branch or financial service area. The environment includes various visual elements such as customer seating, service counters, physical receipts, and terminal devices. The images also suffer from uneven lighting, spatial occlusion, and reflection interference. The system corrects environmental brightness variations through illumination normalization and compensates for imaging distortion using geometric correction methods to ensure accurate representation of spatial structure. After standardized resolution adjustment, the system generates environmental image data with spatial consistency and visual clarity.

[0214] Customers or financial professionals input natural language commands via voice recognition, gesture recognition, or text input modules, such as "Please deliver the customer's loan application documents to the counter window." The system receives the original command, combines semantic disambiguation and intent parsing to extract key task information, and generates intermediate operation instructions. The system further unifies the grammatical structure of the operation instructions to form standardized task instructions, while simultaneously synchronizing the timestamps of the environmental image and the task instructions to ensure the temporal consistency between language information and environmental perception information.

[0215] The system divides the environmental image into multiple image regions, each containing information such as office space, equipment distribution, and voucher storage areas. Through the spatial feature extraction path of the visual encoder, the system acquires the positional relationships and spatial layout structure of each region, outputting spatial feature representations. Simultaneously, through the semantic feature extraction path, the system identifies voucher types, equipment attributes, and personnel locations, generating semantic features for the image regions. The system fuses spatial and semantic features, outputting fused visual features, which are then dimensionality-reduced to form a visual feature vector.

[0216] The task instructions are processed by the language encoder. The system generates a sequence of word vectors through a word embedding layer, extracts contextual information through the encoder layer, aggregates them to form an initial semantic representation, and outputs the final semantic representation after standardization, thus improving the accuracy of language expression. The system establishes a dynamic correspondence between visual feature vectors and semantic representations through an attention-based association mapping mechanism, ensuring the effectiveness of information fusion.

[0217] The system utilizes attention-based association mapping to align visual features with semantic representations, generating aligned visual features and aligned semantic representations. The system then fuses these aligned features through a cross-modal interaction module, generating interactive feature vectors to improve the multimodal information fusion effect. The system concatenates the aligned visual features, aligned semantic representations, and interactive feature vectors to generate primary fused features. A feature compression layer further compresses information redundancy, outputting the final fused features. These fused features are input into the pre-trained model's embedding layer to generate embedded feature representations. These representations are then reinforced by a multi-head cross-attention layer, and finally, a feedforward neural network outputs a task-understanding representation that comprehensively expresses the operational target and environmental state information.

[0218] The system initializes noise distribution parameters based on task understanding and representation, samples to generate an initial noise sequence, sets the current noise sequence and initial iteration count, and iteratively predicts and removes noise components through a noise prediction network in the diffusion model, updating the noise sequence until the maximum number of iterations is met, outputting a preliminary action sequence. The system then constructs a probability flow from the preliminary action sequence to the target action distribution using flow matching technology, optimizes the action sequence along the probability flow direction, generates an optimized action sequence, completes dynamic feasibility verification, and outputs the final action sequence.

[0219] The final motion sequence parsing generates multiple joint angle commands and end effector position commands. Based on this information, the system plans a complete motion trajectory, including obstacle avoidance and spatial precision control. The system performs collision detection analysis, and if potential risks are found, it adjusts the motion trajectory in real time and outputs a safe motion trajectory. The system converts the safe motion trajectory into drive control signals and sends them to the actuator to control the robotic arm to autonomously perform tasks such as credential transfer, document delivery, or auxiliary operations.

[0220] During operation, the system continuously monitors the actual movement trajectory of the robotic arm in real time, comparing it with the planned trajectory. If the deviation is greater than or equal to a threshold, a trajectory compensation signal is generated, the control signal is adjusted and resent to ensure the position and path accuracy of the robotic arm, guaranteeing the accuracy and security of the financial document delivery process. If the deviation is less than the threshold, the system confirms the operation is complete, and the robotic arm accurately delivers the loan application documents to the designated location, assisting in the smooth, efficient, and compliant completion of the financial business processing procedures.

[0221] This embodiment combines drive control signals with a dynamic closed-loop actuator to ensure the operational accuracy of the robotic arm or other execution structures in complex environments. A real-time monitoring mechanism provides high-frequency, high-precision trajectory data, assisting the system in quickly identifying execution deviations. Based on dynamic adjustments to the trajectory compensation signal, the system can correct motion errors in real time, effectively suppressing operational deviations caused by environmental interference, structural deformation, or unexpected factors. The overall closed-loop control process improves spatial positioning accuracy, path tracking stability, and operational reliability during task execution, making it particularly suitable for intelligent operation scenarios involving high precision, complex trajectories, or dynamic environments.

[0222] In one embodiment, a control device based on task understanding representation is provided, which corresponds one-to-one with the control method based on task understanding representation described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the control device based on task understanding representation of the present invention. The modules include a perception input module 10, a feature extraction module 20, a multimodal fusion module 30, an action generation module 40, and an execution control module 50. Detailed descriptions of each functional module are as follows:

[0223] The perception input module 10 is used to acquire environmental images of the scene to be processed and task instructions for specifying the operation task.

[0224] Feature extraction module 20 is used to process the environmental image through a visual encoder to generate a visual feature vector, and to process the task instruction through a language encoder to generate a semantic representation;

[0225] The multimodal fusion module 30 is used to fuse the visual feature vector and the semantic representation to obtain fused features, and input the fused features into a pre-trained model for processing to generate a task understanding representation;

[0226] Action generation module 40 is used to process the task understanding representation using an action decoder based on diffusion model and flow matching technology to generate action sequences;

[0227] The execution control module 50 is used to control the execution device to perform operations according to the action sequence.

[0228] In one embodiment, the sensing input module 10 is specifically used for:

[0229] The original environmental images of the scene to be processed are acquired using multispectral imaging equipment;

[0230] The original environmental image is subjected to illumination normalization and geometric correction to generate a corrected environmental image;

[0231] The corrected environment image is then subjected to resolution normalization processing to obtain the environment image;

[0232] Receive raw operation commands from the multimodal interface;

[0233] The original operation command is semantically disambiguated and its intent is parsed to generate intermediate operation instructions;

[0234] The intermediate operation instructions are processed by standardizing their syntax to generate the task instructions;

[0235] The environmental image and the task instructions are time-stamped and associated.

[0236] In one embodiment, the feature extraction module 20 is specifically used for:

[0237] The environmental image is segmented into multiple image regions;

[0238] The image region is processed by the spatial feature extraction path of the visual encoder to generate spatial features;

[0239] The image region is processed through the semantic feature extraction path of the visual encoder to generate semantic features;

[0240] The spatial and semantic features are fused to generate fused visual features;

[0241] The fused visual features are subjected to dimensionality reduction processing to generate the visual feature vector;

[0242] The task instructions are processed by the word embedding layer of the language encoder to generate a sequence of word vectors;

[0243] The word vector sequence is processed by the encoder layer of the language encoder to generate contextual features;

[0244] Aggregate the contextual features to generate an initial semantic representation;

[0245] The initial semantic representation is standardized to generate the semantic representation.

[0246] Establish an attention association mapping between the visual feature vectors and semantic representations.

[0247] In one embodiment, the multimodal fusion module 30 is specifically used for:

[0248] Based on attention association mapping, feature alignment is performed on the visual feature vector and the semantic representation to generate aligned visual features and aligned semantic representation;

[0249] The alignment visual features and alignment semantic representation are processed by the cross-modal interaction module to generate an interaction feature vector;

[0250] The alignment visual features, the alignment semantic representation, and the interaction feature vector are concatenated to generate a primary fusion feature;

[0251] The initial fusion features are processed by a feature compression layer to generate the fusion features;

[0252] The fused features are input into the embedding layer of the pre-trained model to generate embedded feature representations;

[0253] The embedded feature representation is processed by a multi-head cross-attention layer of a pre-trained model to generate attention-weighted features;

[0254] The attention-weighted features are processed by a feedforward neural network of a pre-trained model to generate the task understanding representation.

[0255] In one embodiment, the action generation module 40 is specifically used for:

[0256] Based on the task understanding, the initial noise distribution parameters are characterized, and an initial noise sequence is generated by sampling from the noise distribution parameters;

[0257] Set the current noise sequence to the initial noise sequence, and set the current iteration count to zero;

[0258] The current noise sequence is processed by the noise prediction network of the diffusion model of the action decoder to generate noise component predictions;

[0259] Remove the predicted noise components from the current noise sequence to generate an updated noise sequence;

[0260] Increment the current iteration count by one;

[0261] Determine whether the current iteration count has reached the preset maximum number of iterations of the action decoder;

[0262] When the current iteration count has not reached the maximum number of iterations, the current noise sequence is set to the updated noise sequence, and the noise prediction network processing is repeated until the iteration count judgment step is reached.

[0263] When the current iteration count reaches the maximum iteration step count, the updated noise sequence is used as the initial action sequence;

[0264] A probabilistic flow from the initial action sequence to the target action distribution is constructed using flow matching technology;

[0265] The initial action sequence is optimized along the direction of the probability flow to generate an optimized action sequence;

[0266] The optimized action sequence is subjected to dynamic feasibility verification, and the optimized action sequence that passes the verification is taken as the final action sequence.

[0267] In one embodiment, the execution control module 50 is specifically used for:

[0268] The action sequence is parsed to extract multiple joint angle commands and end effector position commands.

[0269] Motion trajectory planning is generated based on the joint angle command and the end effector position command;

[0270] Collision detection analysis is performed on the motion trajectory planning;

[0271] When a potential collision is detected, the motion trajectory planning is adjusted to generate a safe motion trajectory;

[0272] The safe motion trajectory is converted into a drive control signal;

[0273] The drive control signal is sent to the actuator to control the actuator to complete the operation task.

[0274] In one embodiment, the execution control module 50 is specifically used for:

[0275] The drive control signal is sent to the actuator to control the actuator to perform the operation corresponding to the drive control signal;

[0276] Continuous real-time monitoring of the actual motion trajectory of the actuator;

[0277] When the deviation between the actual motion trajectory and the planned motion trajectory is greater than or equal to a preset threshold, a trajectory compensation signal is generated.

[0278] The drive control signal is adjusted based on the trajectory compensation signal to generate an adjusted drive control signal;

[0279] The adjusted drive control signal is sent to the actuator to control the actuator to perform the operation corresponding to the adjusted drive control signal;

[0280] When the deviation between the actual motion trajectory and the planned motion trajectory is less than a preset threshold, the control execution device completes the operation task.

[0281] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a task-understanding representation-based control method on the server side.

[0282] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a user-side function or step of a task-understanding representation-based control method.

[0283] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0284] Acquire environmental images of the scene to be processed and task instructions for specifying the operation tasks;

[0285] The environment image is processed by a visual encoder to generate a visual feature vector, and the task instruction is processed by a language encoder to generate a semantic representation.

[0286] The visual feature vector and the semantic representation are fused to obtain fused features, and the fused features are input into a pre-trained model for processing to generate task understanding representations;

[0287] The task understanding representation is processed using an action decoder based on a diffusion model and flow matching technology to generate an action sequence;

[0288] The actuator is controlled to perform operations according to the action sequence.

[0289] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0290] Acquire environmental images of the scene to be processed and task instructions for specifying the operation tasks;

[0291] The environment image is processed by a visual encoder to generate a visual feature vector, and the task instruction is processed by a language encoder to generate a semantic representation.

[0292] The visual feature vector and the semantic representation are fused to obtain fused features, and the fused features are input into a pre-trained model for processing to generate task understanding representations;

[0293] The task understanding representation is processed using an action decoder based on a diffusion model and flow matching technology to generate an action sequence;

[0294] The actuator is controlled to perform operations according to the action sequence.

[0295] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0296] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0297] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0298] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A control method based on task understanding representation, characterized in that, Includes the following steps: Acquire environmental images of the scene to be processed and task instructions for specifying the operation tasks; The environment image is processed by a visual encoder to generate a visual feature vector, and the task instruction is processed by a language encoder to generate a semantic representation. The visual feature vector and the semantic representation are fused to obtain fused features, and the fused features are input into a pre-trained model for processing to generate a task understanding representation; The task understanding representation is processed using an action decoder based on a diffusion model and flow matching technology to generate an action sequence. The actuator is controlled to perform operations according to the action sequence.

2. The control method based on task understanding and representation as described in claim 1, characterized in that, Acquire environmental images of the scene to be processed and task instructions for specifying the operation tasks, including: The original environmental images of the scene to be processed are acquired using multispectral imaging equipment; The original environmental image is subjected to illumination normalization and geometric correction to generate a corrected environmental image; The corrected environment image is then subjected to resolution normalization processing to obtain the environment image; Receive raw operation commands from the multimodal interface; The original operation command is semantically disambiguated and its intent is parsed to generate intermediate operation instructions; The intermediate operation instructions are processed by standardizing their syntax to generate the task instructions; The environmental image and the task instructions are time-stamped and associated.

3. The control method based on task understanding representation as described in claim 1, characterized in that, The environment image is processed by a visual encoder to generate a visual feature vector, and the task instructions are processed by a language encoder to generate a semantic representation, including: The environmental image is segmented into multiple image regions; The image region is processed by the spatial feature extraction path of the visual encoder to generate spatial features; The image region is processed through the semantic feature extraction path of the visual encoder to generate semantic features; The spatial and semantic features are fused to generate fused visual features; The fused visual features are subjected to dimensionality reduction processing to generate the visual feature vector; The task instructions are processed by the word embedding layer of the language encoder to generate a sequence of word vectors; The word vector sequence is processed by the encoder layer of the language encoder to generate contextual features; Aggregate the contextual features to generate an initial semantic representation; The initial semantic representation is standardized to generate the semantic representation. Establish an attention association mapping between the visual feature vectors and semantic representations.

4. The control method based on task understanding representation as described in claim 1, characterized in that, The visual feature vector and the semantic representation are fused to obtain a fused feature, which is then input into a pre-trained model for processing to generate a task-understanding representation, including: Based on attention association mapping, feature alignment is performed on the visual feature vector and the semantic representation to generate aligned visual features and aligned semantic representation; The alignment visual features and alignment semantic representation are processed by the cross-modal interaction module to generate an interaction feature vector; The alignment visual features, the alignment semantic representation, and the interaction feature vector are concatenated to generate a primary fusion feature; The initial fusion features are processed by a feature compression layer to generate the fusion features; The fused features are input into the embedding layer of the pre-trained model to generate embedded feature representations; The embedded feature representation is processed by a multi-head cross-attention layer of a pre-trained model to generate attention-weighted features; The attention-weighted features are processed by a feedforward neural network of a pre-trained model to generate the task understanding representation.

5. The control method based on task understanding representation as described in claim 1, characterized in that, The task understanding representation is processed using an action decoder based on a diffusion model and flow matching technology to generate an action sequence, including: Based on the task understanding, the initial noise distribution parameters are characterized, and an initial noise sequence is generated by sampling from the noise distribution parameters; Set the current noise sequence to the initial noise sequence, and set the current iteration count to zero; The current noise sequence is processed by the noise prediction network of the diffusion model of the action decoder to generate noise component predictions; Remove the predicted noise components from the current noise sequence to generate an updated noise sequence; Increment the current iteration count by one; Determine whether the current iteration count has reached the preset maximum number of iterations of the action decoder; When the current iteration count has not reached the maximum number of iterations, the current noise sequence is set to the updated noise sequence, and the noise prediction network processing is repeated until the iteration count judgment step is reached. When the current iteration count reaches the maximum iteration step count, the updated noise sequence is used as the initial action sequence; A probabilistic flow from the initial action sequence to the target action distribution is constructed using flow matching technology; The initial action sequence is optimized along the direction of the probability flow to generate an optimized action sequence; The optimized action sequence is subjected to dynamic feasibility verification, and the optimized action sequence that passes the verification is taken as the final action sequence.

6. The control method based on task understanding representation as described in claim 1, characterized in that, The execution device is controlled to perform operations according to the action sequence, including: The action sequence is parsed to extract multiple joint angle commands and end effector position commands. Motion trajectory planning is generated based on the joint angle command and the end effector position command; Collision detection analysis is performed on the motion trajectory planning; When a potential collision is detected, the motion trajectory planning is adjusted to generate a safe motion trajectory; The safe motion trajectory is converted into a drive control signal; The drive control signal is sent to the actuator to control the actuator to complete the operation task.

7. The control method based on task understanding representation as described in claim 6, characterized in that, Sending the drive control signal to the actuator to control the actuator to complete the operation task includes: The drive control signal is sent to the actuator to control the actuator to perform the operation corresponding to the drive control signal; Continuous real-time monitoring of the actual motion trajectory of the actuator; When the deviation between the actual motion trajectory and the planned motion trajectory is greater than or equal to a preset threshold, a trajectory compensation signal is generated. The drive control signal is adjusted based on the trajectory compensation signal to generate an adjusted drive control signal; The adjusted drive control signal is sent to the actuator to control the actuator to perform the operation corresponding to the adjusted drive control signal; When the deviation between the actual motion trajectory and the planned motion trajectory is less than a preset threshold, the control execution device completes the operation task.

8. A control device based on task understanding representation, characterized in that, The control device based on task understanding representation includes: The perception input module is used to acquire environmental images of the scene to be processed and task instructions for specifying the operation task. The feature extraction module is used to process the environmental image through a visual encoder to generate a visual feature vector, and to process the task instruction through a language encoder to generate a semantic representation. The multimodal fusion module is used to fuse the visual feature vector and the semantic representation to obtain fused features, and input the fused features into a pre-trained model for processing to generate task understanding representations; The action generation module is used to process the task understanding representation using an action decoder based on a diffusion model and flow matching technology to generate action sequences; The execution control module is used to control the execution device to perform operations according to the action sequence.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a task-understanding representation-based control program stored in the memory and executable on the processor, wherein the task-understanding representation-based control program, when executed by the processor, implements the steps of the task-understanding representation-based control method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a control program based on task understanding representation, which, when executed by a processor, implements the steps of the control method based on task understanding representation as described in any one of claims 1-7.

Images