Flexible robot control method and system based on multi-modal data and heuristic graph search

Abstract

The application discloses a flexible robot control method and system based on multi-modal data and heuristic graph search, and the method comprises the following steps: acquiring multi-modal data; performing feature extraction on the multi-modal data to obtain multi-modal features; performing fusion operation on the multi-modal features to obtain fused features; acquiring a target task path graph according to the fused features; and controlling a target flexible robot to perform a target task according to the target task path graph; wherein the multi-modal features comprise voice features, expression features and gesture features. The application can improve the efficiency and flexibility of task execution of robots, and can be widely applied to the technical field of machine learning.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a flexible robot control method and system based on multimodal data and heuristic graph search. Background Technology

[0002] In flexible manufacturing environments, production lines need to adapt to rapidly changing product types and processes to achieve efficient customized production. Against this backdrop, industrial robots, as core components of automated production, are increasingly taking on more and more tasks. However, traditional industrial robots often lack the ability to understand and adapt to complex scenarios, especially when facing frequently changing assembly tasks, often failing to meet actual production needs. Traditional industrial robots have significant shortcomings in multimodal data processing, typically only handling single-modal data (such as vision or speech), lacking the ability to comprehensively perceive and understand complex human-machine interaction information. This single-modal processing approach makes robots inflexible in receiving and understanding user instructions, struggling to accurately interpret operator voice commands, gestures, and facial expressions, resulting in an unnatural and unsmooth interaction process. Existing robot task planning methods mostly employ rule-based preset paths or traditional search algorithms. These methods often exhibit low efficiency and adaptability when facing changing task scenarios or complex multi-step tasks. Furthermore, existing technologies typically rely on preset processing rules when dealing with changes in the external environment of the production line, making it difficult to adjust strategies in real time, thus affecting the successful completion of tasks. Summary of the Invention

[0003] In view of this, the main objective of the embodiments of the present invention is to provide a flexible robot control method and system based on multimodal data and heuristic graph search, in order to solve at least one of the problems in the prior art. The present invention can improve the efficiency and flexibility of the robot in performing tasks.

[0004] To achieve the above objectives, one aspect of the present invention provides a flexible robot control method based on multimodal data and heuristic graph search, comprising the following steps:

[0005] Acquire multimodal data;

[0006] Feature extraction is performed on the multimodal data to obtain multimodal features;

[0007] The multimodal features are fused to obtain fused features;

[0008] Based on the fusion features, obtain the target task path map;

[0009] Based on the target task path diagram, control the target flexible robot to execute the target task;

[0010] The multimodal features include speech features, facial expression features, and gesture features.

[0011] In some embodiments, a flexible robot control method based on multimodal data and heuristic graph search further includes the following steps:

[0012] The model parameters of the target flexible robot are updated according to the target task.

[0013] In some embodiments, the step of extracting features from the multimodal data to obtain multimodal features includes the following steps:

[0014] Select the corresponding Transformer model based on the modality type of the multimodal data;

[0015] Based on the Transformer model, feature extraction is performed on the multimodal data to obtain the multimodal features.

[0016] In some embodiments, the fusion operation on the multimodal features to obtain fused features includes the following steps:

[0017] Based on the speech features, facial expression features, and gesture features, attention weights are obtained;

[0018] Based on the attention weights, the speech features, facial expression features, and gesture features are fused using a cross-attention mechanism to obtain the fused features.

[0019] In some embodiments, obtaining the target task path map based on the fusion features includes the following steps:

[0020] The fused features are mapped to the joint embedding space to obtain the embedded features;

[0021] The embedded features are parsed using a task decoder to obtain a sequence of task instructions;

[0022] Based on the sequence of task instructions, construct an initial task path diagram;

[0023] Based on the initial task path graph, construct an evaluation function;

[0024] Minimize the evaluation function to obtain the target task path graph.

[0025] In some embodiments, controlling the target flexible robot to perform the target task according to the target task path map includes the following steps:

[0026] Based on the target task path diagram, generate control commands;

[0027] In response to the control command, the target flexible robot is controlled to perform the target task.

[0028] In some embodiments, after the step of controlling the target flexible robot to perform the target task according to the target task path diagram, the following step is further included:

[0029] Get the current environment status;

[0030] Construct a reward function based on the control instructions and the current environmental state;

[0031] The execution strategy is updated based on the reward function.

[0032] To achieve the above objectives, another aspect of this invention proposes a flexible robot control system based on multimodal data and heuristic graph search, the system comprising:

[0033] The first module is used to acquire multimodal data;

[0034] The second module is used to extract features from the multimodal data to obtain multimodal features;

[0035] The third module is used to perform a fusion operation on the multimodal features to obtain fused features;

[0036] The fourth module is used to obtain the target task path map based on the fusion features;

[0037] The fifth module is used to control the target flexible robot to perform the target task according to the target task path map.

[0038] In some embodiments, a flexible robot control system based on multimodal data and heuristic graph search further includes:

[0039] The sixth module is used to update the model parameters of the target flexible robot according to the target task.

[0040] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0041] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0042] To achieve the above objectives, another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0043] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a flexible robot control method and system based on multimodal data and heuristic graph search. This scheme acquires multimodal data; extracts features from the multimodal data to obtain multimodal features; performs a fusion operation on the multimodal features to obtain fused features; obtains a target task path graph based on the fused features; and controls the target flexible robot to perform the target task based on the target task path graph, thereby improving the efficiency and flexibility of the robot in performing tasks. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of a flexible robot control method based on multimodal data and heuristic graph search provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the flexible robot control system structure based on multimodal data and heuristic graph search provided in an embodiment of the present invention;

[0047] Figure 3 This is an overall operation flowchart of the flexible robot control system based on multimodal data and heuristic graph search provided in the embodiments of the present invention;

[0048] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0050] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification, claims, and the foregoing drawings may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0051] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0053] In flexible manufacturing environments, production lines need to adapt to rapidly changing product types and processes to achieve efficient, customized production. Against this backdrop, industrial robots, as core components of automated production, are increasingly taking on more and more tasks. However, traditional industrial robots often lack the ability to understand and adapt to complex scenarios, especially when faced with frequently changing assembly tasks, often failing to meet actual production needs.

[0054] The introduction of multimodal data processing technology has provided new possibilities for human-computer interaction and task understanding. However, although multimodal data processing technology enhances the robot's perception capabilities, in practical applications, assembly tasks in flexible manufacturing environments are typically multi-step and dynamically changing, requiring robots to possess higher-level task planning and execution capabilities. Traditional rule-based planning methods often lack flexibility and adaptability when facing uncertainties and complex tasks. Existing production line scheduling technologies have the following main drawbacks in multi-robot collaboration and dynamic scheduling:

[0055] Insufficient multimodal data processing capabilities: Traditional industrial robots have significant shortcomings in multimodal data processing, typically only able to process single-modal data (such as vision or speech), lacking the ability to comprehensively perceive and understand complex human-machine interaction information. This single-modal processing approach makes the robot inflexible in receiving and understanding user instructions, making it difficult to accurately understand the operator's voice commands, gestures, and facial expressions, resulting in an unnatural and unsmooth interaction process.

[0056] The flexibility and efficiency of task planning are relatively low: Existing robot task planning methods mostly employ rule-based preset paths or traditional search algorithms. These methods often exhibit low efficiency and adaptability when faced with changing task scenarios or complex, multi-step tasks. In flexible manufacturing environments, task flows may change frequently, such as product specification updates or adjustments to process requirements. Traditional methods require frequent reprogramming or manual adjustments in these dynamically changing scenarios, lacking real-time optimization capabilities, leading to decreased production efficiency.

[0057] Limited adaptability to environmental changes: In flexible manufacturing environments, external conditions on the production line (such as changes in lighting, noise interference, and equipment malfunctions) can affect robot task execution. Existing technologies typically rely on preset processing rules when dealing with these environmental changes, making it difficult to adjust their strategies in real time. Especially when faced with unexpected situations during task execution, robots lack adaptive learning capabilities and cannot quickly adjust their actions or strategies based on real-time feedback, thus affecting the successful completion of tasks.

[0058] Lack of intelligent optimization methods in task execution: Existing robotic systems often lack intelligent task optimization methods and cannot self-optimize based on real-time task progress and feedback. Although some systems have introduced simple feedback mechanisms, they cannot achieve optimization and continuous learning for multi-step tasks in complex environments, making it difficult to improve the efficiency and quality of long-term task execution.

[0059] In view of this, such as Figure 1As shown, this embodiment of the invention provides a flexible robot control method based on multimodal data and heuristic graph search, which may include, but is not limited to, steps S100 to S500:

[0060] Step S100: Acquire multimodal data;

[0061] Step S200: Extract features from the multimodal data to obtain multimodal features;

[0062] Step S300: Perform a fusion operation on the multimodal features to obtain fused features;

[0063] Step S400: Obtain the target task path map based on the fusion features;

[0064] Step S500: Control the target flexible robot to perform the target task according to the target task path diagram;

[0065] The multimodal features include speech features, facial expression features, and gesture features.

[0066] In steps S100 to S500 of some embodiments, deep analysis and fusion of multimodal information are achieved by integrating data from multiple modalities such as voice, facial expressions, and gestures, and performing feature extraction. Subsequently, task planning and path optimization are performed, enabling the robot to adaptively complete complex assembly tasks in a flexible manufacturing environment and make real-time adjustments according to environmental changes. Compared to traditional systems, this solution significantly improves the accuracy of task understanding, adaptability, and efficiency of task execution, better meeting the high requirements of modern manufacturing for automation and flexibility.

[0067] In step S100 of some embodiments, multimodal data refers to information from different types of sensors, such as voice, gestures, facial expressions, and environmental images. By processing multimodal data, the robot can achieve information fusion and understanding across different sensory channels, thereby improving its ability to understand the operator's intentions and making human-computer interaction more natural and efficient. For example, multimodal data such as the operator's voice, gestures, and facial expressions can be collected through various sensors. These sensors may include a high-sensitivity microphone array for capturing voice commands, a high-definition camera for acquiring facial expressions and hand gestures, and a depth sensor. To ensure the temporal and spatial consistency of different modal data, the system employs an event-driven temporal coding mechanism for synchronous processing, aligning data from different modalities to ensure synchronization. Furthermore, multimodal data is preprocessed. The data preprocessing stage may include using noise reduction filtering and voice enhancement techniques to preprocess voice data to improve voice signal quality; and performing noise reduction, enhancement, and normalization processing on images and videos to extract key feature points, thereby preprocessing visual data to ensure the accuracy and effectiveness of subsequent feature extraction.

[0068] In some embodiments, step S200 may include, but is not limited to, steps S210 to S220:

[0069] Step S210: Select the corresponding Transformer model according to the modality type of the multimodal data;

[0070] Step 8220: Based on the Transformer model, perform feature extraction on the multimodal data to obtain the multimodal features.

[0071] In steps S210 to S220 of some embodiments, the most suitable Transformer model is automatically selected based on the modality type and features of the input multimodal data. For example, for speech data, the Wav2Vec2.0 Transformer can be selected to extract semantic and pitch features; for facial expression data, the VisionTransformer (ViT) can be selected to obtain fine visual features; for gesture data, the TimeSformer can be selected to process time-series video streams and extract features of dynamic gestures, but is not limited to these. The feature selection controller can dynamically adjust the selection and configuration of the Transformer model according to real-time environmental conditions (such as illumination and noise levels) to ensure efficient and accurate feature extraction from different modal data. The generated high-dimensional feature vectors provide a rich information foundation for subsequent multimodal fusion.

[0072] In some embodiments, feature extraction may include the following process:

[0073] 1) Speech feature extraction: Input speech signal x audio High-dimensional features are extracted using the Wav2Vec 2.0 model:

[0074] h audio =Wav2Vec2(x audio )

[0075] 2) Facial feature extraction: Input facial image x face Features are extracted using the ViT model:

[0076] h face =ViT(x face )

[0077] 3) Gesture feature extraction: Input gesture video sequence X gesture ={x1, x2, ..., x T Features are extracted using the TimeSformer model:

[0078] h gesture =TimeSformer(X gesture )

[0079] Optionally, the high-dimensional feature representation obtained after processing the original speech signal using the Wav2Vec 2.0 model includes semantic information and intonation features in the speech, reflecting the instructions input by the operator through speech. The feature representation obtained after processing the facial image using the Vision Transformer (ViT) model includes the operator's facial expression information; these features can help the robot understand the operator's emotional state and intentions. The temporal features obtained after processing the gesture sequence (video stream) using the TimeSformer model reflect the operator's gesture commands; these features are used to parse the specific task requirements conveyed by the operator through gestures.

[0080] In some embodiments, step S300 may include, but is not limited to, steps S310 to S320:

[0081] Step S310: Obtain attention weights based on the voice features, facial expression features, and gesture features;

[0082] Step S320: Based on the attention weights, the speech features, facial expression features, and gesture features are fused using a cross-attention mechanism to obtain the fused features.

[0083] In step S310 of some embodiments, the attention weights Attention(Q, K, V) can be calculated using voice features as the query (Q), facial expression features as the key (K), and gesture features as the value (V):

[0084]

[0085] Where Q = W Q h audio K = W K [h faace h gesture V = W V [h faace h gesture ]

[0086] In the formula, W Q W K W V d is the learnable parameter matrix; k h represents the dimension of the key vector. audio For speech features; h face For facial expression features; h gesture Gesture characteristics.

[0087] In step S320 of some embodiments, based on the attention weights Attention(Q, K, V), features from different modalities are deeply fused using a Cross-Attention mechanism to obtain the fused feature h. fusion :h fusion =Attention(Q, K, V). The Cross-Attention mechanism deeply fuses features extracted from different modalities, establishing dynamic relationships between multimodal features and automatically adjusting the weights of each modality during the fusion process to ensure that important information is prioritized. Through this optimized mechanism, the robot can establish connections between different modalities, improving its understanding of operator intentions and making human-computer interaction more natural and fluid.

[0088] In some embodiments, step S400 may include, but is not limited to, steps S410 to S450:

[0089] Step S410: Map the fused features to the joint embedding space to obtain the embedding features;

[0090] Step S420: The embedded features are parsed using the task decoder to obtain the task instruction sequence;

[0091] Step S430: Construct an initial task path diagram based on the task instruction sequence;

[0092] Step S440: Construct an evaluation function based on the initial task path graph;

[0093] Step S450: Minimize the evaluation function to obtain the target task path graph.

[0094] In step S410 of some embodiments, mapping the fused features to the joint embedding space yields the embedded features h. embedding This achieves semantic alignment. For example, the expression for the embedded feature is: h embedding =φ(h) fusion ), where φ(·) is a nonlinear mapping function, which can optionally be a fully connected layer plus an activation function. By mapping the fused multimodal features to a unified joint embedding space, semantic alignment and consistency are achieved, enabling the system to fully understand the operator's multimodal instructions, extract a comprehensive intent representation, and provide a solid foundation for task understanding and planning.

[0095] In step S420 of some embodiments, complex natural language instructions and gestures are parsed into a specific sequence of task steps using a task decoder. For example, parsing the embedded features using a task decoder can yield a sequence of task instructions. Optionally, the process expression for converting the embedded features into a sequence of task instructions is:

[0096] Instruction = ψ(h) embedding )

[0097] Where ψ(·) is the task decoder; the task instruction sequence Instruction={T1,T2,...,T N}

[0098] In steps S430 to S450 of some embodiments, a heuristic graph search algorithm is used to plan and optimize the task steps. The heuristic graph search algorithm constructs a task graph, nodes the task steps, and searches for and selects the optimal path based on task priority and execution cost. This algorithm can dynamically adjust the task execution path in a flexible manufacturing environment, ensuring that the robot can efficiently and flexibly cope with changing assembly task requirements, improving the overall efficiency and flexibility of task execution. For example, based on the task instruction sequence, nodes T representing each step of the assembly task are obtained. i Let i ∈ {1, 2, ..., N}, and obtain the edges representing the transition relationships between steps, where the weight of the edge is the execution cost c. ijBased on each node and edge, an initial task path graph can be constructed. Optionally, the actual cost g(n) from the starting node to the current node n is obtained; the minimum cost from the current node n to the target node is estimated by constructing a heuristic function h(n), resulting in the heuristic estimated cost h(n). Then, the evaluation function f(n) = g(n) + h(n) can be constructed using the A* algorithm. By minimizing the evaluation function f(n), optimal path search can be achieved, finding the optimal task execution path from the starting node to the target node, thus forming the target task path graph.

[0099] In some embodiments, step S500 may include, but is not limited to, steps S510 to S520:

[0100] Step S510: Generate control instructions based on the target task path diagram;

[0101] Step S520: In response to the control command, control the target flexible robot to perform the target task.

[0102] In steps S510 to S520 of some embodiments, corresponding robot control instructions a can be generated based on the planned task sequence in the target task path diagram. t Next, the robot actuator can perform corresponding physical actions according to the control commands.

[0103] Following step S500 in some embodiments, the following steps may also be included: acquiring the current environmental state; constructing a reward function based on the control command and the current environmental state; and updating the execution strategy based on the reward function. For example, the current environmental state s is acquired through a sensor. t The reward function is designed based on factors such as task completion and user satisfaction. Optionally, it can be based on control instruction a. t and the current environmental state s t Construct the reward function r t =R(s) t a t Based on the reward function, the execution policy Q(s) can be adjusted. t a t The update process is performed using the following expression:

[0104]

[0105] Where α is the learning rate, γ is the discount factor, a′ is the next possible action, and s t+1 This represents the environment state at time t+1.

[0106] By selecting Q(s) t a tThe largest action a t The robot update strategy π(a) is obtained. t |s t ).

[0107] During task execution, the robot monitors the task status and environmental changes in real time through sensors, collecting feedback information. The system combines reinforcement learning algorithms to adjust its execution strategy based on real-time feedback, optimizing the efficiency and accuracy of task execution. Reinforcement learning, by setting a reward function, drives the robot to continuously improve its behavioral choices during trial and error, enabling it to continuously enhance its task execution capabilities in dynamic environments and ensuring the efficient completion of assembly tasks.

[0108] In some embodiments, the flexible robot control method based on multimodal data and heuristic graph search may also include, but is not limited to, step S600, updating the model parameters of the target flexible robot according to the target task. Through meta-learning for rapid adaptation and federated learning model aggregation, the model parameters of the target flexible robot are updated, achieving continuous learning and knowledge sharing. For example,

[0109] 1) Meta-learning for rapid adaptation

[0110] Meta-model training: Train the meta-model on multiple target tasks to enable it to quickly adapt to new tasks.

[0111] Task adaptation: In the new target task T new To quickly update model parameters using a small amount of data, we have the following expression:

[0112]

[0113] Where θ represents the initial meta-model parameters, β represents the meta-learning rate, and θ new To update the target meta-model parameters, In the new target task T new The loss function on.

[0114] 2) Federated learning model aggregation

[0115] Local model update: Each robot updates its model parameters θ based on local data. i .

[0116] Global model aggregation: The server aggregates all local model parameters to generate a global model.

[0117]

[0118] Where K is the number of robots, n iLet be the data volume of the i-th robot, and n be the total data volume of the robots used, which is the sum of the data samples of all robots.

[0119] Model distribution: Distribute the updated global model parameters Φ to each robot.

[0120] By employing meta-learning and federated learning technologies, robots can achieve adaptive learning and knowledge sharing during task execution. Meta-learning enables robots to quickly adapt to new tasks and environmental changes, requiring only a small amount of data to adjust model parameters and improve their adaptability to new tasks. Federated learning, through a cloud-based knowledge-sharing platform, uploads the experiences and optimization strategies learned locally by each robot to the cloud, aggregates them to generate a global model, and distributes it back to each robot. This collaborative learning mechanism not only improves the intelligence level of individual robots but also enhances the collaborative efficiency and intelligence level of the entire robot group, realizing the accumulation and inheritance of knowledge.

[0121] like Figure 2 As shown, this embodiment of the invention also provides a flexible robot control system based on multimodal data and heuristic graph search, which can implement the above-mentioned flexible robot control method based on multimodal data and heuristic graph search. The system includes:

[0122] The first module (multimodal perception module) is used to acquire multimodal data;

[0123] The second module (feature processing and selection module) is used to extract features from the multimodal data to obtain multimodal features;

[0124] The third module (multimodal fusion module) is used to perform fusion operations on the multimodal features to obtain fused features;

[0125] The fourth module (task planning and optimization module) is used to obtain the target task path map based on the fusion features;

[0126] The fifth module (execution control module) is used to control the target flexible robot to execute the target task according to the target task path diagram;

[0127] The sixth module (continuous learning and cloud collaboration module) is used to update the model parameters of the target flexible robot according to the target task.

[0128] The multimodal perception module includes:

[0129] The voice sensing unit is equipped with a microphone array for capturing and processing the operator's voice commands in real time.

[0130] The visual perception unit is equipped with a high-definition camera and a depth sensor to capture and process the operator's facial expressions and gestures in real time.

[0131] The data synchronization unit is used to synchronize data from different sensing units in time, ensuring the temporal consistency of multimodal data.

[0132] The feature processing and selection module includes:

[0133] The feature selection controller is used to automatically select the most suitable Transformer model based on the modality type of the input data and the real-time environmental conditions.

[0134] The speech Transformer model, specifically the Wav2Vec2-Transformer, is used to process speech data and extract semantic and pitch features.

[0135] The visual Transformer model, specifically the Vision Transformer (ViT), is used to process facial expression data and extract subtle visual features.

[0136] The gesture Transformer model, specifically the TimeSformer, is used to process gesture action data and extract dynamically changing temporal features.

[0137] The multimodal fusion module includes:

[0138] The Cross-Attention fusion unit is equipped with a Cross-Attention mechanism to establish dynamic relationships between features of different modalities and automatically adjust the weights of each modality in the fusion process.

[0139] A joint embedding space builder is used to map fused multimodal features to a unified embedding space, achieving semantic alignment and consistent representation.

[0140] The task planning and optimization module includes:

[0141] The heuristic graph search algorithm unit is equipped with a heuristic graph search algorithm, which is used to transform the parsed task instructions into a task graph and plan the optimal task execution path based on the heuristic search method.

[0142] The dynamic task adjustment unit is used to dynamically adjust the task execution path based on real-time feedback, ensuring the efficient and flexible completion of tasks.

[0143] The execution control module includes:

[0144] The motion control unit is used to generate specific robot control commands based on the instructions from the task planning module;

[0145] The real-time feedback unit is used to monitor the task execution status and environmental changes through sensors and transmit the feedback information to the task planning and optimization module.

[0146] The human-computer interaction unit is used for real-time interaction with operators, including voice confirmation and gesture feedback.

[0147] The continuous learning and cloud collaboration module includes:

[0148] The reinforcement learning optimization unit is equipped with a reinforcement learning algorithm, which is used to continuously optimize the robot's execution strategy based on task completion, execution efficiency, and user satisfaction.

[0149] The meta-learning unit is equipped with meta-learning technology, which enables the robot to quickly adapt to new tasks and environmental changes through meta-learning.

[0150] The federated learning platform is used to upload the optimization strategies and experiences learned locally by each robot to the cloud, aggregate and optimize the global model, and distribute the updated global model back to each robot to achieve knowledge sharing and collaborative optimization.

[0151] Through the collaboration of various modules and units, efficient processing of multimodal data, intelligent planning and execution of tasks, and adaptive learning and optimization are achieved.

[0152] In some embodiments, a verification experiment of a flexible robot control system based on multimodal data and heuristic graph search was conducted in a simulated flexible manufacturing workshop environment, using a multi-jointed humanoid robot as the experimental subject. The robot was equipped with a high-sensitivity microphone array, a high-definition camera, and a depth sensor to capture the operator's voice commands, facial expressions, and hand gestures. For the computing platform, the experiment used a high-performance server equipped with an NVIDIA A100 GPU, running an operating system based on Ubuntu 20.04LTS. Multimodal Transformer models, including Wav2Vec2, Vision Transformer (ViT), and TimeSformer, were built and trained using the PyTorch 2.0 framework. Furthermore, a custom heuristic graph search algorithm and a Proximal Policy Optimization (PPO) reinforcement learning algorithm were integrated to support task planning and execution optimization. The experimental scenario design included various assembly tasks, such as the assembly, fastening, and quality inspection of different types of parts, simulating the variable demands of actual flexible manufacturing. To test the system's adaptability, dynamic factors such as changes in lighting, background noise, and equipment failures were introduced into the experimental environment to observe the robot's performance under different environmental conditions. Meanwhile, by setting up multiple rounds of human-computer interaction, the system's response speed and accuracy in voice command recognition, gesture control, and facial expression understanding were evaluated. A real-time sensor feedback mechanism ensures that the robot can promptly acquire information on task execution status and environmental changes, enabling dynamic adjustments and optimizations.

[0153] To comprehensively evaluate system performance, the experiment employed multiple evaluation metrics, including task completion rate, response time, learning speed, and interaction satisfaction. Task completion rate measures the proportion of the robot that successfully completes assembly tasks within a specified time. Response time assesses the delay between receiving an instruction and executing the action. Learning speed is measured by recording the system's adaptation and optimization process under different task scenarios. Interaction satisfaction is evaluated through subjective ratings of the robot's interactive experience by operators. These metrics aim to verify the effectiveness and advantages of this technical solution in multimodal data processing, task planning and execution, and adaptive learning, providing a basis for further system optimization and application.

[0154] like Figure 3 As shown, one possible procedure for this verification experiment is as follows:

[0155] Step 1: Multimodal Data Acquisition and Preprocessing

[0156] First, in a simulated flexible manufacturing workshop environment, a multimodal perception module is activated, simultaneously collecting voice commands, facial expressions, and gesture data from operators using a microphone array, high-definition camera, and depth sensor. The collected data is synchronized through an event-driven temporal coding mechanism to ensure temporal consistency across different modalities. Subsequently, noise reduction filtering techniques are applied to clean the voice data, image enhancement algorithms are used to optimize the quality of the visual data, and pose estimation algorithms are employed to extract key feature points of gesture movements. This preprocessing aims to improve the accuracy and reliability of the data, laying a solid foundation for subsequent feature extraction and analysis.

[0157] Step 2: Feature Extraction and Encoding

[0158] After data preprocessing, the feature processing and selection module automatically selects the most suitable Transformer model for high-dimensional feature extraction based on the characteristics of different modalities. Specifically, speech data is processed using the Wav2Vec2-Transformer model to extract semantic and pitch features, facial expressions are processed using the Vision Transformer (ViT) model to obtain detailed visual features, and gestures are processed using the TimeSformer model to capture dynamically changing gesture information from the time-series video stream. The extracted modal features are then standardized using feature derivation and coding units to generate high-dimensional feature vectors in a unified format. This process ensures the semantic consistency and complementarity of features from different modalities, providing a rich information foundation for multimodal fusion.

[0159] Step 3: Multimodal Fusion and Task Planning

[0160] Next, the multimodal fusion module utilizes the Cross-Attention mechanism to deeply fuse features from different modalities, dynamically adjusting the weights of each modality during the fusion process to achieve semantic alignment and consistent representation. The fused features are mapped to a joint embedding space to comprehensively understand the operator's multimodal instructions. The task understanding and planning module then uses a task decoder to parse the fused features into specific task instruction sequences and employs a heuristic graph search algorithm to construct a task graph, planning the optimal task execution path. In a flexible manufacturing environment, this algorithm can dynamically adjust the task path based on real-time feedback, ensuring that the robot can efficiently and flexibly complete multi-step assembly tasks and adapt to constantly changing production demands.

[0161] Step 4: Optimization of Execution Control and Real-time Feedback

[0162] After task planning is completed, the execution control module generates specific robot control instructions based on the planning results, guiding the robot to perform assembly actions. During execution, the robot monitors the task status and environmental changes in real time through sensors, collecting feedback information. This feedback data is input into the reinforcement learning optimization unit, which guides the robot to optimize its execution strategy through a set reward function (such as task completion, execution efficiency, and user satisfaction). The reinforcement learning algorithm continuously adjusts the robot's action selection based on real-time feedback, improving the accuracy and efficiency of task execution and ensuring the high efficiency and stability of the assembly process.

[0163] Step 5: Continuous Learning and Knowledge Sharing

[0164] Finally, the continuous learning and collaboration module utilizes meta-learning and federated learning techniques to enable robots to continuously optimize their behavioral strategies during task execution. Meta-learning allows robots to quickly adapt to new tasks and environmental changes, adjusting model parameters with only a small amount of data to improve adaptability. Simultaneously, the federated learning platform uploads the optimization strategies and experiences learned locally by each robot to the cloud for global model aggregation and optimization, then distributes them back to the robots, achieving knowledge sharing and collaborative optimization. This mechanism not only enhances the intelligence level of individual robots but also strengthens the collaborative efficiency and intelligence level of the entire robot group, ensuring the system continues to perform excellently in flexible manufacturing environments.

[0165] In the flexible robot control system based on multimodal data and heuristic graph search in this invention embodiment, by introducing multimodal feature processing technology that automatically selects the Transformer model, the robot can efficiently analyze and fuse data from multiple perception channels such as voice, gestures, and facial expressions, significantly improving the accuracy of understanding the operator's intentions. Simultaneously, by combining task planning and path optimization with heuristic graph search algorithms, the system can quickly adjust strategies during the dynamic execution of multi-step tasks to cope with frequently changing process requirements in flexible manufacturing scenarios. Furthermore, the system integrates reinforcement learning and continuous learning mechanisms, enabling the robot to adaptively adjust its operating strategies during task execution and continuously optimize its behavior in a constantly changing environment, thereby achieving intelligent, adaptive, and efficient assembly task execution. This invention has significant advantages in improving the naturalness of human-machine interaction, task execution efficiency, and environmental adaptability, and can meet the demands of modern manufacturing for high automation and flexibility.

[0166] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0167] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned flexible robot control method based on multimodal data and heuristic graph search. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0168] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] refer to Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0170] The processor 701 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0171] The memory 702 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 702 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 702 and called and executed by the processor 701. (The last sentence appears to be incomplete and possibly refers to a different implementation of the invention: "A flexible robot control method based on multimodal data and heuristic graph search according to an embodiment of the present invention.")

[0172] The input / output interface 703 is used to implement information input and output;

[0173] The communication interface 704 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0174] Bus 705 transmits information between various components of the device (e.g., processor 701, memory 702, input / output interface 703, and communication interface 704);

[0175] The processor 701, memory 702, input / output interface 703, and communication interface 704 are connected to each other within the device via bus 705.

[0176] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0177] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0178] This invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0179] In summary, the flexible robot control method and system based on multimodal data and heuristic graph search of this invention has the following advantages:

[0180] 1. Traditional robots, when faced with multi-step, dynamically changing assembly tasks in flexible manufacturing environments, often rely on preset rules or simple search algorithms, lacking sufficient flexibility and adaptability, and struggling to cope with frequently changing production demands. This invention, through the introduction of a heuristic graph search algorithm, constructs a task graph and performs dynamic path optimization, achieving efficient planning and execution of complex assembly tasks. This algorithm can dynamically adjust the task path based on real-time feedback, ensuring the robot maintains efficient and flexible task execution capabilities in changing production environments, significantly improving production efficiency and assembly quality.

[0181] 2. In flexible manufacturing environments, the external conditions and task requirements of the production line often change. Existing robot systems lack effective adaptive learning mechanisms, making it difficult to quickly adjust their operating strategies. This invention combines reinforcement learning and meta-learning techniques, enabling robots to continuously optimize their execution strategies based on real-time feedback (e.g., changes in the flexible manufacturing environment such as lighting, noise, and equipment status). Federated learning facilitates knowledge sharing and collaborative optimization among multiple robots. Through a reinforcement learning optimization model, robots can adjust their strategies based on task completion and user satisfaction during the task. This continuous learning and optimization mechanism allows robots to quickly adapt to new tasks and conditions in dynamic environments, significantly improving the system's intelligence level and long-term operational stability. Furthermore, the continuous learning mechanism enables robots to continuously learn and accumulate experience during execution, enhancing their adaptability to new tasks and environments through meta-learning and federated learning. This allows robots to automatically optimize their task execution strategies, reducing human intervention and significantly improving the efficiency and quality of assembly tasks.

[0182] 3. The multimodal data processing and intelligent task planning in this embodiment of the invention enable the robot to better understand and respond to the operator's needs. It can not only accurately execute instructions but also adjust its response based on the operator's emotional state and interaction habits. For example, the robot can proactively offer assistance when it detects operator fatigue, or adjust the tone and content of its voice feedback according to different users' preferences. This enhanced human-machine collaboration and personalized interaction experience not only improves the operator's work efficiency and satisfaction but also makes the robot's role in human-machine cooperation more intelligent and user-friendly.

[0183] 4. This invention employs a high-performance computing platform and an advanced deep learning framework, combined with an automated Transformer model selection and optimization strategy, to achieve efficient processing and real-time analysis of multimodal data. Through optimized feature extraction and fusion mechanisms, the system can significantly reduce computing resource consumption while maintaining high accuracy, and improve data processing speed and real-time response. This advantage enables robots to quickly respond to operational commands and adjust task execution strategies in a timely manner during actual production, ensuring the efficient operation of the production line.

[0184] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0185] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0186] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0187] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0188] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0189] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0190] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0191] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0192] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A flexible robot control method based on multimodal data and heuristic graph search, characterized in that, Includes the following steps: Acquire multimodal data; Feature extraction is performed on the multimodal data to obtain multimodal features; The multimodal features are fused to obtain fused features; Based on the fusion features, obtain the target task path map; Based on the target task path diagram, control the target flexible robot to execute the target task; The multimodal features include speech features, facial expression features, and gesture features; The step of obtaining the target task path map based on the fusion features includes: The fused features are mapped to the joint embedding space to obtain the embedded features; The embedded features are parsed using a task decoder to obtain a sequence of task instructions; Based on the sequence of task instructions, construct an initial task path diagram; Based on the initial task path graph, the actual cost from the starting node to the current node is obtained, and the minimum cost from the current node to the target node is estimated using a heuristic function to obtain the heuristic estimated cost. Based on the actual cost and the heuristically estimated cost, construct an evaluation function; Minimize the evaluation function to obtain the target task path graph; The local optimization strategies of multiple robots are aggregated by a federated learning model to generate a global model, and the updated global model parameters are distributed to the target flexible robot to update the model parameters of the target flexible robot. The expression for the global model parameters is: ； In the formula, These are global model parameters; For the number of robots, For the first The amount of data per robot This represents the total amount of data from the robots used, which is the sum of the number of data samples from all robots.

2. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1, characterized in that, It also includes the following steps: The model parameters of the target flexible robot are updated according to the target task.

3. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1, characterized in that, The process of extracting features from the multimodal data to obtain multimodal features includes the following steps: Select the corresponding Transformer model based on the modality type of the multimodal data; Based on the Transformer model, feature extraction is performed on the multimodal data to obtain the multimodal features.

4. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1, characterized in that, The process of fusing the multimodal features to obtain fused features includes the following steps: Based on the speech features, facial expression features, and gesture features, attention weights are obtained; Based on the attention weights, the speech features, facial expression features, and gesture features are fused using a cross-attention mechanism to obtain the fused features.

5. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1, characterized in that, The step of controlling the target flexible robot to execute the target task according to the target task path diagram includes the following steps: Based on the target task path diagram, generate control commands; In response to the control command, the target flexible robot is controlled to perform the target task.

6. The flexible robot control method based on multimodal data and heuristic graph search according to claim 5, characterized in that, After the step of controlling the target flexible robot to execute the target task according to the target task path diagram, the following steps are also included: Get the current environment status; Construct a reward function based on the control instructions and the current environmental state; The execution strategy is updated based on the reward function.

7. A flexible robot control system based on multimodal data and heuristic graph search, characterized in that, include: The first module is used to acquire multimodal data; The second module is used to extract features from the multimodal data to obtain multimodal features; The third module is used to perform a fusion operation on the multimodal features to obtain fused features; The fourth module is used to obtain the target task path map based on the fusion features; The fifth module is used to control the target flexible robot to perform the target task according to the target task path diagram; The multimodal features include speech features, facial expression features, and gesture features; The fourth module is specifically used for: The fused features are mapped to the joint embedding space to obtain the embedded features; The embedded features are parsed using a task decoder to obtain a sequence of task instructions; Based on the sequence of task instructions, construct an initial task path diagram; Based on the initial task path graph, the actual cost from the starting node to the current node is obtained, and the minimum cost from the current node to the target node is estimated using a heuristic function to obtain the heuristic estimated cost. Based on the actual cost and the heuristically estimated cost, construct an evaluation function; Minimize the evaluation function to obtain the target task path graph; The local optimization strategies of multiple robots are aggregated by a federated learning model to generate a global model, and the updated global model parameters are distributed to the target flexible robot to update the model parameters of the target flexible robot. The expression for the global model parameters is: ； In the formula, These are global model parameters; For the number of robots, For the first The amount of data per robot This represents the total amount of data from the robots used, which is the sum of the number of data samples from all robots.

8. A flexible robot control system based on multimodal data and heuristic graph search according to claim 7, characterized in that, Also includes: The sixth module is used to update the model parameters of the target flexible robot according to the target task.

9. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.

Images