Flexible robot control method and system based on multi-modal data and heuristic graph search

Abstract

The invention discloses a flexible robot control method and system based on multi-modal data and heuristic graph search. The method comprises the following steps: acquiring multi-modal data; performing feature extraction on the multi-modal data to obtain multi-modal features; performing fusion operation on the multi-modal features to obtain fusion features; obtaining a target task path diagram according to the fusion features; controlling a target flexible robot to execute a target task according to the target task path diagram; wherein the multi-modal features comprise voice features, expression features and gesture features. The task execution efficiency and flexibility of the robot can be improved, and the method can be widely applied to the technical field of machine learning.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and in particular to a flexible robot control method and system based on multimodal data and heuristic graph search. Background Art

[0002] In a flexible manufacturing environment, production lines need to adapt to rapidly changing product types and process flows to achieve efficient customized production. In this context, industrial robots, as core components of automated production, are gradually taking on more and more tasks. However, traditional industrial robots often lack the ability to understand and adapt to complex scenarios, especially when faced with frequently changing assembly tasks, they often cannot meet actual production needs. Traditional industrial robots have obvious deficiencies in multimodal data processing, and can usually only process single-modal data (such as vision or voice), lacking the ability to fully perceive and understand complex human-computer interaction information. This single-modal processing method makes the robot lack flexibility in receiving and understanding user instructions, and it is difficult to accurately understand the operator's voice instructions, gestures and facial expressions, resulting in an interaction process that is not natural and smooth enough. Existing robot task planning methods mostly use rule-based preset paths or traditional search algorithms. These methods often show low efficiency and adaptability when faced with changes in task scenarios or complex multi-step tasks. In addition, the existing technology usually relies on preset processing rules when dealing with changes in the external environment of the production line, and it is difficult to adjust its strategy in real time, thus affecting the smooth completion of the task. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide a flexible robot control method and system based on multimodal data and heuristic graph search, in order to solve at least one of the problems of the prior art. The present invention can improve the efficiency and flexibility of the robot in performing tasks.

[0004] To achieve the above object, an embodiment of the present invention provides a flexible robot control method based on multimodal data and heuristic graph search, comprising the following steps:

[0005] Acquire multimodal data;

[0006] Performing feature extraction on the multimodal data to obtain multimodal features;

[0007] Performing a fusion operation on the multimodal features to obtain fused features;

[0008] According to the fusion features, a target task path diagram is obtained;

[0009] According to the target task path diagram, controlling the target flexible robot to perform the target task;

[0010] The multimodal features include voice features, facial expression features and gesture features.

[0011] In some embodiments, a flexible robot control method based on multimodal data and heuristic graph search further includes the following steps:

[0012] According to the target task, the model parameters of the target flexible robot are updated.

[0013] In some embodiments, extracting features from the multimodal data to obtain multimodal features comprises the following steps:

[0014] According to the modality type of the multimodal data, select a corresponding Transformer model;

[0015] According to the Transformer model, feature extraction is performed on the multimodal data to obtain the multimodal features.

[0016] In some embodiments, the fusing operation on the multimodal features to obtain the fused features comprises the following steps:

[0017] Obtaining attention weights according to the voice features, the facial expression features, and the gesture features;

[0018] According to the attention weight, the speech feature, the expression feature and the gesture feature are fused through a cross-attention mechanism to obtain the fused feature.

[0019] In some embodiments, obtaining a target task path map according to the fusion feature comprises the following steps:

[0020] Mapping the fused features to a joint embedding space to obtain embedded features;

[0021] Parsing the embedded features through a task decoder to obtain a task instruction sequence;

[0022] Constructing an initial task path diagram according to the task instruction sequence;

[0023] Constructing an evaluation function according to the initial task path diagram;

[0024] Minimize the evaluation function to obtain the target task path diagram.

[0025] In some embodiments, controlling the target flexible robot to perform the target task according to the target task path diagram comprises the following steps:

[0026] Generate control instructions according to the target task path diagram;

[0027] In response to the control instruction, the target flexible robot is controlled to perform the target task.

[0028] In some embodiments, after the step of controlling the target flexible robot to perform the target task according to the target task path diagram, the following steps are also included:

[0029] Get the current environment status;

[0030] Constructing a reward function according to the control instruction and the current environment state;

[0031] The execution strategy is updated according to the reward function.

[0032] To achieve the above object, another aspect of an embodiment of the present invention provides a flexible robot control system based on multimodal data and heuristic graph search, the system comprising:

[0033] The first module is used to obtain multimodal data;

[0034] The second module is used to extract features from the multimodal data to obtain multimodal features;

[0035] The third module is used to perform a fusion operation on the multimodal features to obtain fused features;

[0036] The fourth module is used to obtain a target task path diagram according to the fusion features;

[0037] The fifth module is used to control the target flexible robot to perform the target task according to the target task path diagram.

[0038] In some embodiments, a flexible robot control system based on multimodal data and heuristic graph search further includes:

[0039] The sixth module is used to update the model parameters of the target flexible robot according to the target task.

[0040] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the flexible robot control method based on multimodal data and heuristic graph search as described above.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned flexible robot control method based on multimodal data and heuristic graph search.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0043] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a flexible robot control method and system based on multimodal data and heuristic graph search, which obtains multimodal data; extracts features from the multimodal data to obtain multimodal features; fuses the multimodal features to obtain fused features; obtains a target task path graph based on the fused features; and controls the target flexible robot to perform the target task based on the target task path graph, thereby improving the efficiency and flexibility of the robot in performing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flow chart of a flexible robot control method based on multimodal data and heuristic graph search provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the structure of a flexible robot control system based on multimodal data and heuristic graph search provided by an embodiment of the present invention;

[0047] Figure 3 It is an overall operation flow chart of a flexible robot control system based on multimodal data and heuristic graph search provided by an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0050] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0051] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0053] In a flexible manufacturing environment, production lines need to adapt to rapidly changing product types and process flows to achieve efficient customized production. In this context, industrial robots, as core components of automated production, are gradually taking on more and more tasks. However, traditional industrial robots often lack the ability to understand and adapt to complex scenarios, especially when faced with frequently changing assembly tasks, and often cannot meet actual production needs.

[0054] The introduction of multimodal data processing technology provides new possibilities for human-machine interaction and task understanding. However, although multimodal data processing technology improves the perception ability of robots, in actual applications, assembly tasks in flexible manufacturing environments usually have multi-step and dynamically changing characteristics, requiring robots to have higher-level task planning and execution capabilities. Traditional rule-based planning methods often lack flexibility and adaptability when faced with uncertainty and complex tasks. Existing production line scheduling technologies have the following main shortcomings in multi-robot collaboration and dynamic scheduling:

[0055] Insufficient multimodal data processing capabilities: Traditional industrial robots have obvious deficiencies in multimodal data processing. They can usually only process single-mode data (such as vision or voice) and lack the ability to fully perceive and understand complex human-machine interaction information. This single-mode processing method makes the robot lack flexibility in receiving and understanding user instructions, and it is difficult to accurately understand the operator's voice instructions, gestures and facial expressions, resulting in an unnatural and unsmooth interaction process.

[0056] Low flexibility and efficiency in task planning: Existing robot task planning methods mostly use rule-based preset paths or traditional search algorithms. These methods often show low efficiency and adaptability when faced with changing task scenarios or complex multi-step tasks. In a flexible manufacturing environment, task processes may change frequently, such as changes in product specifications and adjustments to process requirements. Traditional methods require frequent reprogramming or manual adjustments in these dynamically changing scenarios, lack the ability to optimize in real time, and lead to reduced production efficiency.

[0057] Limited adaptability to environmental changes: In a flexible manufacturing environment, the external conditions of the production line (such as lighting changes, noise interference, equipment failure, etc.) may affect the robot's task execution. Existing technologies usually rely on preset processing rules when dealing with these environmental changes, and it is difficult to adjust their strategies in real time. In particular, when faced with unexpected situations during task execution, the robot lacks adaptive learning capabilities and cannot quickly adjust its actions or strategies based on real-time feedback, which affects the smooth completion of the task.

[0058] Lack of intelligent optimization methods in task execution: Existing robot systems often lack intelligent task optimization methods and cannot optimize themselves according to the real-time progress and feedback of tasks. Although some systems have introduced simple feedback mechanisms, they cannot achieve optimization and continuous learning of multi-step tasks in complex environments, and it is difficult to improve the efficiency and quality of long-term task execution.

[0059] In view of this, if Figure 1As shown, an embodiment of the present invention provides a flexible robot control method based on multimodal data and heuristic graph search, which may include but is not limited to steps S100 to S500:

[0060] Step S100, acquiring multimodal data;

[0061] Step S200, extracting features from the multimodal data to obtain multimodal features;

[0062] Step S300, performing a fusion operation on the multimodal features to obtain fused features;

[0063] Step S400, obtaining a target task path diagram according to the fusion features;

[0064] Step S500, controlling the target flexible robot to perform the target task according to the target task path diagram;

[0065] The multimodal features include voice features, facial expression features and gesture features.

[0066] In steps S100 to S500 of some embodiments, by integrating data of multiple modalities such as voice, expression, gesture, etc. and performing feature extraction, in-depth analysis and fusion of multimodal information is achieved. Task planning and path optimization are then performed, so that the robot can adaptively complete complex assembly tasks in a flexible manufacturing environment and make real-time adjustments based on environmental changes. Compared with traditional systems, this solution has significantly improved the accuracy of task understanding, adaptive capabilities, and efficiency of task execution, and can better meet the high requirements of modern manufacturing for automation and flexibility. Among them,

[0067] In step S100 of some embodiments, multimodal data refers to information from different types of sensors, such as voice, gesture, expression, environmental image, etc. By processing multimodal data, the robot can achieve information fusion and understanding between different perception channels, thereby improving the ability to understand the operator's intentions and making human-computer interaction more natural and efficient. Exemplarily, multimodal data such as voice, gesture and expression of the operator can be collected through a variety of sensors. These sensors may include a high-sensitivity microphone array for capturing voice commands, a high-definition camera and a depth sensor for acquiring facial expressions and gestures. In order to ensure the temporal and spatial consistency of data in different modalities, the system uses an event-driven time encoding mechanism for synchronous processing, aligns data in different modalities, and ensures synchronization. In addition, the multimodal data is preprocessed. In the data preprocessing stage, it can include preprocessing the voice data using noise reduction filtering and speech enhancement technology to improve the quality of the voice signal; denoising, enhancing and normalizing the image and video, extracting key feature points, and thus preprocessing the visual data to ensure the accuracy and effectiveness of subsequent feature extraction.

[0068] In some embodiments, step S200 may include but is not limited to steps S210 to S220:

[0069] Step S210, selecting a corresponding Transformer model according to the modality type of the multimodal data;

[0070] Step 8220: Perform feature extraction on the multimodal data according to the Transformer model to obtain the multimodal features.

[0071] In steps S210 to S220 of some embodiments, the most suitable Transformer model is automatically selected according to the modal type and features of the input multimodal data. For example, for speech data, Wav2Vec2.0 Transformer can be selected to extract semantic and tonal features in speech; for facial expression data, Vision Transformer (ViT) can be selected to obtain fine visual features; for gesture data, TimeSformer can be selected to process time-series video streams and extract features of dynamic gestures, but is not limited to this. The feature selection controller can dynamically adjust the selection and configuration of the Transformer model according to real-time environmental conditions (such as lighting and noise levels) to ensure efficient and accurate feature extraction of different modal data. The generated high-dimensional feature vector provides a rich information basis for subsequent multimodal fusion.

[0072] In some embodiments, feature extraction may include the following processes:

[0073] 1) Speech feature extraction: Input speech signal x audio , extract high-dimensional features through the Wav2Vec 2.0 model:

[0074] h audio =Wav2Vec2(x audio )

[0075] 2) Expression feature extraction: Input expression image x face , extract features through the ViT model:

[0076] h face =ViT(x face )

[0077] 3) Gesture feature extraction: Input gesture video sequence X gesture ={x 1 , x 2 , ..., x T}, extract features through the TimeSformer model:

[0078] h gesture =TimeSformer(X gesture )

[0079] Optionally, the high-dimensional feature representation obtained by processing the original speech signal through the Wav2Vec 2.0 model contains the semantic information and intonation features in the speech, reflecting the content of the operator's command input through voice. The feature representation obtained by processing the facial image through the Vision Transformer (ViT) model contains the operator's expression information, which can help the robot understand the operator's emotional state and intention. The time series features obtained by processing the gesture action sequence (video stream) through the TimeSformer model reflect the operator's gesture instructions. These features are used to parse the specific task requirements conveyed by the operator through gestures.

[0080] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0081] Step S310, obtaining an attention weight according to the voice feature, the expression feature and the gesture feature;

[0082] Step S320, according to the attention weight, through the cross-attention mechanism, the voice feature, the expression feature and the gesture feature are fused to obtain the fused feature.

[0083] In step S310 of some embodiments, the speech feature is used as the query (Query, Q), the expression feature is used as the key (Key, K), and the gesture feature is used as the value (Value, V), and the attention weight Attention (Q, K, V) can be calculated:

[0084]

[0085] Where Q = W Q h audio ; K = W K [h faace ,h gesture ]; V = W V [h faace ,h gesture ]

[0086] Where W Q , W K , W V is the learnable parameter matrix; d k is the dimension of the key vector; h audio is the speech feature; h face h is the facial expression feature; gesture The gesture feature.

[0087] In step S320 of some embodiments, according to the attention weights Attention(Q, K, V), the features of different modalities are deeply fused through the Cross-Attention mechanism to obtain the fused feature h fusion :h fusion =Attention(Q, K, V). The Cross-Attention mechanism deeply fuses the features extracted from different modalities, establishes dynamic associations between multimodal features, and automatically adjusts the weights of each modality in the fusion process to ensure that important information is processed first. Through the optimized Cross-Attention mechanism, the robot can establish associations between different modalities, improve its ability to understand the operator's intentions, and make human-machine interaction more natural and smooth.

[0088] In some embodiments, step S400 may include but is not limited to steps S410 to S450:

[0089] Step S410, mapping the fused features to a joint embedding space to obtain embedded features;

[0090] Step S420, parsing the embedded features through a task decoder to obtain a task instruction sequence;

[0091] Step S430, constructing an initial task path diagram according to the task instruction sequence;

[0092] Step S440, constructing an evaluation function according to the initial task path diagram;

[0093] Step S450, minimizing the evaluation function to obtain the target task path diagram.

[0094] In step S410 of some embodiments, the fused features are mapped to the joint embedding space to obtain the embedded features h embedding , to achieve semantic alignment. For example, the expression of embedding features is: embedding =φ(h fusion ), where φ(·) is a nonlinear mapping function. Optionally, the nonlinear mapping function can be a fully connected layer plus an activation function. By mapping the fused multimodal features to a unified joint embedding space to achieve semantic alignment and consistency, the system can fully understand the operator's multimodal instructions, extract a comprehensive intent representation, and provide a solid foundation for task understanding and planning.

[0095] In step S420 of some embodiments, the complex natural language instructions and gesture actions are parsed into a specific task step sequence by a task decoder. Exemplarily, the task decoder is used to parse the embedded features to obtain a task instruction sequence. Optionally, the process expression for converting the embedded features into the task instruction sequence Instruction is:

[0096] Instruction = ψ(h embedding )

[0097] Among them, ψ(·) is the task decoder; the task instruction sequence Instruction={T 1 , T 2 , ..., T N}.

[0098] In steps S430 to S450 of some embodiments, a heuristic graph search algorithm is used to plan and optimize the task steps. The heuristic graph search algorithm constructs a task graph, nodes the task steps, and searches and selects the optimal path based on the priority and execution cost of the task. The algorithm can dynamically adjust the task execution path in a flexible manufacturing environment, ensure that the robot can efficiently and flexibly respond to the changing assembly task requirements, and improve the efficiency and flexibility of the overall task execution. Exemplarily, according to the task instruction sequence, the node T representing each step of the assembly task is obtained. i , i∈{1, 2, ..., N}, and obtain the edge representing the transfer relationship between steps, where the weight of the edge is the execution cost c ij. Based on each node and each edge, the initial task path graph can be constructed. Optionally, the actual cost g(n) from the starting node to the current node n is obtained; the minimum cost from the current node n to the target node is estimated by constructing a heuristic function h(n), and the heuristic estimated cost h(n) is obtained. Then the evaluation function f(n) = g(n) + h(n) can be constructed by the A* algorithm. By minimizing the evaluation function f(n), the optimal path search can be achieved, and the optimal task execution path from the starting node to the target node can be found to form the target task path graph.

[0099] In some embodiments, step S500 may include but is not limited to steps S510 to S520:

[0100] Step S510, generating a control instruction according to the target task path diagram;

[0101] Step S520, in response to the control instruction, controlling the target flexible robot to perform the target task.

[0102] In steps S510 to S520 of some embodiments, the corresponding robot control instructions a may be generated according to the task sequence planned in the target task path diagram. t Then, the robot actuator can perform the corresponding physical actions according to the control instructions.

[0103] After step S500 in some embodiments, the following steps may be further included: obtaining the current environment state; constructing a reward function according to the control instruction and the current environment state; and updating the execution strategy according to the reward function. t The reward function can be designed based on factors such as task completion and user satisfaction. Optionally, the control instruction a t and the current environment status s t , construct the reward function r t =R(s t , a t ). According to the reward function, the execution strategy Q(s) can be t , a t ) is updated, and the expression of the update process is:

[0104]

[0105] Among them, α is the learning rate, γ is the discount factor, a′ is the possible next action, and s t+1 is the environmental state at time t+1.

[0106] By choosing Q(s t , a t) The largest action a t , and get the robot update strategy π(a t |s t ).

[0107] During the task execution, the robot uses sensors to monitor the task status and environmental changes in real time and collect feedback information. The system combines reinforcement learning algorithms to adjust the execution strategy based on real-time feedback to optimize the efficiency and accuracy of task execution. Reinforcement learning drives the robot to continuously improve its behavior choices during the trial and error process by setting a reward function, enabling the robot to continuously improve its task execution capabilities in a dynamic environment and ensure the efficient completion of assembly tasks.

[0108] In some embodiments, the flexible robot control method based on multimodal data and heuristic graph search may also include but is not limited to step S600, updating the model parameters of the target flexible robot according to the target task. Through meta-learning rapid adaptation and federated learning model aggregation, the model parameters of the target flexible robot are updated to achieve continuous learning and knowledge sharing. For example,

[0109] 1) Meta-learning for fast adaptation

[0110] Metamodel training: Train the metamodel on multiple target tasks to enable it to quickly adapt to new tasks.

[0111] Task adaptation: In the new target task T new On the other hand, using a small amount of data to quickly update the model parameters, we have the following expression:

[0112]

[0113] Among them, θ is the initial meta-model parameter, β is the meta-learning rate, and θ new is the updated target metamodel parameter, For the new target task T new The loss function on .

[0114] 2) Federated Learning Model Aggregation

[0115] Local model update: Each robot updates the model parameters θ based on local data i .

[0116] Global model aggregation: The server aggregates all local model parameters to generate a global model:

[0117]

[0118] Where K is the number of robots, n iis the data volume of the ith robot, and n is the total data volume of the robots used, that is, the sum of the data samples of all robots.

[0119] Model distribution: Distribute the updated global model parameters Φ to each robot.

[0120] Through meta-learning and federated learning technologies, robots can achieve adaptive learning and knowledge sharing during task execution. Meta-learning enables robots to quickly adapt to new tasks and environmental changes, and only requires a small amount of data to adjust model parameters, thereby improving their ability to adapt to new tasks. Federated learning uploads the experience and optimization strategies learned locally by each robot to the cloud through a cloud-based knowledge sharing platform, aggregates them to generate a global model, and distributes it back to each robot. This collaborative learning mechanism not only improves the intelligence level of a single robot, but also enhances the collaborative efficiency and intelligence level of the entire robot group, realizing the accumulation and inheritance of knowledge.

[0121] like Figure 2 As shown, an embodiment of the present invention further provides a flexible robot control system based on multimodal data and heuristic graph search, which can implement the above-mentioned flexible robot control method based on multimodal data and heuristic graph search, and the system includes:

[0122] The first module (multimodal perception module) is used to obtain multimodal data;

[0123] The second module (feature processing and selection module) is used to extract features from the multimodal data to obtain multimodal features;

[0124] The third module (multimodal fusion module) is used to perform a fusion operation on the multimodal features to obtain fusion features;

[0125] The fourth module (task planning and optimization module) is used to obtain a target task path diagram according to the fusion features;

[0126] The fifth module (execution control module) is used to control the target flexible robot to perform the target task according to the target task path diagram;

[0127] The sixth module (continuous learning and cloud collaboration module) is used to update the model parameters of the target flexible robot according to the target task.

[0128] Among them, the multimodal perception module includes:

[0129] A speech perception unit, equipped with a microphone array, is used to capture and process the operator's voice commands in real time;

[0130] A visual perception unit, equipped with a high-definition camera and depth sensor, is used to capture and process the operator's facial expressions and gestures in real time;

[0131] The data synchronization unit is used to perform temporal synchronization processing on the data from different perception units to ensure the temporal consistency of multimodal data.

[0132] The feature processing and selection module includes:

[0133] A feature selection controller that automatically selects the most suitable Transformer model based on the modality type of the input data and the real-time environmental conditions;

[0134] The speech Transformer model uses Wav2Vec2-Transformer to process speech data and extract semantic and tonal features;

[0135] The Vision Transformer model (ViT) is used to process facial expression data and extract delicate visual features.

[0136] The gesture Transformer model uses TimeSformer to process gesture action data and extract dynamically changing time series features.

[0137] The multimodal fusion module includes:

[0138] The Cross-Attention fusion unit is equipped with a Cross-Attention mechanism, which is used to establish dynamic associations between different modal features and automatically adjust the weights of each modality during the fusion process;

[0139] The joint embedding space builder is used to map the fused multimodal features into a unified embedding space to achieve semantic alignment and consistent representation.

[0140] Mission planning and optimization modules include:

[0141] A heuristic graph search algorithm unit, configured with a heuristic graph search algorithm, for converting the parsed task instructions into a task graph and planning an optimal task execution path based on a heuristic search method;

[0142] The dynamic task adjustment unit is used to dynamically adjust the task execution path according to real-time feedback to ensure efficient and flexible completion of tasks.

[0143] The execution control module includes:

[0144] An action control unit, used to generate specific robot control instructions according to the instructions of the task planning module;

[0145] A real-time feedback unit, which is used to monitor the task execution status and environmental changes through sensors and transmit feedback information to the task planning and optimization module;

[0146] Human-machine interaction unit for real-time interaction with operators, including voice confirmation and gesture feedback.

[0147] The continuous learning and cloud collaboration modules include:

[0148] A reinforcement learning optimization unit, which is configured with a reinforcement learning algorithm and is used to continuously optimize the robot execution strategy through the reinforcement learning algorithm based on task completion, execution efficiency, and user satisfaction;

[0149] A meta-learning unit, which is equipped with a meta-learning technique and is used to enable the robot to quickly adapt to new tasks and environmental changes through meta-learning;

[0150] The federated learning platform is used to upload the optimization strategies and experiences learned locally by each robot to the cloud, aggregate and optimize the global model, and distribute the updated global model back to each robot to achieve knowledge sharing and collaborative optimization.

[0151] Through the mutual collaboration of various modules and units, efficient processing of multimodal data, intelligent planning and execution of tasks, and adaptive learning and optimization are achieved.

[0152] In some embodiments, a flexible robot control system based on multimodal data and heuristic graph search is verified in a simulated flexible manufacturing workshop environment, using a multi-joint humanoid robot as the experimental subject. The robot is equipped with a highly sensitive microphone array, a high-definition camera, and a depth sensor to capture the operator's voice commands, facial expressions, and gestures. In terms of the computing platform, the experiment uses a high-performance server equipped with an NVIDIA A100 GPU, running an operating system based on Ubuntu 20.04LTS, and uses the PyTorch 2.0 framework to build and train multimodal Transformer models, including Wav2Vec2, Vision Transformer (ViT), and TimeSformer. In addition, the experiment also integrates a custom heuristic graph search algorithm and a Proximal Policy Optimization (PPO) reinforcement learning algorithm to support task planning and execution optimization. The experimental scenario design includes a variety of assembly tasks, such as assembly, tightening, and quality inspection of different types of parts, simulating the changing needs in actual flexible manufacturing. In order to test the adaptive ability of the system, dynamic changing factors such as illumination changes, background noise, and equipment failures are introduced into the experimental environment to observe the performance of the robot under different environmental conditions. At the same time, by setting up multiple rounds of human-computer interaction, the system's response speed and accuracy in voice command recognition, gesture control, and expression understanding are evaluated. The real-time sensor feedback mechanism ensures that the robot can obtain task execution status and environmental change information in a timely manner for dynamic adjustment and optimization.

[0153] In order to comprehensively evaluate the system performance, the experiment used a number of evaluation indicators, including task completion rate, response time, learning speed and interaction satisfaction. The task completion rate measures the proportion of the robot successfully completing the assembly task within the specified time, the response time evaluates the delay from receiving the instruction to executing the action, the learning speed is measured by recording the adaptation and optimization process of the system in different task scenarios, and the interaction satisfaction is evaluated by the operator's subjective rating of the robot's interaction experience. Through these indicators, the experiment aims to verify the effectiveness and advantages of this technical solution in multimodal data processing, task planning and execution, adaptive learning, etc., and provide a basis for further optimization and application of the system.

[0154] like Figure 3 As shown, an optional step process of the verification experiment is as follows:

[0155] Step 1: Multimodal data collection and preprocessing

[0156] First, in a simulated flexible manufacturing workshop environment, the multimodal perception module is started, and the operator's voice commands, facial expressions and gesture data are collected simultaneously using microphone arrays, high-definition cameras and depth sensors. The collected data is synchronized through an event-driven time encoding mechanism to ensure the temporal consistency of different modal data. Subsequently, the voice data is cleaned up using noise reduction filtering technology, the quality of visual data is optimized using image enhancement algorithms, and the key feature points of gestures are extracted using posture estimation algorithms. This preprocessing process aims to improve the accuracy and reliability of the data and lay a solid foundation for subsequent feature extraction and analysis.

[0157] Step 2: Feature extraction and encoding

[0158] After completing the data preprocessing, the feature processing and selection module automatically selects the most appropriate Transformer model to extract high-dimensional features according to the characteristics of different modal data. Specifically, the speech data is extracted with the Wav2Vec2-Transformer model to extract semantic and tonal features, the facial expression is obtained through the Vision Transformer (ViT) model to obtain delicate visual features, and the gesture movement is processed through the TimeSformer model to process the time series video stream to capture dynamically changing gesture information. The extracted features of each modality are standardized through feature derivation and coding units to generate high-dimensional feature vectors in a unified format. This process ensures the consistency and complementarity of the features of different modal data at the semantic level, providing a rich information basis for multimodal fusion.

[0159] Step 3: Multimodal Fusion and Task Planning

[0160] Next, the multimodal fusion module uses the Cross-Attention mechanism to deeply fuse the features of different modalities, dynamically adjust the weights of each modality in the fusion process, and achieve semantic alignment and consistent representation. The fused features are mapped to a joint embedding space to fully understand the operator's multimodal instructions. The task understanding and planning module then parses the fused features into a specific task instruction sequence through the task decoder, and uses a heuristic graph search algorithm to construct a task graph and plan the optimal task execution path. In a flexible manufacturing environment, the algorithm can dynamically adjust the task path based on real-time feedback, ensuring that the robot can efficiently and flexibly complete multi-step assembly tasks and adapt to changing production needs.

[0161] Step 4: Execution control and real-time feedback optimization

[0162] After the task planning is completed, the execution control module generates specific robot control instructions based on the planning results to guide the robot to perform assembly actions. During the execution process, the robot monitors the task status and environmental changes in real time through sensors and collects feedback information. These feedback data are input into the reinforcement learning optimization unit, which guides the robot to optimize its execution strategy through the set reward function (such as task completion, execution efficiency, and user satisfaction). The reinforcement learning algorithm continuously adjusts the robot's action selection based on real-time feedback, improves the accuracy and efficiency of task execution, and ensures the efficiency and stability of the assembly process.

[0163] Step 5: Continuous learning and knowledge sharing

[0164] Finally, the continuous learning and collaboration module uses meta-learning and federated learning technologies to enable robots to continuously optimize their behavioral strategies during task execution. Meta-learning enables robots to quickly adapt to new tasks and environmental changes, and only requires a small amount of data to adjust model parameters and improve adaptability. At the same time, the federated learning platform uploads the optimization strategies and experiences learned locally by each robot to the cloud, aggregates and optimizes the global model, and then distributes it back to each robot to achieve knowledge sharing and collaborative optimization. This mechanism not only improves the intelligence level of a single robot, but also enhances the collaborative efficiency and intelligence level of the entire robot group, ensuring that the system continues to perform well in a flexible manufacturing environment.

[0165] In the flexible robot control system based on multimodal data and heuristic graph search of the embodiment of the present invention, by introducing the multimodal feature processing technology of automatically selecting the Transformer model, the robot can efficiently parse and fuse data from multiple perception channels such as voice, gestures, and expressions, significantly improving the accuracy of understanding the operator's intentions. At the same time, combined with the heuristic graph search algorithm for task planning and path optimization, it is possible to quickly adjust the strategy during the dynamic execution of multi-step tasks to cope with the frequently changing process requirements in flexible manufacturing scenarios. In addition, the system integrates reinforcement learning and continuous learning mechanisms, and the robot can adaptively adjust the operation strategy during the task execution process, and continuously optimize its own behavior in a constantly changing environment, thereby realizing intelligent, adaptive and efficient assembly task execution. The present invention has significant advantages in improving the naturalness of human-computer interaction, task execution efficiency and environmental adaptability, and can meet the needs of modern manufacturing for high automation and flexibility.

[0166] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0167] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned flexible robot control method based on multimodal data and heuristic graph search is implemented. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0168] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0169] refer to Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0170] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0171] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes a flexible robot control method based on multimodal data and heuristic graph search according to an embodiment of the present invention;

[0172] Input / output interface 703, used to implement information input and output;

[0173] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0174] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0175] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0176] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned flexible robot control method based on multimodal data and heuristic graph search.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the aforementioned flexible robot control method based on multimodal data and heuristic graph search.

[0179] In summary, the flexible robot control method and system based on multimodal data and heuristic graph search according to the embodiment of the present invention has the following advantages:

[0180] 1. When faced with multi-step, dynamically changing assembly tasks in a flexible manufacturing environment, traditional robots often rely on preset rules or simple search algorithms, lacking sufficient flexibility and adaptability, and are difficult to cope with frequently changing production needs. The embodiments of the present invention achieve efficient planning and execution of complex assembly tasks by introducing a heuristic graph search algorithm, constructing a task graph and dynamic path optimization. The algorithm can dynamically adjust the task path based on real-time feedback, ensuring that the robot maintains efficient and flexible task execution capabilities in a changing production environment, greatly improving production efficiency and assembly quality.

[0181] 2. In a flexible manufacturing environment, the external conditions and task requirements of the production line often change. The existing robot system lacks an effective adaptive learning mechanism and is difficult to quickly adjust its operation strategy. The embodiment of the present invention combines reinforcement learning with meta-learning technology to enable the robot to continuously optimize the execution strategy according to real-time feedback (for example, according to changes in the flexible manufacturing environment, such as changes in light, noise, equipment status, etc.), and realize knowledge sharing and collaborative optimization among multiple robots through federated learning. Through the reinforcement learning optimization model, the robot can adjust its strategy according to task completion and user satisfaction during the task process. This continuous learning and optimization mechanism enables the robot to have the ability to quickly adapt to new tasks and new conditions in a dynamic environment, significantly improves the intelligence level of the system and the stability of long-term operation, and through the continuous learning mechanism, the robot can continuously learn and accumulate experience during the execution process, and improve its adaptability in new tasks and environments through methods such as meta-learning and federated learning, so that the robot can automatically optimize its task execution strategy, reduce human intervention, and thus significantly improve the efficiency and quality of assembly tasks.

[0182] 3. The multimodal data processing and intelligent task planning of the embodiments of the present invention enable the robot to better understand and respond to the needs of the operator, not only to accurately execute instructions, but also to adjust the response method according to the operator's emotional state and interaction habits. For example, the robot can actively provide help when it detects operator fatigue, or adjust the tone and content of voice feedback according to the preferences of different users. This enhanced human-machine collaboration and personalized interaction experience not only improves the work efficiency and satisfaction of the operator, but also makes the role of the robot in human-machine collaboration more intelligent and friendly.

[0183] 4. The embodiment of the present invention adopts a high-performance computing platform and an advanced deep learning framework, combined with an automated Transformer model selection and optimization strategy, to achieve efficient processing and real-time analysis of multimodal data. Through the optimized feature extraction and fusion mechanism, the system can significantly reduce the consumption of computing resources while ensuring high accuracy, and improve the speed of data processing and the real-time response. This advantage enables the robot to quickly respond to operating instructions in actual production, adjust the task execution strategy in a timely manner, and ensure the efficient operation of the production line.

[0184] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0185] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0186] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0187] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0188] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0189] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0190] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0191] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0192] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A flexible robot control method based on multimodal data and heuristic graph search, characterized in that: The following steps are involved: Acquire multimodal data; Performing feature extraction on the multimodal data to obtain multimodal features; Performing a fusion operation on the multimodal features to obtain fused features; According to the fusion features, a target task path diagram is obtained; According to the target task path diagram, controlling the target flexible robot to perform the target task; The multimodal features include voice features, facial expression features and gesture features.

2. A flexible robot control method based on multimodal data and heuristic graph search according to claim 1, characterized in that: The following steps are also included: According to the target task, the model parameters of the target flexible robot are updated.

3. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1 is characterized in that: The step of extracting features from the multimodal data to obtain multimodal features comprises the following steps: According to the modality type of the multimodal data, select a corresponding Transformer model; According to the Transformer model, feature extraction is performed on the multimodal data to obtain the multimodal features.

4. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1 is characterized in that: The step of fusing the multimodal features to obtain fused features comprises the following steps: Obtaining attention weights according to the voice features, the facial expression features, and the gesture features; According to the attention weight, the speech feature, the expression feature and the gesture feature are fused through a cross-attention mechanism to obtain the fused feature.

5. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1 is characterized in that: The step of obtaining a target task path diagram according to the fusion features comprises the following steps: Mapping the fused features to a joint embedding space to obtain embedded features; Parsing the embedded features through a task decoder to obtain a task instruction sequence; Constructing an initial task path diagram according to the task instruction sequence; Constructing an evaluation function according to the initial task path diagram; Minimize the evaluation function to obtain the target task path diagram.

6. The flexible robot control method based on multimodal data and heuristic graph search according to claim 1 is characterized in that: The step of controlling the target flexible robot to perform the target task according to the target task path diagram comprises the following steps: Generate control instructions according to the target task path diagram; In response to the control instruction, the target flexible robot is controlled to perform the target task.

7. A flexible robot control method based on multimodal data and heuristic graph search according to claim 6, characterized in that: After the step of controlling the target flexible robot to perform the target task according to the target task path diagram, the following steps are also included: Get the current environment status; Constructing a reward function according to the control instruction and the current environment state; The execution strategy is updated according to the reward function.

8. A flexible robot control system based on multimodal data and heuristic graph search, characterized in that: include: The first module is used to obtain multimodal data; The second module is used to extract features from the multimodal data to obtain multimodal features; The third module is used to perform a fusion operation on the multimodal features to obtain fused features; The fourth module is used to obtain a target task path diagram according to the fusion features; The fifth module is used to control the target flexible robot to perform the target task according to the target task path diagram.

9. The flexible robot control system based on multimodal data and heuristic graph search according to claim 8, characterized in that: Also includes: The sixth module is used to update the model parameters of the target flexible robot according to the target task.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

Images