Multi-modal fusion robot task flow orchestration and dynamic adaptation method and system

Abstract

The application provides a multi-modal fusion robot task flow arrangement and dynamic adaptation method and system. The method comprises the following steps: monitoring a preset task flow and the operation of a target robot, collecting real-time data of each mode, environmental change data and real-time task scene data, obtaining historical task data of the target robot, processing the real-time data of each mode, the environmental change data and the real-time task scene data, obtaining mode weight parameters, screening the mode weight parameters, performing fusion processing to obtain fused multi-modal data, processing the preset task flow to obtain an optimal task execution sequence, performing a task flow arrangement and dynamic adaptation scheme on the preset task flow according to the fused multi-modal data and the optimal task execution sequence, monitoring the operation of the scheme in a preset time period, extracting operation index data and evaluating the effectiveness of the scheme, thereby realizing the multi-modal fusion robot task flow arrangement and dynamic adaptation technology.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and more specifically, to a method and system for multimodal fusion robot task flow orchestration and dynamic adaptation. Background Technology

[0002] In the field of robotics applications, task execution often involves multimodal information such as vision, hearing, and touch. However, in existing technologies, robot task flow orchestration is mostly based on fixed patterns, which cannot flexibly adjust the task execution order according to real-time environment and task requirements. Furthermore, the fusion and processing of multimodal data is not intelligent enough, resulting in low task execution efficiency and poor adaptability. At the same time, in the face of environmental changes and task scenario switching, robots have difficulty optimizing task processes in real time, and cannot meet the needs of efficient and accurate operation in complex scenarios.

[0003] Therefore, there is an urgent need for a method that can effectively fuse multimodal data and achieve dynamic orchestration and adaptation of task flows to improve the robot's task execution capability and environmental adaptability in complex scenarios. Summary of the Invention

[0004] The purpose of this application is to provide a method and system for multimodal fusion robot task flow orchestration and dynamic adaptation. This method monitors the operation of a preset task flow and the target robot, collects real-time data from each modality, environmental change data, and real-time task scenario data, obtains historical task data of the target robot, processes the real-time data from each modality, environmental change data, and real-time task scenario data to obtain modal weight parameters, filters these parameters, and performs fusion processing to obtain fused multimodal data. The preset task flow is then processed to obtain the optimal task execution order. Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow. The operation of the scheme is monitored within a preset time period, operational indicator data is extracted, and the effectiveness of the scheme is evaluated, thereby realizing the technology of multimodal fusion robot task flow orchestration and dynamic adaptation.

[0005] This application also provides a method for multimodal fusion robot task flow orchestration and dynamic adaptation, including the following steps:

[0006] Monitor the preset task flow and the operation of the target robot, and collect real-time data of each modality, environmental change data and real-time task scenario data;

[0007] Historical task data of the target robot is acquired and processed in combination with real-time data of each modality, environmental change data and real-time task scenario data to obtain the weight parameters of each modality.

[0008] The modal weight parameters are filtered and fused to obtain fused multimodal data;

[0009] The preset task flow is processed to obtain the optimal task execution order;

[0010] Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow;

[0011] Monitor the operation of the scheme within a preset time period, extract operational indicator data, and evaluate the effectiveness of the scheme.

[0012] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation method described in this application, the monitoring of the preset task flow and the operation of the target robot, and the collection of real-time data of each modality, environmental change data, and real-time task scenario data, include:

[0013] Monitor preset task flows and collect environmental change data as well as real-time task scenario data;

[0014] The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data;

[0015] The real-time task scenario data includes real-time task target data and real-time task instruction data;

[0016] Monitor the operation of the target robot and collect real-time data for each modality, including real-time data for visual modality, auditory modality, tactile modality, motion and positioning modality, and environmental perception modality.

[0017] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation method described in this application, the step of obtaining historical task data of the target robot, and processing it in combination with real-time data of each modality, environmental change data, and real-time task scenario data to obtain the weight parameters of each modality includes:

[0018] Acquire historical task data of the target robot, including task type data, task result data, raw sensor data, and environmental parameter data;

[0019] Based on the historical task data, combined with the real-time data of each modality, environmental change data, and real-time task scenario data, the data is processed using a preset gradient descent algorithm to obtain the weight parameters of each modality.

[0020] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation method described in this application, the step of filtering the modal weight parameters and performing fusion processing to obtain fused multimodal data includes:

[0021] Get the preset threshold parameter;

[0022] Each modality weight parameter is compared with the preset threshold parameter;

[0023] If the modal weight parameter is less than the preset threshold parameter, then the modal data corresponding to the modal weight parameter does not meet the requirements;

[0024] If the modal weight parameter is greater than or equal to the preset threshold parameter, then the modal data corresponding to the modal weight parameter meets the requirements and is marked as modal data to be fused.

[0025] The fused multimodal data is obtained by combining the modal data to be fused with the corresponding modal weight parameters.

[0026] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation method described in this application, the step of processing the preset task flow to obtain the optimal task execution order includes:

[0027] Generate a task flow topology graph based on the preset task flow;

[0028] Obtain constraints, including resource constraints, dynamic priorities, and parallel tasks;

[0029] The optimal task execution order is generated by processing the task flow topology graph using a preset topology sorting algorithm and constraints.

[0030] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation method described in this application, the step of monitoring the operation of the scheme within a preset time period, extracting operation index data, and evaluating the effectiveness of the scheme includes:

[0031] Monitor the operation of the scheme within a preset time period and extract operation indicator data, including multimodal fusion effect data, task flow orchestration efficiency data, and dynamic adaptation performance data.

[0032] The multimodal fusion effect data includes accuracy and recall, which are weighted to obtain the fusion accuracy coefficient.

[0033] The task flow orchestration efficiency data includes the average task completion time and resource utilization rate, which are processed through a preset orchestration efficiency evaluation model to obtain the orchestration rationality coefficient.

[0034] The dynamic adaptation performance data includes task execution success rate and task execution response time, which are processed by a preset adaptation performance evaluation model to obtain the adaptation effectiveness coefficient.

[0035] The performance index coefficient is obtained by weighting the fusion accuracy coefficient, the arrangement rationality coefficient, and the adaptation effectiveness coefficient.

[0036] The effectiveness of the scheme is evaluated by comparing the effectiveness index coefficient with the preset effectiveness index threshold.

[0037] Secondly, this application provides a multimodal fusion robot task flow orchestration and dynamic adaptation system. The system includes a memory and a processor. The memory includes a program for a multimodal fusion robot task flow orchestration and dynamic adaptation method. When the program for the multimodal fusion robot task flow orchestration and dynamic adaptation method is executed by the processor, it performs the following steps:

[0038] Monitor the preset task flow and the operation of the target robot, and collect real-time data of each modality, environmental change data and real-time task scenario data;

[0039] Historical task data of the target robot is acquired and processed in combination with real-time data of each modality, environmental change data and real-time task scenario data to obtain the weight parameters of each modality.

[0040] The modal weight parameters are filtered and fused to obtain fused multimodal data;

[0041] The preset task flow is processed to obtain the optimal task execution order;

[0042] Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow;

[0043] Monitor the operation of the scheme within a preset time period, extract operational indicator data, and evaluate the effectiveness of the scheme.

[0044] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation system described in this application, the monitoring of the preset task flow and the operation of the target robot, and the collection of real-time data of each modality, environmental change data, and real-time task scenario data, include:

[0045] Monitor preset task flows and collect environmental change data as well as real-time task scenario data;

[0046] The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data;

[0047] The real-time task scenario data includes real-time task target data and real-time task instruction data;

[0048] Monitor the operation of the target robot and collect real-time data for each modality, including real-time data for visual modality, auditory modality, tactile modality, motion and positioning modality, and environmental perception modality.

[0049] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation system described in this application, the step of acquiring historical task data of the target robot, and processing it in conjunction with real-time data of each modality, environmental change data, and real-time task scenario data to obtain the weight parameters of each modality includes:

[0050] Acquire historical task data of the target robot, including task type data, task result data, raw sensor data, and environmental parameter data;

[0051] Based on the historical task data, combined with the real-time data of each modality, environmental change data, and real-time task scenario data, the data is processed using a preset gradient descent algorithm to obtain the weight parameters of each modality.

[0052] Optionally, in the multimodal fusion robot task flow orchestration and dynamic adaptation system described in this application, the step of filtering the modal weight parameters and performing fusion processing to obtain fused multimodal data includes:

[0053] Get the preset threshold parameter;

[0054] Each modality weight parameter is compared with the preset threshold parameter;

[0055] If the modal weight parameter is less than the preset threshold parameter, then the modal data corresponding to the modal weight parameter does not meet the requirements;

[0056] If the modal weight parameter is greater than or equal to the preset threshold parameter, then the modal data corresponding to the modal weight parameter meets the requirements and is marked as modal data to be fused.

[0057] The fused multimodal data is obtained by combining the modal data to be fused with the corresponding modal weight parameters.

[0058] As can be seen from the above, the multimodal fusion robot task flow orchestration and dynamic adaptation method and system provided in this application monitors the operation of the preset task flow and the target robot, collects real-time data of each modality, environmental change data, and real-time task scenario data, obtains historical task data of the target robot, processes the real-time data of each modality, environmental change data, and real-time task scenario data to obtain the weight parameters of each modality, filters the weight parameters of each modality, and performs fusion processing to obtain fused multimodal data, processes the preset task flow to obtain the optimal task execution order, executes the task flow orchestration and dynamic adaptation scheme on the preset task flow according to the fused multimodal data and the optimal task execution order, monitors the operation of the scheme within a preset time period, extracts operation index data and evaluates the effectiveness of the scheme, thereby realizing the technology of multimodal fusion robot task flow orchestration and dynamic adaptation.

[0059] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0060] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating the multimodal fusion robot task flow orchestration and dynamic adaptation method provided in this application embodiment;

[0062] Figure 2 A flowchart illustrating the process of obtaining the modal weight parameters for the multimodal fusion robot task flow orchestration and dynamic adaptation method provided in this application embodiment;

[0063] Figure 3 A flowchart illustrating the process of obtaining fused multimodal data in the multimodal fusion robot task flow orchestration and dynamic adaptation method provided in this application embodiment;

[0064] Figure 4 The flowchart illustrates how the multimodal fusion robot task flow orchestration and dynamic adaptation method provided in this application obtains the optimal task execution order. Detailed Implementation

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0066] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0067] Please refer to Figure 1 , Figure 1 This is a flowchart of a multimodal fusion robot task flow orchestration and dynamic adaptation method according to some embodiments of this application. This multimodal fusion robot task flow orchestration and dynamic adaptation method is used in terminal devices, such as computers and mobile terminals. The multimodal fusion robot task flow orchestration and dynamic adaptation method includes the following steps:

[0068] S11. Monitor the preset task flow and the operation of the target robot, and collect real-time data of each modality, environmental change data and real-time task scenario data;

[0069] S12. Obtain historical task data of the target robot, and process it in combination with real-time data of each modality, environmental change data and real-time task scenario data to obtain weight parameters of each modality.

[0070] S13. Filter the modal weight parameters and perform fusion processing to obtain fused multimodal data;

[0071] S14. Process the preset task flow to obtain the optimal task execution order;

[0072] S15. Based on the fused multimodal data and the optimal task execution order, perform a task flow orchestration and dynamic adaptation scheme on the preset task flow;

[0073] S16. Monitor the operation of the scheme within a preset time period, extract operation indicator data, and evaluate the effectiveness of the scheme.

[0074] It should be noted that, given the increasing demand for intelligent robot operation, it is necessary to integrate multi-source data to optimize modal weights, plan task sequences, and implement dynamic adaptation schemes to achieve scientific management and effectiveness evaluation of the robot's task execution process. Therefore, by monitoring the preset task flow and the target robot's operation, real-time data from each modality, environmental change data, and real-time task scenario data are collected to obtain the target robot's historical task data, including task type data, task result data, raw sensor data, and environmental parameter data. This data, combined with the real-time data from each modality, environmental change data, and real-time task scenario data, is processed using a preset gradient descent algorithm to obtain... The modal weight parameters are obtained, and each modal weight parameter is compared with a preset threshold parameter and filtered. Then, the data is fused to obtain fused multimodal data. The preset task flow is processed by a preset topology sorting algorithm and constraints to obtain the optimal task execution order. Based on the fused multimodal data and the optimal task execution order, the preset task flow orchestration and dynamic adaptation scheme is executed. The operation of the scheme is monitored within a preset time period, and operation index data, including multimodal fusion effect data, task flow orchestration efficiency data, and dynamic adaptation performance data, are extracted. The effectiveness of the scheme is evaluated, thereby realizing the technology of multimodal fusion robot task flow orchestration and dynamic adaptation.

[0075] According to an embodiment of the present invention, the monitoring of the preset task flow and the operation of the target robot, and the collection of real-time data of each modality, environmental change data, and real-time task scenario data, include:

[0076] Monitor preset task flows and collect environmental change data as well as real-time task scenario data;

[0077] The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data;

[0078] The real-time task scenario data includes real-time task target data and real-time task instruction data;

[0079] Monitor the operation of the target robot and collect real-time data for each modality, including real-time data for visual modality, auditory modality, tactile modality, motion and positioning modality, and environmental perception modality.

[0080] It should be noted that the monitoring system tracks the preset task flow and collects real-time environmental change data and real-time task scenario data. The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data. Temperature and humidity change rate data reflect dynamic fluctuations in environmental temperature and humidity, helping the robot predict equipment operational risks. Obstacle movement speed data assists the robot in planning obstacle avoidance paths in advance. Target object position offset data provides accurate positioning data for the robot to perform tasks such as grasping and tracking. Real-time task scenario data includes real-time task objective data, clearly defining the specific goals the robot needs to achieve, such as transporting designated goods or reaching a specific coordinate point. Real-time task instruction data... These are the operational instructions issued by the operator or system to guide the robot's actions. Simultaneously, it is necessary to closely monitor the target robot's operational status and comprehensively collect real-time data from each modality. Specifically, real-time visual modality data is obtained by capturing image frames from cameras, which are then processed to output the identification and positioning information of the target object. Real-time auditory modality data is obtained by collecting ambient sounds through microphones to detect abnormal noises or receive voice commands. Real-time tactile modality data relies on force sensors and other devices to provide feedback on the magnitude and distribution of contact forces. Real-time motion and positioning modality data is provided by inertial measurement units, odometry, and other devices to provide robot posture and displacement information. Real-time environmental perception modality data utilizes lidar, ultrasonic sensors, and other technologies to acquire information on the distance to surrounding obstacles and changes in environmental parameters.

[0081] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the process of obtaining modal weight parameters in a multimodal fusion robot task flow orchestration and dynamic adaptation method according to some embodiments of this application. According to embodiments of the present invention, obtaining historical task data of the target robot and processing it in conjunction with real-time data of each modality, environmental change data, and real-time task scenario data to obtain modal weight parameters includes:

[0082] S21. Obtain historical task data of the target robot, including task type data, task result data, raw sensor data, and environmental parameter data;

[0083] S22. Based on the historical task data, combined with the real-time data of each modality, environmental change data, and real-time task scenario data, the data is processed using a preset gradient descent algorithm to obtain the weight parameters of each modality.

[0084] It should be noted that, to optimize decision-making, the historical task data of the target robot is acquired, including task type data, task result data, raw sensor data, and environmental parameter data. Task type data clearly defines the task attributes, such as handling, inspection, and assembly. Task result data records the task completion status, indicating success or failure and the quality of completion. Raw sensor data retains the unprocessed raw information collected by each modality's sensors, providing a basis for retrospective analysis. Environmental parameter data includes environmental indicators such as temperature, humidity, and light intensity during task execution. Then, this is combined with real-time data from each modality (visual, auditory, tactile, motion and localization, and environmental perception) and environmental change data (temperature changes...). The algorithm processes real-time task scenario data (real-time task objectives and instructions) including humidity change rate, obstacle movement speed, and target object position offset, using a pre-defined gradient descent algorithm. First, a loss function is constructed to quantify the deviation between the current decision result and the expected goal. Then, the gradient descent algorithm iteratively calculates the gradient direction of the weights of each modality data, adjusting the weight parameters in the direction that minimizes the loss function. For example, when performing inspection tasks in complex lighting environments, if the visual modality data has a large error, the algorithm will reduce its weight and relatively increase the weight ratio of other modal data such as LiDAR, ultimately obtaining the optimal weight parameters for each modality that are suitable for the current task scenario.

[0085] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the process of obtaining fused multimodal data in a robot task flow orchestration and dynamic adaptation method for multimodal fusion according to some embodiments of this application. According to embodiments of the present invention, the step of filtering the modal weight parameters and performing fusion processing to obtain fused multimodal data includes:

[0086] S31. Obtain the preset threshold parameter;

[0087] S32. Compare each modal weight parameter with the preset threshold parameter respectively;

[0088] S33. If the modal weight parameter is less than the preset threshold parameter, then the modal data corresponding to the modal weight parameter does not meet the requirements.

[0089] S34. If the modal weight parameter is greater than or equal to the preset threshold parameter, then the modal data corresponding to the modal weight parameter meets the requirements and is marked as modal data to be fused.

[0090] S35. Perform fusion processing based on the modal data to be fused and the corresponding modal weight parameters to obtain fused multimodal data.

[0091] It should be noted that, firstly, a preset threshold parameter is obtained. This parameter is set comprehensively based on the robot's hardware performance, task type, and historical operating data, and is usually in the form of an interval or critical value. Next, each modal weight parameter is compared with the preset threshold parameter. If the modal weight parameter is less than the preset threshold parameter, it indicates that the reliability or contribution of the modal data in the current task scenario is insufficient and cannot meet the task requirements. For example, when the robot operates in a dense fog environment, if the weight parameter of the visual modality is lower than the threshold, it means that the image data collected by the camera is greatly affected by visibility and cannot provide effective information; its proportion in decision-making needs to be reduced. If the modal weight parameter is greater than or equal to the preset threshold parameter, the modal data is determined to meet the requirements and is marked as modal data to be fused. For example, in complex terrain navigation, if the weight parameter of LiDAR is higher than the threshold, it indicates that the environmental distance information it collects is stable and reliable and can be included in subsequent fusion calculations. Finally, the modal data to be fused is fused in combination with the corresponding modal weight parameters. Through weighted summation or more complex fusion algorithms, the various modal data are integrated into unified multimodal data.

[0092] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the process of obtaining the optimal task execution order using a multimodal fusion robot task flow orchestration and dynamic adaptation method in some embodiments of this application. According to an embodiment of the present invention, processing the preset task flow to obtain the optimal task execution order includes:

[0093] S41. Generate a task flow topology diagram based on the preset task flow;

[0094] S42. Obtain constraints, including resource constraints, dynamic priorities, and parallel tasks;

[0095] S43. Based on the task flow topology graph, the optimal task execution order is generated by processing it through a preset topology sorting algorithm and constraints.

[0096] It should be noted that, firstly, a task flow topology graph is generated based on the preset task flow, breaking down tasks into nodes and using directed edges to represent the sequential dependencies between tasks. For example, in a warehouse handling scenario, cargo scanning and path planning are preceding nodes, while cargo picking and transportation are subsequent nodes, forming a clear task logic chain. Next, constraints are obtained, including resource constraints, dynamic priorities, and parallel tasks. Resource constraints mainly consider hardware limitations, such as a robotic arm only being able to perform a single operation at a time. Dynamic priorities are adjusted according to the urgency of the tasks, such as prioritizing equipment failure repair tasks over routine inspections. Parallel tasks identify tasks without dependencies, such as scanning and weighing during sorting. Finally, based on the task flow topology graph, preset topology sorting algorithms such as the Kahn algorithm or depth-first search are used for optimization in conjunction with the constraints. The algorithm prioritizes nodes without preceding dependencies and that meet resource allocation requirements. For conflicting tasks, the order is dynamically adjusted according to priority. For example, when resources are scarce, high-priority tasks are prioritized, while tasks without dependencies are processed in parallel, ultimately generating the optimal task execution order that balances efficiency and resource utilization.

[0097] According to an embodiment of the present invention, monitoring the operation of the scheme within a preset time period, extracting operational indicator data, and evaluating the effectiveness of the scheme includes:

[0098] Monitor the operation of the scheme within a preset time period and extract operation indicator data, including multimodal fusion effect data, task flow orchestration efficiency data, and dynamic adaptation performance data.

[0099] The multimodal fusion effect data includes accuracy and recall, which are weighted to obtain the fusion accuracy coefficient.

[0100] The task flow orchestration efficiency data includes the average task completion time and resource utilization rate, which are processed through a preset orchestration efficiency evaluation model to obtain the orchestration rationality coefficient.

[0101] The dynamic adaptation performance data includes task execution success rate and task execution response time, which are processed by a preset adaptation performance evaluation model to obtain the adaptation effectiveness coefficient.

[0102] The performance index coefficient is obtained by weighting the fusion accuracy coefficient, the arrangement rationality coefficient, and the adaptation effectiveness coefficient.

[0103] The effectiveness of the scheme is evaluated by comparing the effectiveness index coefficient with the preset effectiveness index threshold.

[0104] It is important to note that in robot operation management, quantitative evaluation of the program's operation is a crucial basis for optimization decisions. Specifically, the program must first be monitored throughout a pre-defined time period, extracting key operational metrics in real time, including multimodal fusion performance data, task flow orchestration efficiency data, and dynamic adaptation performance data. Multimodal fusion performance data focuses on the quality of data integration, measured by accuracy and recall. Accuracy reflects the proportion of correct information in the fused data, while recall reflects the completeness of capturing important information. These two metrics are weighted to form a fusion accuracy coefficient. Task flow orchestration efficiency data focuses on evaluating task execution effectiveness, including average task completion time and resource utilization. The system employs a pre-defined orchestration efficiency evaluation model to comprehensively calculate these two data points, yielding an orchestration rationality coefficient. Dynamic adaptation performance data focuses on the robot's responsiveness to changes in the environment and tasks, selecting task execution success rate and task execution response time as evaluation criteria. Using a pre-defined adaptation performance evaluation model, these two indicators are transformed into adaptation effectiveness coefficients. Finally, the fusion accuracy coefficient, orchestration rationality coefficient, and adaptation effectiveness coefficient are weighted to generate an effectiveness index coefficient. This coefficient is compared with a pre-defined effectiveness index threshold. If it exceeds the threshold, the solution is deemed effective; otherwise, targeted optimization of the multimodal fusion strategy, task orchestration algorithm, or dynamic adaptation mechanism is required to continuously improve the robot's operational efficiency.

[0105] Secondly, the present invention also discloses a multimodal fusion robot task flow orchestration and dynamic adaptation system, including a memory and a processor. The memory includes a multimodal fusion robot task flow orchestration and dynamic adaptation method program. When the multimodal fusion robot task flow orchestration and dynamic adaptation method program is executed by the processor, it performs the following steps:

[0106] Monitor the preset task flow and the operation of the target robot, and collect real-time data of each modality, environmental change data and real-time task scenario data;

[0107] Historical task data of the target robot is acquired and processed in combination with real-time data of each modality, environmental change data and real-time task scenario data to obtain the weight parameters of each modality.

[0108] The modal weight parameters are filtered and fused to obtain fused multimodal data;

[0109] The preset task flow is processed to obtain the optimal task execution order;

[0110] Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow;

[0111] Monitor the operation of the scheme within a preset time period, extract operational indicator data, and evaluate the effectiveness of the scheme.

[0112] It should be noted that, given the increasing demand for intelligent robot operation, it is necessary to integrate multi-source data to optimize modal weights, plan task sequences, and implement dynamic adaptation schemes to achieve scientific management and effectiveness evaluation of the robot's task execution process. Therefore, by monitoring the preset task flow and the target robot's operation, real-time data from each modality, environmental change data, and real-time task scenario data are collected to obtain the target robot's historical task data, including task type data, task result data, raw sensor data, and environmental parameter data. This data, combined with the real-time data from each modality, environmental change data, and real-time task scenario data, is processed using a preset gradient descent algorithm to obtain... The modal weight parameters are obtained, and each modal weight parameter is compared with a preset threshold parameter and filtered. Then, the data is fused to obtain fused multimodal data. The preset task flow is processed by a preset topology sorting algorithm and constraints to obtain the optimal task execution order. Based on the fused multimodal data and the optimal task execution order, the preset task flow orchestration and dynamic adaptation scheme is executed. The operation of the scheme is monitored within a preset time period, and operation index data, including multimodal fusion effect data, task flow orchestration efficiency data, and dynamic adaptation performance data, are extracted. The effectiveness of the scheme is evaluated, thereby realizing the technology of multimodal fusion robot task flow orchestration and dynamic adaptation.

[0113] According to an embodiment of the present invention, the monitoring of the preset task flow and the operation of the target robot, and the collection of real-time data of each modality, environmental change data, and real-time task scenario data, include:

[0114] Monitor preset task flows and collect environmental change data as well as real-time task scenario data;

[0115] The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data;

[0116] The real-time task scenario data includes real-time task target data and real-time task instruction data;

[0117] Monitor the operation of the target robot and collect real-time data for each modality, including real-time data for visual modality, auditory modality, tactile modality, motion and positioning modality, and environmental perception modality.

[0118] It should be noted that the monitoring system tracks the preset task flow and collects real-time environmental change data and real-time task scenario data. The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data. Temperature and humidity change rate data reflect dynamic fluctuations in environmental temperature and humidity, helping the robot predict equipment operational risks. Obstacle movement speed data assists the robot in planning obstacle avoidance paths in advance. Target object position offset data provides accurate positioning data for the robot to perform tasks such as grasping and tracking. Real-time task scenario data includes real-time task objective data, clearly defining the specific goals the robot needs to achieve, such as transporting designated goods or reaching a specific coordinate point. Real-time task instruction data... These are the operational instructions issued by the operator or system to guide the robot's actions. Simultaneously, it is necessary to closely monitor the target robot's operational status and comprehensively collect real-time data from each modality. Specifically, real-time visual modality data is obtained by capturing image frames from cameras, which are then processed to output the identification and positioning information of the target object. Real-time auditory modality data is obtained by collecting ambient sounds through microphones to detect abnormal noises or receive voice commands. Real-time tactile modality data relies on force sensors and other devices to provide feedback on the magnitude and distribution of contact forces. Real-time motion and positioning modality data is provided by inertial measurement units, odometry, and other devices to provide robot posture and displacement information. Real-time environmental perception modality data utilizes lidar, ultrasonic sensors, and other technologies to acquire information on the distance to surrounding obstacles and changes in environmental parameters.

[0119] According to an embodiment of the present invention, the step of acquiring historical task data of the target robot, and processing it in conjunction with real-time data of each modality, environmental change data, and real-time task scenario data to obtain modal weight parameters includes:

[0120] Acquire historical task data of the target robot, including task type data, task result data, raw sensor data, and environmental parameter data;

[0121] Based on the historical task data, combined with the real-time data of each modality, environmental change data, and real-time task scenario data, the data is processed using a preset gradient descent algorithm to obtain the weight parameters of each modality.

[0122] It should be noted that, to optimize decision-making, the historical task data of the target robot is acquired, including task type data, task result data, raw sensor data, and environmental parameter data. Task type data clearly defines the task attributes, such as handling, inspection, and assembly. Task result data records the task completion status, indicating success or failure and the quality of completion. Raw sensor data retains the unprocessed raw information collected by each modality's sensors, providing a basis for retrospective analysis. Environmental parameter data includes environmental indicators such as temperature, humidity, and light intensity during task execution. Then, this is combined with real-time data from each modality (visual, auditory, tactile, motion and localization, and environmental perception) and environmental change data (temperature changes...). The algorithm processes real-time task scenario data (real-time task objectives and instructions) including humidity change rate, obstacle movement speed, and target object position offset, using a pre-defined gradient descent algorithm. First, a loss function is constructed to quantify the deviation between the current decision result and the expected goal. Then, the gradient descent algorithm iteratively calculates the gradient direction of the weights of each modality data, adjusting the weight parameters in the direction that minimizes the loss function. For example, when performing inspection tasks in complex lighting environments, if the visual modality data has a large error, the algorithm will reduce its weight and relatively increase the weight ratio of other modal data such as LiDAR, ultimately obtaining the optimal weight parameters for each modality that are suitable for the current task scenario.

[0123] According to an embodiment of the present invention, the step of filtering the modal weight parameters and performing fusion processing to obtain fused multimodal data includes:

[0124] Get the preset threshold parameter;

[0125] Each modality weight parameter is compared with the preset threshold parameter;

[0126] If the modal weight parameter is less than the preset threshold parameter, then the modal data corresponding to the modal weight parameter does not meet the requirements;

[0127] If the modal weight parameter is greater than or equal to the preset threshold parameter, then the modal data corresponding to the modal weight parameter meets the requirements and is marked as modal data to be fused.

[0128] The fused multimodal data is obtained by combining the modal data to be fused with the corresponding modal weight parameters.

[0129] It should be noted that, firstly, a preset threshold parameter is obtained. This parameter is set comprehensively based on the robot's hardware performance, task type, and historical operating data, and is usually in the form of an interval or critical value. Next, each modal weight parameter is compared with the preset threshold parameter. If the modal weight parameter is less than the preset threshold parameter, it indicates that the reliability or contribution of the modal data in the current task scenario is insufficient and cannot meet the task requirements. For example, when the robot operates in a dense fog environment, if the weight parameter of the visual modality is lower than the threshold, it means that the image data collected by the camera is greatly affected by visibility and cannot provide effective information; its proportion in decision-making needs to be reduced. If the modal weight parameter is greater than or equal to the preset threshold parameter, the modal data is determined to meet the requirements and is marked as modal data to be fused. For example, in complex terrain navigation, if the weight parameter of LiDAR is higher than the threshold, it indicates that the environmental distance information it collects is stable and reliable and can be included in subsequent fusion calculations. Finally, the modal data to be fused is fused in combination with the corresponding modal weight parameters. Through weighted summation or more complex fusion algorithms, the various modal data are integrated into unified multimodal data.

[0130] According to an embodiment of the present invention, processing the preset task flow to obtain the optimal task execution order includes:

[0131] Generate a task flow topology graph based on the preset task flow;

[0132] Obtain constraints, including resource constraints, dynamic priorities, and parallel tasks;

[0133] The optimal task execution order is generated by processing the task flow topology graph using a preset topology sorting algorithm and constraints.

[0134] It should be noted that, firstly, a task flow topology graph is generated based on the preset task flow, breaking down tasks into nodes and using directed edges to represent the sequential dependencies between tasks. For example, in a warehouse handling scenario, cargo scanning and path planning are preceding nodes, while cargo picking and transportation are subsequent nodes, forming a clear task logic chain. Next, constraints are obtained, including resource constraints, dynamic priorities, and parallel tasks. Resource constraints mainly consider hardware limitations, such as a robotic arm only being able to perform a single operation at a time. Dynamic priorities are adjusted according to the urgency of the tasks, such as prioritizing equipment failure repair tasks over routine inspections. Parallel tasks identify tasks without dependencies, such as scanning and weighing during sorting. Finally, based on the task flow topology graph, preset topology sorting algorithms such as the Kahn algorithm or depth-first search are used for optimization in conjunction with the constraints. The algorithm prioritizes nodes without preceding dependencies and that meet resource allocation requirements. For conflicting tasks, the order is dynamically adjusted according to priority. For example, when resources are scarce, high-priority tasks are prioritized, while tasks without dependencies are processed in parallel, ultimately generating the optimal task execution order that balances efficiency and resource utilization.

[0135] According to an embodiment of the present invention, monitoring the operation of the scheme within a preset time period, extracting operational indicator data, and evaluating the effectiveness of the scheme includes:

[0136] Monitor the operation of the scheme within a preset time period and extract operation indicator data, including multimodal fusion effect data, task flow orchestration efficiency data, and dynamic adaptation performance data.

[0137] The multimodal fusion effect data includes accuracy and recall, which are weighted to obtain the fusion accuracy coefficient.

[0138] The task flow orchestration efficiency data includes the average task completion time and resource utilization rate, which are processed through a preset orchestration efficiency evaluation model to obtain the orchestration rationality coefficient.

[0139] The dynamic adaptation performance data includes task execution success rate and task execution response time, which are processed by a preset adaptation performance evaluation model to obtain the adaptation effectiveness coefficient.

[0140] The performance index coefficient is obtained by weighting the fusion accuracy coefficient, the arrangement rationality coefficient, and the adaptation effectiveness coefficient.

[0141] The effectiveness of the scheme is evaluated by comparing the effectiveness index coefficient with the preset effectiveness index threshold.

[0142] It is important to note that in robot operation management, quantitative evaluation of the program's operation is a crucial basis for optimization decisions. Specifically, the program must first be monitored throughout a pre-defined time period, extracting key operational metrics in real time, including multimodal fusion performance data, task flow orchestration efficiency data, and dynamic adaptation performance data. Multimodal fusion performance data focuses on the quality of data integration, measured by accuracy and recall. Accuracy reflects the proportion of correct information in the fused data, while recall reflects the completeness of capturing important information. These two metrics are weighted to form a fusion accuracy coefficient. Task flow orchestration efficiency data focuses on evaluating task execution effectiveness, including average task completion time and resource utilization. The system employs a pre-defined orchestration efficiency evaluation model to comprehensively calculate these two data points, yielding an orchestration rationality coefficient. Dynamic adaptation performance data focuses on the robot's responsiveness to changes in the environment and tasks, selecting task execution success rate and task execution response time as evaluation criteria. Using a pre-defined adaptation performance evaluation model, these two indicators are transformed into adaptation effectiveness coefficients. Finally, the fusion accuracy coefficient, orchestration rationality coefficient, and adaptation effectiveness coefficient are weighted to generate an effectiveness index coefficient. This coefficient is compared with a pre-defined effectiveness index threshold. If it exceeds the threshold, the solution is deemed effective; otherwise, targeted optimization of the multimodal fusion strategy, task orchestration algorithm, or dynamic adaptation mechanism is required to continuously improve the robot's operational efficiency.

[0143] This invention discloses a multimodal fusion robot task flow orchestration and dynamic adaptation method and system. It monitors a preset task flow and the operation of a target robot, collects real-time data from each modality, environmental change data, and real-time task scenario data, obtains historical task data of the target robot, processes the real-time data from each modality, environmental change data, and real-time task scenario data to obtain modal weight parameters, filters these parameters, and performs fusion processing to obtain fused multimodal data. The preset task flow is then processed to obtain the optimal task execution order. Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow. The operation of the scheme is monitored within a preset time period, operational indicator data is extracted, and the effectiveness of the scheme is evaluated, thereby realizing the technology of multimodal fusion robot task flow orchestration and dynamic adaptation.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0145] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0147] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

Claims

1. A multimodal fusion-based robot task flow orchestration and dynamic adaptation method, characterized in that, Includes the following steps: Monitor the preset task flow and the operation of the target robot, and collect real-time data of each modality, environmental change data and real-time task scenario data; Historical task data of the target robot is acquired and processed in combination with real-time data of each modality, environmental change data and real-time task scenario data to obtain the weight parameters of each modality. The modal weight parameters are filtered and fused to obtain fused multimodal data; The preset task flow is processed to obtain the optimal task execution order; Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow; Monitor the operation of the scheme within a preset time period, extract operational indicator data, and evaluate the effectiveness of the scheme; The historical task data of the target robot is acquired and processed in conjunction with the real-time data of each modality, environmental change data, and real-time task scenario data to obtain the weight parameters of each modality, including: Acquire historical task data of the target robot, including task type data, task result data, raw sensor data, and environmental parameter data; Based on the historical task data, combined with the real-time data of each modality, environmental change data, and real-time task scenario data, the weight parameters of each modality are obtained by processing them through a preset gradient descent algorithm. The process of filtering and fusing the modal weight parameters to obtain fused multimodal data includes: Get the preset threshold parameter; Each modality weight parameter is compared with the preset threshold parameter; If the modal weight parameter is less than the preset threshold parameter, then the modal data corresponding to the modal weight parameter does not meet the requirements; If the modal weight parameter is greater than or equal to the preset threshold parameter, then the modal data corresponding to the modal weight parameter meets the requirements and is marked as modal data to be fused. The fused multimodal data is obtained by combining the modal data to be fused with the corresponding modal weight parameters.

2. The multimodal fusion robot task flow orchestration and dynamic adaptation method according to claim 1, characterized in that, The monitoring of the preset task flow and the operation of the target robot collects real-time data of each modality, environmental change data, and real-time task scenario data, including: Monitor preset task flows and collect environmental change data as well as real-time task scenario data; The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data; The real-time task scenario data includes real-time task target data and real-time task instruction data; Monitor the operation of the target robot and collect real-time data for each modality, including real-time data for visual modality, auditory modality, tactile modality, motion and positioning modality, and environmental perception modality.

3. The multimodal fusion robot task flow orchestration and dynamic adaptation method according to claim 2, characterized in that, The step of processing the preset task flow to obtain the optimal task execution order includes: Generate a task flow topology graph based on the preset task flow; Obtain constraints, including resource constraints, dynamic priorities, and parallel tasks; The optimal task execution order is generated by processing the task flow topology graph using a preset topology sorting algorithm and constraints.

4. The multimodal fusion robot task flow orchestration and dynamic adaptation method according to claim 3, characterized in that, The monitoring of the operation of the scheme within a preset time period, the extraction of operational indicator data, and the evaluation of the scheme's effectiveness include: Monitor the operation of the scheme within a preset time period and extract operation indicator data, including multimodal fusion effect data, task flow orchestration efficiency data, and dynamic adaptation performance data. The multimodal fusion effect data includes accuracy and recall, which are weighted to obtain the fusion accuracy coefficient. The task flow orchestration efficiency data includes the average task completion time and resource utilization rate, which are processed through a preset orchestration efficiency evaluation model to obtain the orchestration rationality coefficient. The dynamic adaptation performance data includes task execution success rate and task execution response time, which are processed by a preset adaptation performance evaluation model to obtain the adaptation effectiveness coefficient. The performance index coefficient is obtained by weighting the fusion accuracy coefficient, the arrangement rationality coefficient, and the adaptation effectiveness coefficient. The effectiveness of the scheme is evaluated by comparing the effectiveness index coefficient with the preset effectiveness index threshold.

5. A multimodal fusion robot task flow orchestration and dynamic adaptation system, characterized in that, The system includes a memory and a processor. The memory contains a program for a multimodal fusion robot task flow orchestration and dynamic adaptation method. When the program for the multimodal fusion robot task flow orchestration and dynamic adaptation method is executed by the processor, it performs the following steps: Monitor the preset task flow and the operation of the target robot, and collect real-time data of each modality, environmental change data and real-time task scenario data; Historical task data of the target robot is acquired and processed in combination with real-time data of each modality, environmental change data and real-time task scenario data to obtain the weight parameters of each modality. The modal weight parameters are filtered and fused to obtain fused multimodal data; The preset task flow is processed to obtain the optimal task execution order; Based on the fused multimodal data and the optimal task execution order, a task flow orchestration and dynamic adaptation scheme is executed on the preset task flow; Monitor the operation of the scheme within a preset time period, extract operational indicator data, and evaluate the effectiveness of the scheme; The historical task data of the target robot is acquired and processed in conjunction with the real-time data of each modality, environmental change data, and real-time task scenario data to obtain the weight parameters of each modality, including: Acquire historical task data of the target robot, including task type data, task result data, raw sensor data, and environmental parameter data; Based on the historical task data, combined with the real-time data of each modality, environmental change data, and real-time task scenario data, the weight parameters of each modality are obtained by processing them through a preset gradient descent algorithm. The process of filtering and fusing the modal weight parameters to obtain fused multimodal data includes: Get the preset threshold parameter; Each modality weight parameter is compared with the preset threshold parameter; If the modal weight parameter is less than the preset threshold parameter, then the modal data corresponding to the modal weight parameter does not meet the requirements; If the modal weight parameter is greater than or equal to the preset threshold parameter, then the modal data corresponding to the modal weight parameter meets the requirements and is marked as modal data to be fused. The fused multimodal data is obtained by combining the modal data to be fused with the corresponding modal weight parameters.

6. The multimodal fusion robot task flow orchestration and dynamic adaptation system according to claim 5, characterized in that, The monitoring of the preset task flow and the operation of the target robot collects real-time data of each modality, environmental change data, and real-time task scenario data, including: Monitor preset task flows and collect environmental change data as well as real-time task scenario data; The environmental change data includes temperature change rate data, humidity change rate data, obstacle movement speed, and target object position offset data; The real-time task scenario data includes real-time task target data and real-time task instruction data; Monitor the operation of the target robot and collect real-time data for each modality, including real-time data for visual modality, auditory modality, tactile modality, motion and positioning modality, and environmental perception modality.

Images