A robot control method and device based on multi-modal task planning

Abstract

The present application discloses a robot control method and device based on multi-modal task planning, which relates to the field of multi-modal control and includes: obtaining multi-modal environment data of a robot to be controlled based on sensors, and fusing the multi-modal environment data to obtain multi-modal environment features; screening, based on a similarity algorithm, the reference environment features with the highest similarity to the multi-modal environment features from a historical task database, and obtaining reference environment influence features according to the historical task sequence corresponding to the reference environment features; performing correlation analysis on the executed task sequence of the robot to be controlled and the multi-modal environment features to obtain environment influence features; planning and correcting the to-be-executed task sequence of the robot to be controlled according to the reference environment influence features and the environment influence features to obtain a target task sequence, and controlling the robot to be controlled according to the target task sequence.

Description

Technical Field

[0001] The present application relates to the field of multimodal control, and in particular to a robot control method and device based on multimodal task planning. Background Art

[0002] As the complexity of robotic tasks continues to increase, research on automated control technology has gradually gained momentum. Traditional automated robotic control often relies on fixed control algorithms and program judgments, lacking flexibility and intelligence. Specifically, traditional robotic control methods can be categorized as control-based and perception-based. The former relies on specific open-loop or closed-loop control algorithms to complete tasks, lacking perception technology. This makes it impossible to read environmental feedback during task execution and adjust task planning, making it suitable only for simple tasks with low precision requirements. The latter relies on multiple sensors to acquire diverse environmental information and make decisions and actions accordingly. However, it struggles to process information integrated from multiple modalities, and suffers from the problem of biased decision-making based on a single modality. Both approaches, when faced with complex or high-precision tasks, can lead to distorted robot control due to inaccurate task planning, resulting in mission failure. Therefore, how to avoid control distortion and improve robot control accuracy remains a pressing challenge in existing technologies. Summary of the Invention

[0003] The present application provides a robot control method and device based on multimodal task planning to solve the technical problem of insufficient accuracy of existing robot control.

[0004] According to a first aspect of the embodiments of the present application, a robot control method based on multimodal task planning is provided, comprising:

[0005] Based on sensors, multimodal environmental data of the robot to be controlled is acquired, and the multimodal environmental data is fused to obtain multimodal environmental features; wherein the multimodal environmental data includes sound data, image data, and posture data;

[0006] Based on the similarity algorithm, the reference environment feature with the highest similarity to the multimodal environment feature is screened from the historical task database, and the reference environment impact feature is obtained according to the historical task sequence corresponding to the reference environment feature;

[0007] Performing correlation analysis on the executed task sequence of the robot to be controlled and the multimodal environment characteristics to obtain environmental impact characteristics;

[0008] According to the reference environmental impact characteristics and the environmental impact characteristics, plan and correct the to-be-executed task sequence of the to-be-controlled robot to obtain a target task sequence, and control the to-be-controlled robot according to the target task sequence.

[0009] In this application, first, multi-modal environmental data of the to-be-controlled robot is obtained based on sensors and fused to obtain multi-modal environmental characteristics. Then, reference environmental characteristics are selected from the historical task database based on a similarity algorithm, and thus reference environmental impact characteristics are obtained. Next, correlation analysis is performed on the executed task sequence and the multi-modal environmental characteristics to obtain environmental impact characteristics. Finally, the to-be-executed task sequence is planned and corrected by combining the reference environmental impact characteristics and the environmental impact characteristics to obtain a target task sequence and corresponding control is performed. Compared with the prior art, this application can sense the environment and obtain environmental feedback by obtaining and fusing multi-modal environmental data through sensors, and make decisions by combining multiple modal data, avoiding one-sided decisions and improving the accuracy of robot control. At the same time, by selecting reference environmental characteristics to obtain reference environmental impact characteristics, it can provide a reference for the correction of the task sequence, and combine the environmental impact characteristics obtained by correlation analysis of the executed task sequence to restrict each other with the reference environmental impact characteristics, preventing over-correction and improving the accuracy of the task planning of the robot, thereby improving the accuracy of robot control.

[0010] In some embodiments of this application, based on the sensors, obtaining multi-modal environmental data of the to-be-controlled robot and fusing the multi-modal environmental data to obtain multi-modal environmental characteristics specifically includes:

[0011] Based on the sensors, obtaining the initial sound data, initial image data, and initial pose data corresponding to the to-be-controlled robot;

[0012] Performing temporal feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound characteristics, environmental image characteristics, and motion control characteristics;

[0013] Based on a convolutional neural network and a self-attention encoder, unifying the dimensions and fusing the environmental sound characteristics, the environmental image characteristics, and the motion control characteristics to obtain multi-modal environmental characteristics.

[0014] In this application, first, the initial sound data, initial image data, and initial pose data of the to-be-controlled robot are obtained based on sensors, and corresponding characteristics are obtained through temporal feature analysis respectively. Then, based on a convolutional neural network and a self-attention encoder, the three types of characteristics are unified in dimension and fused to obtain multi-modal environmental characteristics, which can sense environmental feedback and combine multiple modal data, providing a prerequisite for subsequent decision-making and control.

[0015] In some embodiments of the present application, obtaining the reference environment impact feature according to the historical task sequence corresponding to the reference environment feature specifically includes:

[0016] According to the reference environment feature, determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environment feature;

[0017] According to the task feature correlation degree of each historical subtask, integrate the reference environment feature according to the execution time sequence of the historical subtasks in the historical task sequence to obtain the reference environment impact feature.

[0018] The present application first determines the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environment feature according to the reference environment feature, can determine the correlation degree between the historical subtask and different types of features, and then integrates the reference environment feature according to the execution time sequence of the historical subtasks in the historical task sequence to obtain the reference environment impact feature, can integrate the reference environment feature according to the task feature correlation degree, make it more adaptable to the current demand, and provide a reference for the correction of the subsequent to-be-executed task sequence.

[0019] In some embodiments of the present application, performing correlation analysis on the executed task sequence of the to-be-controlled robot and the multimodal environment feature to obtain the environment impact feature specifically includes:

[0020] Statistically analyze the fluctuation of each feature of the multimodal environment feature corresponding to each executed subtask in the executed task sequence during the execution period to obtain the environment fluctuation trend;

[0021] According to the environment fluctuation trend, determine the task environment fluctuation correlation degree of each executed subtask, and according to the task environment fluctuation correlation degree of each executed subtask, integrate the multimodal environment feature according to the execution time sequence of the executed subtasks in the executed task sequence to obtain the environment impact feature.

[0022] The present application first statistically analyzes the fluctuation of each feature of the multimodal environment feature corresponding to each executed subtask in the executed task sequence during the execution period to obtain the environment fluctuation trend, and then determines the task environment fluctuation correlation degree of each executed subtask, can determine the correlation degree between the executed subtask and the environment fluctuation, and then integrates the multimodal environment feature according to the execution time sequence of the executed subtasks in the executed task sequence to obtain the environment impact feature, can integrate the multimodal environment feature according to the task environment fluctuation correlation degree, make it more adaptable to the current demand, and provide a reference for the correction of the subsequent to-be-executed task sequence.

[0023] In some embodiments of the present application, planning and modifying the to-be-executed task sequence of the to-be-controlled robot according to the reference environmental impact feature and the environmental impact feature to obtain a target task sequence specifically includes:

[0024] Performing multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree;

[0025] Pruning the task decision prediction tree according to the environmental impact feature to obtain multiple predicted task sequences;

[0026] Based on the prediction probabilities of the multiple predicted task sequences respectively, combining the multiple predicted task sequences to perform planning and modification on the to-be-executed task sequence to obtain a target task sequence.

[0027] The present application first performs multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree, which can provide a reference for task sequence modification, and then performs pruning according to the environmental impact feature, which can avoid incorrect modification caused by inaccurate prediction, thereby improving the accuracy of the task planning of the robot and the accuracy of robot control.

[0028] According to the second aspect of the embodiments of the present application, a robot control device based on multi-modal task planning is provided, including a multi-modal environmental feature fusion module, a reference environmental feature acquisition module, an environmental feature analysis and acquisition module, and a task planning modification module;

[0029] The multi-modal environmental feature fusion module is used to obtain multi-modal environmental data of the to-be-controlled robot based on sensors and fuse the multi-modal environmental data to obtain multi-modal environmental features; wherein, the multi-modal environmental data includes sound data, image data, and pose data;

[0030] The reference environmental feature acquisition module is used to screen out the reference environmental feature with the highest similarity to the multi-modal environmental feature from the historical task database based on a similarity algorithm, and obtain a reference environmental impact feature according to the historical task sequence corresponding to the reference environmental feature;

[0031] The environmental feature analysis and acquisition module is used to perform correlation analysis on the executed task sequence of the to-be-controlled robot and the multi-modal environmental feature to obtain an environmental impact feature;

[0032] The task planning modification module is used to perform planning and modification on the to-be-executed task sequence of the to-be-controlled robot according to the reference environmental impact feature and the environmental impact feature to obtain a target task sequence, and control the to-be-controlled robot according to the target task sequence.

[0033] In some embodiments of the present application, the multi-modal environmental feature fusion module includes a data acquisition unit, a feature analysis unit, and a feature fusion unit;

[0034] The data acquisition unit is configured to obtain initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled based on sensors;

[0035] The feature analysis unit is configured to perform temporal feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound features, environmental image features, and action control features;

[0036] The feature fusion unit is configured to unify and fuse the environmental sound features, the environmental image features, and the action control features in dimension based on a convolutional neural network and a self-attention encoder to obtain multi-modal environmental features.

[0037] In some embodiments of the present application, the reference environmental feature acquisition module includes a feature correlation analysis unit and a reference feature integration unit;

[0038] The feature correlation analysis unit is configured to determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental feature according to the reference environmental feature;

[0039] The reference feature integration unit is configured to integrate the reference environmental feature according to the execution time sequence of the historical subtasks in the historical task sequence according to the task feature correlation degree of each historical subtask to obtain a reference environmental influence feature.

[0040] In some embodiments of the present application, the environmental feature analysis and acquisition module includes a fluctuation trend statistics unit and an environmental feature analysis unit;

[0041] The fluctuation trend statistics unit is configured to statistically analyze the fluctuation of each feature of the multi-modal environmental feature corresponding to the execution period of each executed subtask in the executed task sequence to obtain an environmental fluctuation trend;

[0042] The environmental feature analysis unit is configured to determine the task environmental fluctuation correlation degree of each executed subtask according to the environmental fluctuation trend, and integrate the multi-modal environmental feature according to the execution time sequence of the executed subtasks in the executed task sequence according to the task environmental fluctuation correlation degree of each executed subtask to obtain an environmental influence feature.

[0043] In some embodiments of the present application, the task planning correction module includes a task branch prediction unit, a prediction branch pruning unit, and a task planning correction unit;

[0044] The task branch prediction unit is configured to perform multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature, so as to obtain a task decision prediction tree;

[0045] The predicted branch pruning unit is configured to prune the task decision prediction tree according to the environmental impact feature, so as to obtain a plurality of predicted task sequences;

[0046] The task planning correction unit is configured to perform planning correction on the to-be-executed task sequence based on the prediction probabilities of the plurality of predicted task sequences and in combination with the plurality of predicted task sequences, so as to obtain a target task sequence.

[0047] In this application, first, multi-modal environmental data of the to-be-controlled robot is acquired based on sensors and fused to obtain multi-modal environmental features, then reference environmental features are selected from the historical task database based on a similarity algorithm, and further reference environmental impact features are obtained. Then, correlation analysis is performed on the executed task sequence and the multi-modal environmental features to obtain environmental impact features. Finally, the to-be-executed task sequence is planned and corrected by combining the reference environmental impact features and the environmental impact features to obtain a target task sequence and corresponding control. Compared with the prior art, in this application, multi-modal environmental data is acquired by sensors and fused, so that the environment can be sensed and environmental feedback can be obtained, and decisions are made by combining multiple modal data, avoiding one-sided decisions and improving the accuracy of robot control. At the same time, by selecting reference environmental features to obtain reference environmental impact features, it can provide a reference for the correction of the task sequence, and in combination with the environmental impact features obtained by the correlation analysis of the executed task sequence, they restrict each other to prevent over-correction and improve the accuracy of the task planning of the robot, thereby improving the accuracy of robot control. Description of the Drawings

[0048] Figure 1 : is a schematic flowchart of a robot control method based on multi-modal task planning shown in some embodiments of this application;

[0049] Figure 2 : is a module structure diagram of a robot control device based on multi-modal task planning shown in some embodiments of this application. Detailed Embodiments

[0050] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by combining the accompanying drawings are exemplary and are only used to explain some embodiments of the present application, and should not be construed as a limitation on the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments shown in the present application without creative efforts belong to the protection scope of the present application.

[0051] In the description of the present application, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present application, unless otherwise specifically defined, the meaning of "a plurality" and "several" is two or more.

[0052] In existing robot control methods, the robot control method based on control technology lacks the ability of perception, cannot read environmental feedback, and thus cannot adjust task planning; the robot control method based on perception technology has multi-type perception capabilities, but it is difficult to process information integrated with multiple modalities, and there is a problem of one-sided decision-making caused by a single modality. Both have the defect of insufficient control accuracy. Therefore, how to avoid control distortion of the robot and improve the control accuracy of the robot is still a difficult problem to be solved in the prior art.

[0053] Based on the above technical background, please refer to Figure 1 , the embodiments of the present application provide a robot control method based on multi-modal task planning, including steps S101 to S104, and the specific steps are as follows:

[0054] Step S101: Based on sensors, obtain multi-modal environment data of the robot to be controlled, and fuse the multi-modal environment data to obtain multi-modal environment features; wherein, the multi-modal environment data includes sound data, image data, and pose data.

[0055] In some embodiments of the present application, the step of based on sensors, obtaining multi-modal environment data of the robot to be controlled, and fusing the multi-modal environment data to obtain multi-modal environment features specifically includes:

[0056] Based on sensors, obtain the initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled;

[0057] Perform temporal feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound features, environmental image features, and action control features;

[0058] Based on a convolutional neural network and a self-attention encoder, unify the dimensions and fuse the environmental sound features, the environmental image features, and the action control features to obtain multi-modal environmental features.

[0059] In some embodiments of the present application, dimension unification based on a convolutional neural network is specifically a variable-dimension operation based on a convolutional neural network. When changing dimensions, the dimensions of the environmental sound features and the action control features are transformed to be the same as the dimensions of the environmental image features. Generally, the implementation manners of the convolutional neural network include but are not limited to CNN and its improved models, and the preferred implementation manner is CNN.

[0060] In this application, first, based on sensors, the initial sound data, the initial image data, and the initial pose data of the robot to be controlled are obtained, and temporal feature analysis is performed respectively to obtain corresponding features. Then, based on a convolutional neural network and a self-attention encoder, the three types of features are unified in dimension and fused to obtain multi-modal environmental features, which can perceive environmental feedback and combine multiple modal data, providing a prerequisite for subsequent decision-making control.

[0061] Step S102: Based on a similarity algorithm, screen out the reference environmental features with the highest similarity to the multi-modal environmental features from the historical task database, and obtain reference environmental impact features according to the historical task sequence corresponding to the reference environmental features.

[0062] In some embodiments of the present application, the obtaining of the reference environmental impact features according to the historical task sequence corresponding to the reference environmental features specifically includes:

[0063] According to the reference environmental features, determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental features;

[0064] According to the task feature correlation degree of each historical subtask, integrate the reference environmental features according to the execution time sequence of the historical subtasks in the historical task sequence to obtain reference environmental impact features.

[0065] It should be understood that the task feature correlation degree described in this application is specifically the probability that the current subtask in the historical task sequence is associated with each feature in the reference environmental features. Exemplarily, considering the current subtask A, the reference environmental features include three types of features: type X features, type Y features, and type Z features. Then the task feature correlation degree is the probability that A is associated with X, the probability that A is associated with Y, and the probability that A is associated with Z.

[0066] In this application, first, according to the reference environmental characteristics, the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental characteristics is determined. The correlation degree between the historical subtasks and different types of features can be determined. Furthermore, the reference environmental characteristics are integrated according to the execution time sequence of the historical subtasks in the historical task sequence to obtain the reference environmental impact characteristics. The reference environmental characteristics can be integrated according to the task feature correlation degree to make it more suitable for the current requirements and provide a reference for the correction of the subsequent to-be-executed task sequence.

[0067] Step S103: Perform correlation analysis on the executed task sequence of the robot to be controlled and the multimodal environmental characteristics to obtain environmental impact characteristics.

[0068] In some embodiments of this application, the performing correlation analysis on the executed task sequence of the robot to be controlled and the multimodal environmental characteristics to obtain environmental impact characteristics specifically includes:

[0069] Statistical fluctuations of each feature of the multimodal environmental characteristics corresponding to each executed subtask in the execution period of the executed task sequence to obtain an environmental fluctuation trend;

[0070] According to the environmental fluctuation trend, determine the task environmental fluctuation correlation degree of each executed subtask, and according to the task environmental fluctuation correlation degree of each executed subtask, integrate the multimodal environmental characteristics according to the execution time sequence of the executed subtasks in the executed task sequence to obtain environmental impact characteristics.

[0071] It should be understood that the task environmental fluctuation correlation degree described in this application is specifically the probability that the current executed subtask in the executed task sequence is associated with environmental fluctuations.

[0072] In this application, first, statistical fluctuations of each feature of the multimodal environmental characteristics corresponding to each executed subtask in the execution period of the executed task sequence are performed to obtain an environmental fluctuation trend. Furthermore, the task environmental fluctuation correlation degree of each executed subtask is determined. The correlation degree between the executed subtasks and environmental fluctuations can be determined. Furthermore, the multimodal environmental characteristics are integrated according to the execution time sequence of the executed subtasks in the executed task sequence to obtain environmental impact characteristics. The multimodal environmental characteristics can be integrated according to the task environmental fluctuation correlation degree to make it more suitable for the current requirements and provide a reference for the correction of the subsequent to-be-executed task sequence.

[0073] Step S104: According to the reference environmental impact characteristics and the environmental impact characteristics, plan and correct the to-be-executed task sequence of the robot to be controlled to obtain a target task sequence, and control the robot to be controlled according to the target task sequence.

[0074] In some embodiments of the present application, planning and modifying the to-be-executed task sequence of the robot to be controlled according to the reference environmental impact feature and the environmental impact feature to obtain a target task sequence specifically includes:

[0075] Performing multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree;

[0076] Pruning the task decision prediction tree according to the environmental impact feature to obtain a plurality of predicted task sequences;

[0077] Based on the prediction probabilities of the respective plurality of predicted task sequences, combining the plurality of predicted task sequences, and performing planning and modification on the to-be-executed task sequence to obtain a target task sequence.

[0078] In some embodiments of the present application, a preferred embodiment of the task decision prediction tree obtained by multi-branch prediction is a decision tree; pruning the task decision prediction tree specifically means pruning the branches on the tree with prediction probabilities less than the pruning threshold, and the pruning threshold is calculated based on the environmental impact feature.

[0079] In the present application, first, multi-branch prediction is performed on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree, which can provide a reference for task sequence modification. Then, pruning is performed according to the environmental impact feature, which can avoid incorrect modification caused by inaccurate prediction, thereby improving the accuracy of the task planning for the robot and the accuracy of robot control.

[0080] Compared with the prior art, in the present application, first, multi-modal environmental data of the robot to be controlled is acquired based on sensors and fused to obtain multi-modal environmental features, then the reference environmental features are selected from the historical task database based on the similarity algorithm to obtain the reference environmental impact features, and then the executed task sequence and the multi-modal environmental features are analyzed for association to obtain the environmental impact features. Finally, the to-be-executed task sequence is planned and modified by combining the reference environmental impact feature and the environmental impact feature to obtain a target task sequence and corresponding control. Compared with the prior art, in the present application, multi-modal environmental data is acquired through sensors and fused, which can sense the environment and obtain environmental feedback, and decision-making is performed by combining multiple modal data to avoid one-sided decision-making and improve the accuracy of robot control; at the same time, by selecting the reference environmental features to obtain the reference environmental impact features, it can provide a reference for the modification of the task sequence, and combined with the environmental impact features obtained by the association analysis of the executed task sequence, they restrict each other to prevent over-modification and improve the accuracy of the task planning for the robot, thereby improving the accuracy of robot control.

[0081] Corresponding to the foregoing method, please refer to Figure 2, an embodiment of the present application provides a robot control device based on multi-modal task planning, including a multi-modal environment feature fusion module 210, a reference environment feature acquisition module 220, an environment feature analysis and acquisition module 230, and a task planning correction module 240;

[0082] The multi-modal environment feature fusion module 210 is configured to obtain multi-modal environment data of the robot to be controlled based on sensors, and fuse the multi-modal environment data to obtain multi-modal environment features; wherein, the multi-modal environment data includes sound data, image data, and pose data;

[0083] The reference environment feature acquisition module 220 is configured to screen out the reference environment feature with the highest similarity to the multi-modal environment feature from the historical task database based on a similarity algorithm, and obtain a reference environment influence feature according to the historical task sequence corresponding to the reference environment feature;

[0084] The environment feature analysis and acquisition module 230 is configured to perform correlation analysis on the executed task sequence of the robot to be controlled and the multi-modal environment feature to obtain an environment influence feature;

[0085] The task planning correction module 240 is configured to plan and correct the to-be-executed task sequence of the robot to be controlled according to the reference environment influence feature and the environment influence feature to obtain a target task sequence, and control the robot to be controlled according to the target task sequence.

[0086] In some embodiments of the present application, the multi-modal environment feature fusion module 210 includes a data acquisition unit, a feature analysis unit, and a feature fusion unit;

[0087] The data acquisition unit is configured to obtain initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled based on sensors;

[0088] The feature analysis unit is configured to perform time-series feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain an environmental sound feature, an environmental image feature, and an action control feature;

[0089] The feature fusion unit is configured to unify and fuse the environmental sound feature, the environmental image feature, and the action control feature in dimension based on a convolutional neural network and a self-attention encoder to obtain multi-modal environment features.

[0090] In some embodiments of the present application, the reference environment feature acquisition module 220 includes a feature correlation analysis unit and a reference feature integration unit;

[0091] The feature association analysis unit is configured to determine the task feature association degree between each historical subtask in the historical task sequence and each feature of the reference environment feature according to the reference environment feature;

[0092] The reference feature integration unit is configured to integrate the reference environment features according to the execution time sequence of the historical subtasks in the historical task sequence according to the task feature association degree of each historical subtask, so as to obtain reference environment impact features.

[0093] In some embodiments of the present application, the environment feature analysis and acquisition module 230 includes a fluctuation trend statistics unit and an environment feature analysis unit;

[0094] The fluctuation trend statistics unit is configured to statistically analyze the fluctuation of each feature of the multimodal environment feature corresponding to each executed subtask in the executed task sequence during the execution period, so as to obtain an environment fluctuation trend;

[0095] The environment feature analysis unit is configured to determine the task environment fluctuation association degree of each executed subtask according to the environment fluctuation trend, and integrate the multimodal environment features according to the execution time sequence of the executed subtasks in the executed task sequence according to the task environment fluctuation association degree of each executed subtask, so as to obtain environment impact features.

[0096] In some embodiments of the present application, the task planning correction module 240 includes a task branch prediction unit, a prediction branch pruning unit, and a task planning correction unit;

[0097] The task branch prediction unit is configured to perform multi-branch prediction on the to-be-executed task sequence according to the reference environment impact feature, so as to obtain a task decision prediction tree;

[0098] The prediction branch pruning unit is configured to prune the task decision prediction tree according to the environment impact feature, so as to obtain a plurality of predicted task sequences;

[0099] The task planning correction unit is configured to perform planning correction on the to-be-executed task sequence based on the prediction probabilities of the plurality of predicted task sequences and in combination with the plurality of predicted task sequences, so as to obtain a target task sequence.

[0100] In this application, first, multi-modal environmental data of the robot to be controlled is obtained based on sensors and fused to obtain multi-modal environmental features. Then, reference environmental features are selected from the historical task database based on a similarity algorithm, and then reference environmental impact features are obtained. Next, correlation analysis is performed on the executed task sequence and the multi-modal environmental features to obtain environmental impact features. Finally, the reference environmental impact features and the environmental impact features are combined to plan and correct the to-be-executed task sequence to obtain the target task sequence and perform corresponding control. Compared with the prior art, this application can sense the environment and obtain environmental feedback by obtaining and fusing multi-modal environmental data through sensors, and make decisions by combining multiple modal data, avoiding one-sided decisions and improving the accuracy of robot control. At the same time, by selecting reference environmental features to obtain reference environmental impact features, it can provide a reference for the correction of the task sequence, and combine the environmental impact features obtained by the correlation analysis of the executed task sequence to restrict each other with the reference environmental impact features, preventing overcorrection and improving the accuracy of the task planning of the robot, thereby improving the accuracy of robot control.

[0101] It should be understood that the device provided by the embodiment of this application corresponds to the foregoing method. A robot control device based on multi-modal task planning provided by the embodiment of this application can implement a robot control method based on multi-modal task planning provided by any embodiment of this application.

[0102] Adaptively, the embodiment of this application also provides a computer device and a computer-readable storage medium.

[0103] The computer device includes: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor;

[0104] Wherein, when the processor executes the computer program, it implements a robot control method based on multi-modal task planning of this application.

[0105] The computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by the processor to execute a robot control method based on multi-modal task planning of this application.

[0106] The above is a partial embodiment of this application. The purpose, technical solution, and beneficial effects of this application are further described in detail. It should be clear that the above partial embodiment of this application should not be construed as a limitation of this application. In particular, for those skilled in the art, any changes, modifications, equivalent replacements, and variations made within the spirit and principle of this application should be included within the protection scope of this application.

Claims

1. A robot control method based on multi-modal task planning, characterized in that, Including: Based on sensors, obtain multi-modal environmental data of the robot to be controlled, and fuse the multi-modal environmental data to obtain multi-modal environmental features; wherein, the multi-modal environmental data includes sound data, image data, and pose data; Based on a similarity algorithm, screen out the reference environmental feature with the highest similarity to the multi-modal environmental feature from the historical task database, and obtain a reference environmental impact feature according to the historical task sequence corresponding to the reference environmental feature; Perform correlation analysis on the executed task sequence of the robot to be controlled and the multi-modal environmental feature to obtain an environmental impact feature; According to the reference environmental impact feature and the environmental impact feature, plan and correct the to-be-executed task sequence of the robot to be controlled to obtain a target task sequence, and control the robot to be controlled according to the target task sequence; The obtaining of the reference environmental impact feature according to the historical task sequence corresponding to the reference environmental feature specifically includes: determining the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental feature according to the reference environmental feature; according to the task feature correlation degree of each historical subtask, integrating the reference environmental feature according to the execution time sequence of the historical subtasks in the historical task sequence to obtain a reference environmental impact feature.

2. The robot control method based on multi-modal task planning according to claim 1, wherein The obtaining of the multi-modal environmental feature by obtaining the multi-modal environmental data of the robot to be controlled based on sensors and fusing the multi-modal environmental data specifically includes: Based on sensors, obtain the initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled; Perform time-series feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound features, environmental image features, and action control features; Based on a convolutional neural network and a self-attention encoder, unify the dimensions and fuse the environmental sound features, the environmental image features, and the action control features to obtain multi-modal environmental features.

3. The robot control method based on multi-modal task planning according to claim 1, wherein, The performing of correlation analysis on the executed task sequence of the robot to be controlled and the multi-modal environmental feature to obtain an environmental impact feature specifically includes: Statistically analyze the fluctuation of each feature of the multi-modal environmental feature corresponding to each executed subtask in the executed task sequence to obtain an environmental fluctuation trend; According to the environmental fluctuation trend, determine the task-environment fluctuation correlation degree of each executed subtask, and according to the task-environment fluctuation correlation degree of each executed subtask, integrate the multi-modal environmental feature according to the execution time sequence of the executed subtasks in the executed task sequence to obtain an environmental impact feature.

4. A robot control method based on multimodal task planning according to claim 1, characterized in that, The planning and correction of the to-be-executed task sequence of the robot to be controlled according to the reference environmental impact feature and the environmental impact feature to obtain a target task sequence specifically includes: Perform multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree; Prune the task decision prediction tree according to the environmental impact feature to obtain multiple predicted task sequences; Based on the prediction probabilities of the respective multiple prediction task sequences, the multiple prediction task sequences are combined to perform planning correction on the to-be-executed task sequence, and a target task sequence is obtained.

5. A robot control device based on multi-modal task planning, characterized in that, It includes a multi-modal environment feature fusion module, a reference environment feature acquisition module, an environment feature analysis and acquisition module, and a task planning correction module; The multi-modal environment feature fusion module is used to obtain multi-modal environment data of the robot to be controlled based on sensors, and fuse the multi-modal environment data to obtain multi-modal environment features; wherein, the multi-modal environment data includes sound data, image data, and pose data; The reference environment feature acquisition module is used to screen out the reference environment feature with the highest similarity to the multi-modal environment feature from the historical task database based on the similarity algorithm, and obtain the reference environment influence feature according to the historical task sequence corresponding to the reference environment feature; The environment feature analysis and acquisition module is used to perform correlation analysis on the executed task sequence of the robot to be controlled and the multi-modal environment feature to obtain the environment influence feature; The task planning correction module is used to perform planning correction on the to-be-executed task sequence of the robot to be controlled according to the reference environment influence feature and the environment influence feature, obtain a target task sequence, and control the robot to be controlled according to the target task sequence; The reference environment feature acquisition module includes a feature correlation analysis unit and a reference feature integration unit; the feature correlation analysis unit is used to determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environment feature according to the reference environment feature; the reference feature integration unit is used to integrate the reference environment feature according to the execution time sequence of the historical subtasks in the historical task sequence according to the task feature correlation degree of each historical subtask to obtain the reference environment influence feature.

6. The robot control device based on multi-modal task planning according to claim 5, characterized in that, The multi-modal environment feature fusion module includes a data acquisition unit, a feature analysis unit, and a feature fusion unit; The data acquisition unit is used to obtain the initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled based on sensors; The feature analysis unit is used to perform time series feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain the environmental sound feature, the environmental image feature, and the action control feature; The feature fusion unit is used to unify and fuse the environmental sound feature, the environmental image feature, and the action control feature based on a convolutional neural network and a self-attention encoder to obtain multi-modal environment features.

7. A robot control device based on multi-modal task planning according to claim 5, characterized in that, The environment feature analysis and acquisition module includes a fluctuation trend statistics unit and an environment feature analysis unit; The fluctuation trend statistics unit is used to statistically analyze the fluctuation of each feature of the multi-modal environment feature corresponding to the execution period of each executed subtask in the executed task sequence to obtain the environmental fluctuation trend; The environmental feature analysis unit is configured to determine the task environment fluctuation correlation degree of each executed subtask according to the environmental fluctuation trend, and integrate the multi-modal environmental features according to the execution time sequence of the executed subtasks in the executed task sequence based on the task environment fluctuation correlation degree of each executed subtask, so as to obtain the environmental impact feature.

8. A robot control device based on multi-modal task planning according to claim 5, characterized in that, The task planning correction module includes a task branch prediction unit, a prediction branch pruning unit, and a task planning correction unit; The task branch prediction unit is configured to perform multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature, so as to obtain a task decision prediction tree; The prediction branch pruning unit is configured to prune the task decision prediction tree according to the environmental impact feature, so as to obtain a plurality of predicted task sequences; The task planning correction unit is configured to perform planning correction on the to-be-executed task sequence based on the respective prediction probabilities of the plurality of predicted task sequences and in combination with the plurality of predicted task sequences, so as to obtain a target task sequence.

Images