Robot control method and device based on multi-modal task planning

Abstract

The invention discloses a robot control method and device based on multi-modal task planning, and relates to the field of multi-modal control, and the method comprises the steps: obtaining multi-modal environment data of a to-be-controlled robot based on a sensor, and carrying out the fusion of the multi-modal environment data, and obtaining a multi-modal environment feature; based on a similarity algorithm, screening a reference environment feature having the highest similarity with the multi-modal environment feature from a historical task database, and obtaining a reference environment influence feature according to a historical task sequence corresponding to the reference environment feature; performing correlation analysis on an executed task sequence of a to-be-controlled robot and the multi-modal environment characteristics to obtain environment influence characteristics; and according to the reference environment influence characteristics and the environment influence characteristics, planning and correcting a to-be-executed task sequence of a to-be-controlled robot to obtain a target task sequence, and according to the target task sequence, controlling the to-be-controlled robot.

Description

Technical Field

[0001] This application relates to the field of multimodal control, and in particular, to a robot control method and device based on multimodal task planning. Background Art

[0002] As the complexity of tasks executed by robots continues to increase, the research on robot automation control technology has gradually emerged. Traditional robot automation control often relies on fixed control algorithms and program judgments, lacking corresponding flexibility and intelligence. Specifically, traditional robot control methods can be divided into robot control methods based on control technology and robot control methods based on perception technology. The former relies on specific open-loop or closed-loop control algorithms to complete tasks, lacking perception technology. During the task execution process, it cannot read environmental feedback and thus cannot adjust task planning, and is only applicable to simple task scenarios with low accuracy requirements; the latter relies on multiple sensors to obtain various environmental information and make corresponding decisions and actions, but it is difficult to process information data integrated with multiple modalities, and there is a problem of one-sided decision-making based on a single modality. Both of the above may lead to inaccurate task planning and thus distorted control of the robot when facing complex or high-precision task scenarios, resulting in task failure. Therefore, how to avoid distorted control of the robot and improve the accuracy of robot control remains a difficult problem to be solved in the prior art. Summary of the Invention

[0003] This application provides a robot control method and device based on multimodal task planning to solve the technical problem of the lack of accuracy in existing robot control.

[0004] According to the first aspect of the embodiment of the present application, a robot control method based on multimodal task planning is provided, including: Based on sensors, obtain multimodal environmental data of the robot to be controlled, and fuse the multimodal environmental data to obtain multimodal environmental features; wherein, the multimodal environmental data includes sound data, image data, and pose data; Based on a similarity algorithm, screen out the reference environmental feature with the highest similarity to the multimodal environmental feature from the historical task database, and obtain a reference environmental impact feature according to the historical task sequence corresponding to the reference environmental feature; Perform correlation analysis on the executed task sequence of the robot to be controlled and the multimodal environmental feature to obtain an environmental impact feature; According to the reference environmental impact feature and the environmental impact feature, plan and correct the to-be-executed task sequence of the robot to be controlled to obtain a target task sequence, and control the robot to be controlled according to the target task sequence.

[0005] In this application, first, multi-modal environmental data of the robot to be controlled is obtained based on sensors and fused to obtain multi-modal environmental features. Then, reference environmental features are selected from the historical task database based on a similarity algorithm, and then reference environmental impact features are obtained. Next, correlation analysis is performed on the executed task sequence and the multi-modal environmental features to obtain environmental impact features. Finally, the reference environmental impact features and the environmental impact features are combined to plan and correct the task sequence to be executed to obtain a target task sequence and perform corresponding control. Compared with the prior art, in this application, multi-modal environmental data is obtained through sensors and fused, which can perceive the environment and obtain environmental feedback, and make decisions by combining multiple modal data, avoiding one-sided decisions and improving the accuracy of robot control. At the same time, by selecting reference environmental features to obtain reference environmental impact features, it can provide a reference for correcting the task sequence, and combining the environmental impact features obtained from the correlation analysis of the executed task sequence, which restricts each other with the reference environmental impact features, preventing over-correction and improving the accuracy of task planning for the robot, thereby improving the accuracy of robot control.

[0006] In some embodiments of this application, based on the sensors, multi-modal environmental data of the robot to be controlled is obtained, and the multi-modal environmental data is fused to obtain multi-modal environmental features, which specifically includes: Based on the sensors, initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled are obtained; Temporal feature analysis is respectively performed on the initial sound data, the initial image data, and the initial pose data to obtain environmental sound features, environmental image features, and motion control features; Based on a convolutional neural network and a self-attention encoder, the environmental sound features, the environmental image features, and the motion control features are unified in dimension and fused to obtain multi-modal environmental features.

[0007] In this application, first, the initial sound data, initial image data, and initial pose data of the robot to be controlled are obtained based on sensors, and temporal feature analysis is respectively performed to obtain corresponding features. Then, based on a convolutional neural network and a self-attention encoder, the three types of features are unified in dimension and fused to obtain multi-modal environmental features, which can perceive environmental feedback and combine multiple modal data to provide a prerequisite for subsequent decision-making and control.

[0008] In some embodiments of this application, obtaining the reference environmental impact features according to the historical task sequence corresponding to the reference environmental features specifically includes: According to the reference environmental features, determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental features; Integrate the reference environmental features according to the task feature correlation degree of each historical subtask in chronological order of the historical subtasks in the historical task sequence to obtain reference environmental impact features.

[0009] In this application, first, according to the reference environmental features, determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental features, which can determine the correlation degree between historical subtasks and different types of features. Furthermore, integrate the reference environmental features in chronological order of the historical subtasks in the historical task sequence to obtain reference environmental impact features, which can integrate the reference environmental features according to the task feature correlation degree, making it more suitable for current requirements and providing a reference for the correction of the subsequent to-be-executed task sequence.

[0010] In some embodiments of this application, the correlation analysis of the executed task sequence of the to-be-controlled robot and the multimodal environmental features is performed to obtain environmental impact features, which specifically includes: Statistically analyze the fluctuation of each feature of the multimodal environmental features corresponding to each executed subtask in the execution period of the executed task sequence to obtain an environmental fluctuation trend; According to the environmental fluctuation trend, determine the task-environment fluctuation correlation degree of each executed subtask, and according to the task-environment fluctuation correlation degree of each executed subtask, integrate the multimodal environmental features in chronological order of the executed subtasks in the executed task sequence to obtain environmental impact features.

[0011] In this application, first, statistically analyze the fluctuation of each feature of the multimodal environmental features corresponding to each executed subtask in the execution period of the executed task sequence to obtain an environmental fluctuation trend, and then determine the task-environment fluctuation correlation degree of each executed subtask, which can determine the correlation degree between the executed subtask and the environmental fluctuation. Furthermore, integrate the multimodal environmental features in chronological order of the executed subtasks in the executed task sequence to obtain environmental impact features, which can integrate the multimodal environmental features according to the task-environment fluctuation correlation degree, making it more suitable for current requirements and providing a reference for the correction of the subsequent to-be-executed task sequence.

[0012] In some embodiments of this application, according to the reference environmental impact features and the environmental impact features, plan and correct the to-be-executed task sequence of the to-be-controlled robot to obtain a target task sequence, which specifically includes: Perform multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact features to obtain a task decision prediction tree; Prune the task decision prediction tree according to the environmental impact features to obtain multiple predicted task sequences; Based on the prediction probabilities of each of the multiple prediction task sequences, combine the multiple prediction task sequences to perform planning correction on the to-be-executed task sequence, and obtain a target task sequence.

[0013] In this application, first, according to the reference environmental impact characteristics, a multi-branch prediction is performed on the to-be-executed task sequence to obtain a task decision prediction tree, which can provide a reference for task sequence correction. Then, pruning is performed according to the environmental impact characteristics, which can avoid incorrect corrections caused by inaccurate predictions, thereby improving the accuracy of the task planning for the robot and the accuracy of robot control.

[0014] According to the second aspect of the embodiments of this application, a robot control device based on multi-modal task planning is provided, including a multi-modal environmental feature fusion module, a reference environmental feature acquisition module, an environmental feature analysis and acquisition module, and a task planning correction module; The multi-modal environmental feature fusion module is used to obtain multi-modal environmental data of the to-be-controlled robot based on sensors, and fuse the multi-modal environmental data to obtain multi-modal environmental features; wherein, the multi-modal environmental data includes sound data, image data, and pose data; The reference environmental feature acquisition module is used to screen out the reference environmental feature with the highest similarity to the multi-modal environmental feature from the historical task database based on a similarity algorithm, and obtain a reference environmental impact feature according to the historical task sequence corresponding to the reference environmental feature; The environmental feature analysis and acquisition module is used to perform correlation analysis on the executed task sequence of the to-be-controlled robot and the multi-modal environmental features to obtain environmental impact features; The to-be-executed task planning correction module is used to perform planning correction on the to-be-executed task sequence of the to-be-controlled robot according to the reference environmental impact feature and the environmental impact feature, obtain a target task sequence, and control the to-be-controlled robot according to the target task sequence.

[0015] In some embodiments of this application, the multi-modal environmental feature fusion module includes a data acquisition unit, a feature analysis unit, and a feature fusion unit; The data acquisition unit is used to obtain the initial sound data, initial image data, and initial pose data corresponding to the to-be-controlled robot based on sensors; The feature analysis unit is used to perform temporal feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound features, environmental image features, and action control features; The feature fusion unit is used to unify and fuse the dimensions of the environmental sound features, the environmental image features, and the action control features based on a convolutional neural network and a self-attention encoder to obtain multi-modal environmental features.

[0016] In some embodiments of the present application, the reference environmental feature acquisition module includes a feature correlation analysis unit and a reference feature integration unit; The feature correlation analysis unit is used to determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental feature according to the reference environmental feature; The reference feature integration unit is used to integrate the reference environmental feature according to the execution time sequence of the historical subtasks in the historical task sequence according to the task feature correlation degree of each historical subtask to obtain a reference environmental impact feature.

[0017] In some embodiments of the present application, the environmental feature analysis and acquisition module includes a fluctuation trend statistics unit and an environmental feature analysis unit; The fluctuation trend statistics unit is used to statistically analyze the fluctuation of each feature of the multi-modal environmental feature corresponding to each executed subtask during the execution period of the executed task sequence to obtain an environmental fluctuation trend; The environmental feature analysis unit is used to determine the task environmental fluctuation correlation degree of each executed subtask according to the environmental fluctuation trend, and integrate the multi-modal environmental feature according to the execution time sequence of the executed subtasks in the executed task sequence according to the task environmental fluctuation correlation degree of each executed subtask to obtain an environmental impact feature.

[0018] In some embodiments of the present application, the to-be-executed task planning and correction module includes a task branch prediction unit, a prediction branch pruning unit, and a task planning correction unit; The task branch prediction unit is used to perform multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree; The prediction branch pruning unit is used to prune the task decision prediction tree according to the environmental impact feature to obtain a plurality of predicted task sequences; The task planning correction unit is used to perform planning correction on the to-be-executed task sequence based on the prediction probabilities of the plurality of predicted task sequences and in combination with the plurality of predicted task sequences to obtain a target task sequence.

[0019] In this application, first, multi-modal environmental data of the robot to be controlled is obtained based on sensors and fused to obtain multi-modal environmental features. Then, reference environmental features are selected from the historical task database based on a similarity algorithm, and thus reference environmental impact features are obtained. Next, correlation analysis is performed on the executed task sequence and the multi-modal environmental features to obtain environmental impact features. Finally, the reference environmental impact features and the environmental impact features are combined to plan and correct the task sequence to be executed to obtain a target task sequence and perform corresponding control. Compared with the prior art, in this application, multi-modal environmental data is obtained through sensors and fused, enabling the perception of the environment and obtaining environmental feedback. Moreover, by making decisions by combining multiple modal data, one-sided decision-making is avoided, and the accuracy of robot control is improved. At the same time, by selecting reference environmental features to obtain reference environmental impact features, it can provide a reference for the correction of the task sequence. By combining the environmental impact features obtained from the correlation analysis of the executed task sequence and the reference environmental impact features, they restrict each other to prevent overcorrection and improve the accuracy of the task planning of the robot, thereby improving the accuracy of robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 : A schematic flowchart of a robot control method based on multi-modal task planning shown in some embodiments of this application; Figure 2 : A module structure diagram of a robot control device based on multi-modal task planning shown in some embodiments of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The following details the embodiments of this application. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by combining the drawings are exemplary and are only used to explain some embodiments of this application and cannot be understood as a limitation to the embodiments of this application. Based on the embodiments shown in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0022] In the description of this application, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of this application, unless otherwise specifically defined, the meaning of "multiple" and "several" is two or more.

[0023] In existing robot control methods, the robot control method based on control technology lacks the ability to perceive, cannot read environmental feedback, and thus cannot adjust task planning; the robot control method based on sensing technology has multi-type sensing capabilities, but it is difficult to process information integrated with multiple modalities, and there is a problem of one-sided decision-making caused by a single modality. Both have the defect of insufficient control accuracy. Therefore, how to avoid control distortion of the robot and improve the control accuracy of the robot is still a difficult problem that needs to be solved urgently in the existing technology.

[0024] Based on the above technical background, please refer to Figure 1 , the embodiments of the present application provide a robot control method based on multi-modal task planning, including steps S101 to S104, and the specific steps are as follows: Step S101: Based on sensors, obtain multi-modal environmental data of the robot to be controlled, and fuse the multi-modal environmental data to obtain multi-modal environmental features; wherein, the multi-modal environmental data includes sound data, image data, and pose data.

[0025] In some embodiments of the present application, the step of obtaining multi-modal environmental data of the robot to be controlled based on sensors and fusing the multi-modal environmental data to obtain multi-modal environmental features specifically includes: Based on sensors, obtain the initial sound data, initial image data, and initial pose data corresponding to the robot to be controlled; Perform time-series feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound features, environmental image features, and motion control features; Based on a convolutional neural network and a self-attention encoder, unify the dimensions of the environmental sound features, the environmental image features, and the motion control features and fuse them to obtain multi-modal environmental features.

[0026] In some embodiments of the present application, the dimension unification based on the convolutional neural network is specifically a variable-dimension operation based on the convolutional neural network. When changing the dimension, the dimensions of the environmental sound features and the motion control features are transformed to be the same as the dimension of the environmental image features. Generally, the implementation manners of the convolutional neural network include but are not limited to CNN and its improved models, and the preferred implementation manner is CNN.

[0027] The present application first obtains the initial sound data, initial image data, and initial pose data of the robot to be controlled based on sensors, and performs time-series feature analysis on them respectively to obtain corresponding features, and then unifies the dimensions and fuses the three types of features based on a convolutional neural network and a self-attention encoder to obtain multi-modal environmental features, which can perceive environmental feedback and combine multiple modal data, providing a prerequisite for subsequent decision-making control.

[0028] Step S102: Based on the similarity algorithm, screen out the reference environmental features with the highest similarity to the multi-modal environmental features from the historical task database, and obtain the reference environmental impact features according to the historical task sequence corresponding to the reference environmental features.

[0029] In some embodiments of the present application, the obtaining the reference environmental impact features according to the historical task sequence corresponding to the reference environmental features specifically includes: Determine the task feature association degree between each historical subtask in the historical task sequence and each feature of the reference environmental features according to the reference environmental features; Integrate the reference environmental features according to the execution time sequence of the historical subtasks in the historical task sequence according to the task feature association degree of each historical subtask, and obtain the reference environmental impact features.

[0030] It should be understood that the task feature association degree described in the present application is specifically the probability that the current subtask in the historical task sequence is associated with each feature in the reference environmental features. Exemplarily, considering the current subtask A, the reference environmental features include three types of features: feature X, feature Y, and feature Z. Then the task feature association degrees are the probability that A is associated with X, the probability that A is associated with Y, and the probability that A is associated with Z.

[0031] In the present application, first, according to the reference environmental features, the task feature association degree between each historical subtask in the historical task sequence and each feature of the reference environmental features is determined, so as to determine the association degree between the historical subtask and different types of features. Furthermore, the reference environmental features are integrated according to the execution time sequence of the historical subtasks in the historical task sequence to obtain the reference environmental impact features, which can integrate the reference environmental features according to the task feature association degree, making it more suitable for the current requirements and providing a reference for the correction of the subsequent task sequence to be executed.

[0032] Step S103: Perform an association analysis on the executed task sequence of the robot to be controlled and the multi-modal environmental features to obtain the environmental impact features.

[0033] In some embodiments of the present application, the performing an association analysis on the executed task sequence of the robot to be controlled and the multi-modal environmental features to obtain the environmental impact features specifically includes: Statistically analyze the fluctuation of each feature of the multi-modal environmental features corresponding to the execution period of each executed subtask in the executed task sequence to obtain the environmental fluctuation trend; According to the environmental fluctuation trend, determine the task environment fluctuation correlation degree of each executed subtask, and according to the task environment fluctuation correlation degree of each executed subtask, integrate the multi-modal environmental features in the execution time sequence of the executed subtasks in the executed task sequence to obtain environmental impact features.

[0034] It should be understood that the task environment fluctuation correlation degree described in this application is specifically the probability that the currently executed subtask in the executed task sequence is correlated with the environmental fluctuation.

[0035] This application first counts the fluctuation of each feature of the multi-modal environmental features corresponding to the execution period of each executed subtask in the executed task sequence to obtain the environmental fluctuation trend, and then determines the task environment fluctuation correlation degree of each executed subtask, which can determine the correlation degree between the executed subtask and the environmental fluctuation. Furthermore, integrating the multi-modal environmental features in the execution time sequence of the executed subtasks in the executed task sequence to obtain environmental impact features can integrate the multi-modal environmental features according to the task environment fluctuation correlation degree, making it more suitable for the current requirements and providing a reference for the correction of the subsequent to-be-executed task sequence.

[0036] Step S104: According to the reference environmental impact feature and the environmental impact feature, plan and correct the to-be-executed task sequence of the robot to be controlled to obtain a target task sequence, and control the robot to be controlled according to the target task sequence.

[0037] In some embodiments of this application, the planning and correction of the to-be-executed task sequence of the robot to be controlled according to the reference environmental impact feature and the environmental impact feature to obtain a target task sequence specifically includes: Perform multi-branch prediction on the to-be-executed task sequence according to the reference environmental impact feature to obtain a task decision prediction tree; Prune the task decision prediction tree according to the environmental impact feature to obtain multiple predicted task sequences; Based on the prediction probabilities of the multiple predicted task sequences respectively, combine the multiple predicted task sequences to plan and correct the to-be-executed task sequence to obtain a target task sequence.

[0038] In some embodiments of this application, the preferred implementation of the task decision prediction tree obtained by multi-branch prediction is a decision tree; pruning the task decision prediction tree specifically means pruning the branches on the tree with a prediction probability less than the pruning threshold, and the pruning threshold is calculated based on the environmental impact feature.

[0039] In this application, first, based on the reference environmental impact characteristics, a multi-branch prediction is performed on the task sequence to be executed to obtain a task decision prediction tree, which can provide a reference for task sequence correction. Then, pruning is performed according to the environmental impact characteristics, which can avoid incorrect corrections caused by inaccurate predictions, thereby improving the accuracy of the task planning for the robot and the accuracy of robot control.

[0040] Compared with the prior art, in this application, first, multi-modal environmental data of the robot to be controlled is obtained based on sensors and fused to obtain multi-modal environmental characteristics. Then, based on a similarity algorithm, reference environmental characteristics are selected from the historical task database, and then reference environmental impact characteristics are obtained. Next, an association analysis is performed on the executed task sequence and the multi-modal environmental characteristics to obtain environmental impact characteristics. Finally, the task sequence to be executed is planned and corrected by combining the reference environmental impact characteristics and the environmental impact characteristics to obtain a target task sequence and corresponding control. Compared with the prior art, in this application, multi-modal environmental data is obtained through sensors and fused, which can sense the environment and obtain environmental feedback, and decisions are made by combining multiple modal data to avoid one-sided decisions and improve the accuracy of robot control. At the same time, by selecting reference environmental characteristics to obtain reference environmental impact characteristics, it can provide a reference for the correction of the task sequence, and combined with the environmental impact characteristics obtained from the association analysis of the executed task sequence, they restrict each other to prevent over-correction and improve the accuracy of the task planning for the robot, thereby improving the accuracy of robot control.

[0041] Corresponding to the foregoing method, please refer to Figure 2 , an embodiment of this application provides a robot control device based on multi-modal task planning, including a multi-modal environmental characteristic fusion module 210, a reference environmental characteristic acquisition module 220, an environmental characteristic analysis and acquisition module 230, and a task planning correction module 240; The multi-modal environmental characteristic fusion module 210 is configured to obtain multi-modal environmental data of the robot to be controlled based on sensors and fuse the multi-modal environmental data to obtain multi-modal environmental characteristics. Among them, the multi-modal environmental data includes sound data, image data, and pose data; The reference environmental characteristic acquisition module 220 is configured to screen and obtain the reference environmental characteristic with the highest similarity to the multi-modal environmental characteristic from the historical task database based on a similarity algorithm, and obtain reference environmental impact characteristics according to the historical task sequence corresponding to the reference environmental characteristic; The environmental characteristic analysis and acquisition module 230 is configured to perform an association analysis on the executed task sequence of the robot to be controlled and the multi-modal environmental characteristics to obtain environmental impact characteristics; The to-be-executed task planning correction module 240 is configured to correct the to-be-executed task sequence of the to-be-controlled robot according to the reference environmental impact characteristics and the environmental impact characteristics, obtain a target task sequence, and control the to-be-controlled robot according to the target task sequence.

[0042] In some embodiments of the present application, the multimodal environmental feature fusion module 210 includes a data acquisition unit, a feature analysis unit, and a feature fusion unit; The data acquisition unit is configured to obtain initial sound data, initial image data, and initial pose data corresponding to the to-be-controlled robot based on sensors; The feature analysis unit is configured to perform temporal feature analysis on the initial sound data, the initial image data, and the initial pose data respectively to obtain environmental sound features, environmental image features, and motion control features; The feature fusion unit is configured to unify and fuse the environmental sound features, the environmental image features, and the motion control features in dimension based on a convolutional neural network and a self-attention encoder to obtain multimodal environmental features.

[0043] In some embodiments of the present application, the reference environmental feature acquisition module 220 includes a feature correlation analysis unit and a reference feature integration unit; The feature correlation analysis unit is configured to determine the task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environmental features according to the reference environmental features; The reference feature integration unit is configured to integrate the reference environmental features according to the execution time sequence of the historical subtasks in the historical task sequence according to the task feature correlation degree of each historical subtask to obtain reference environmental impact features.

[0044] In some embodiments of the present application, the environmental feature analysis and acquisition module 230 includes a fluctuation trend statistics unit and an environmental feature analysis unit; The fluctuation trend statistics unit is configured to statistically analyze the fluctuation of each feature of the multimodal environmental features corresponding to each executed subtask during the execution period of the executed task sequence to obtain an environmental fluctuation trend; The environmental feature analysis unit is configured to determine the task environment fluctuation correlation degree of each executed subtask according to the environmental fluctuation trend, and integrate the multimodal environmental features according to the execution time sequence of the executed subtasks in the executed task sequence according to the task environment fluctuation correlation degree of each executed subtask to obtain environmental impact features.

[0045] In some embodiments of the present application, the to-be-executed task planning correction module 240 includes a task branch prediction unit, a predicted branch pruning unit, and a task planning correction unit; The task branch prediction unit is configured to perform multi-branch prediction on the to-be-executed task sequence according to the reference environment impact feature to obtain a task decision prediction tree; The predicted branch pruning unit is configured to prune the task decision prediction tree according to the environment impact feature to obtain a plurality of predicted task sequences; The task planning correction unit is configured to perform planning correction on the to-be-executed task sequence based on the respective prediction probabilities of the plurality of predicted task sequences and in combination with the plurality of predicted task sequences to obtain a target task sequence.

[0046] In the present application, first, multi-modal environment data of the to-be-controlled robot is acquired based on a sensor and fused to obtain multi-modal environment features, then reference environment features are selected from a historical task database based on a similarity algorithm, and further reference environment impact features are obtained. Then, correlation analysis is performed on the executed task sequence and the multi-modal environment features to obtain environment impact features. Finally, the to-be-executed task sequence is planned and corrected in combination with the reference environment impact features and the environment impact features to obtain a target task sequence and corresponding control is performed. Compared with the prior art, by acquiring and fusing multi-modal environment data through a sensor, the present application can perceive the environment and obtain environmental feedback, and make decisions in combination with multiple modal data to avoid one-sided decisions and improve the accuracy of robot control. At the same time, by selecting reference environment features to obtain reference environment impact features, a reference can be provided for the correction of the task sequence, and in combination with the environment impact features obtained by correlation analysis of the executed task sequence, they restrict each other to prevent over-correction and improve the accuracy of the task planning of the robot, thereby improving the accuracy of robot control.

[0047] It should be understood that the device provided by the embodiment of the present application corresponds to the foregoing method. A robot control device based on multi-modal task planning provided by the embodiment of the present application can implement a robot control method based on multi-modal task planning provided by any embodiment of the present application.

[0048] Adaptively, the embodiment of the present application further provides a computer device and a computer-readable storage medium.

[0049] The computer device includes: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; Wherein, when the processor executes the computer program, a robot control method based on multi-modal task planning of the present application is implemented.

[0050] The computer-readable storage medium stores multiple instructions, and the instructions are adapted to be loaded by a processor to execute a robot control method based on multi-modal task planning of the present application.

[0051] The above are partial embodiments of the present application, which have further elaborated on the purpose, technical solutions, and beneficial effects of the present application. It should be clear that the above partial embodiments of the present application should not be construed as a limitation of the present application. In particular, for those skilled in the art, any changes, modifications, equivalent replacements, and variations made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A robot control method based on multimodal task planning, characterized in that: include: Based on the sensor, multimodal environment data of the robot to be controlled is obtained, and the multimodal environment data is fused to obtain multimodal environment features; wherein the multimodal environment data includes sound data, image data and posture data; Based on a similarity algorithm, a reference environment feature with the highest similarity to the multimodal environment feature is screened from a historical task database, and a reference environment impact feature is obtained according to a historical task sequence corresponding to the reference environment feature; Performing correlation analysis on the executed task sequence of the robot to be controlled and the multimodal environment characteristics to obtain the environmental impact characteristics; According to the reference environmental impact characteristics and the environmental impact characteristics, the task sequence to be executed by the robot to be controlled is planned and corrected to obtain a target task sequence, and the robot to be controlled is controlled according to the target task sequence.

2. A robot control method based on multimodal task planning according to claim 1, characterized in that: The method of acquiring multimodal environment data of the robot to be controlled based on the sensor and fusing the multimodal environment data to obtain multimodal environment features specifically includes: Based on the sensor, initial sound data, initial image data and initial posture data corresponding to the robot to be controlled are obtained; Performing time series feature analysis on the initial sound data, the initial image data, and the initial posture data respectively to obtain environmental sound features, environmental image features, and motion control features; Based on a convolutional neural network and a self-attention encoder, the environmental sound features, the environmental image features and the motion control features are dimensionally unified and fused to obtain multimodal environmental features.

3. The robot control method based on multimodal task planning according to claim 1, characterized in that: The obtaining of the reference environment impact characteristics according to the historical task sequence corresponding to the reference environment characteristics specifically includes: Determining, based on the reference environment characteristics, a task feature correlation degree between each historical subtask in the historical task sequence and each feature of the reference environment characteristics; According to the task feature correlation of each historical subtask, the reference environment features are integrated according to the execution timing of the historical subtasks in the historical task sequence to obtain reference environment impact features.

4. The robot control method based on multimodal task planning according to claim 1, characterized in that: The correlation analysis of the executed task sequence of the robot to be controlled and the multimodal environment characteristics to obtain the environmental impact characteristics specifically includes: Counting the fluctuation of each feature of the multimodal environment feature corresponding to the execution period of each executed subtask in the executed task sequence to obtain an environment fluctuation trend; According to the environmental fluctuation trend, the task environment fluctuation correlation of each executed subtask is determined, and according to the task environment fluctuation correlation of each executed subtask, the multimodal environmental features are integrated according to the execution timing of the executed subtasks in the executed task sequence to obtain the environmental impact features.

5. The robot control method based on multimodal task planning according to claim 1, characterized in that: The planning and correction of the task sequence to be executed by the robot to be controlled according to the reference environmental impact characteristics and the environmental impact characteristics to obtain the target task sequence specifically includes: According to the reference environment impact characteristics, multi-branch prediction is performed on the task sequence to be executed to obtain a task decision prediction tree; Pruning the task decision prediction tree according to the environmental impact characteristics to obtain multiple prediction task sequences; Based on the prediction probabilities of the multiple predicted task sequences, the multiple predicted task sequences are combined to perform planning correction on the task sequence to be executed to obtain a target task sequence.

6. A robot control device based on multimodal task planning, characterized in that: It includes a multi-modal environment feature fusion module, a reference environment feature acquisition module, an environment feature analysis acquisition module and a task planning correction module; The multimodal environment feature fusion module is used to obtain multimodal environment data of the robot to be controlled based on sensors, and fuse the multimodal environment data to obtain multimodal environment features; wherein the multimodal environment data includes sound data, image data and posture data; The reference environment feature acquisition module is used to screen the reference environment feature with the highest similarity to the multimodal environment feature from the historical task database based on a similarity algorithm, and obtain the reference environment impact feature according to the historical task sequence corresponding to the reference environment feature; The environmental feature analysis and acquisition module is used to perform correlation analysis on the executed task sequence of the robot to be controlled and the multimodal environmental features to obtain environmental impact features; The to-be-executed task planning correction module is used to plan and correct the to-be-executed task sequence of the robot to be controlled according to the reference environmental impact characteristics and the environmental impact characteristics, obtain a target task sequence, and control the robot to be controlled according to the target task sequence.

7. A robot control device based on multimodal task planning according to claim 6, characterized in that: The multimodal environment feature fusion module includes a data acquisition unit, a feature analysis unit and a feature fusion unit; The data acquisition unit is used to acquire initial sound data, initial image data and initial posture data corresponding to the robot to be controlled based on the sensor; The feature analysis unit is used to perform time series feature analysis on the initial sound data, the initial image data and the initial posture data respectively to obtain environmental sound features, environmental image features and action control features; The feature fusion unit is used to unify and fuse the dimensions of the environmental sound features, the environmental image features and the action control features based on a convolutional neural network and a self-attention encoder to obtain multimodal environmental features.

8. A robot control device based on multimodal task planning according to claim 6, characterized in that: The reference environment feature acquisition module includes a feature association analysis unit and a reference feature integration unit; The feature association analysis unit is used to determine the task feature association degree of each historical subtask in the historical task sequence and each feature of the reference environment feature according to the reference environment feature; The reference feature integration unit is used to integrate the reference environment features according to the execution sequence of the historical subtasks in the historical task sequence according to the task feature correlation of each historical subtask, so as to obtain the reference environment impact features.

9. A robot control device based on multimodal task planning according to claim 6, characterized in that: The environmental characteristics analysis and acquisition module includes a fluctuation trend statistics unit and an environmental characteristics analysis unit; The fluctuation trend statistics unit is used to count the fluctuation of each feature of the multimodal environment feature corresponding to the execution period of each executed subtask in the executed task sequence to obtain the environment fluctuation trend; The environmental feature analysis unit is used to determine the task environment fluctuation correlation of each executed subtask according to the environmental fluctuation trend, and integrate the multimodal environmental features according to the execution timing of the executed subtasks in the executed task sequence according to the task environment fluctuation correlation of each executed subtask to obtain the environmental impact features.

10. A robot control device based on multimodal task planning according to claim 6, characterized in that: The to-be-executed task planning correction module includes a task branch prediction unit, a prediction branch pruning unit and a task planning correction unit; The task branch prediction unit is used to perform multi-branch prediction on the to-be-executed task sequence according to the reference environment impact characteristics to obtain a task decision prediction tree; The prediction branch pruning unit is used to prune the task decision prediction tree according to the environmental impact characteristics to obtain multiple prediction task sequences; The task planning correction unit is used to perform planning correction on the to-be-executed task sequence based on the prediction probabilities of the multiple predicted task sequences and in combination with the multiple predicted task sequences to obtain a target task sequence.

Images