Body learning method and device based on dynamic subtask planning and storage medium

Through the dynamic subtask planning method, the robot's perception ability improvement under limited training conditions is optimized, which solves the problems of large data demand and exploration redundancy in reinforcement learning, and achieves efficient environmental understanding and perception performance improvement.

CN118963125AActive Publication Date: 2024-11-15TSINGHUA UNIVERSITY

Patent Information

Application Number
CN202411026767.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing reinforcement learning methods have problems such as large data demand, exploration path redundancy and data set long-tail effects in robot embodied learning, making it difficult to effectively improve the robot's perception ability under limited training data and time.

Method used

Adopt a well-known learning method based on dynamic subtask planning, generate subtasks through task planning large models, optimize subtask priorities and content using image features and observation information knowledge bases, dynamically adjust navigation order, and realize efficient environmental exploration and data collection.

Benefits of technology

With limited training data and time, quickly improve the robot's understanding and perception ability of the environment, avoid repeated exploration and data set imbalance, and achieve efficient environmental exploration and perception performance improvement.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118963125A_ABST
    Figure CN118963125A_ABST
Patent Text Reader

Abstract

The invention provides a self learning method based on dynamic sub-task planning, which comprises the following steps: a task planning large model generates a plurality of sub-tasks based on a task instruction, the performance of a to-be-improved perception model and environment information captured by a robot, performs priority ranking on the sub-tasks, and determines a current sub-task; the task analysis large model analyzes the subtask content to guide the robot to navigate in the environment, the pose prediction large model is used for guiding the robot to collect data, and in the navigation process, the task analysis large model dynamically adjusts the subtask content according to the environment information; after the current subtask is completed, the task planning large model dynamically plans subsequent subtasks; and after the subtasks are completed, the performance of the perception model is improved by using a data set acquired in the navigation process. According to the invention, the robot can acquire new data or new knowledge beneficial to improvement of perception ability, and the understanding of the robot on the environment is further improved by learning the acquired new data or new knowledge.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the fields of robot navigation and computer vision, and in particular to an embodied learning method, device and storage medium based on dynamic subtask planning. Background Art

[0002] The diversity of application scenarios places demands on the robot's ability to learn knowledge. For example, in the military field, robots need to continuously learn combat strategies and tactics to cope with changing war situations; in the medical field, robots need to continuously learn diagnostic and treatment methods to provide better medical services; in the field of service robots, robots need to continuously learn how to better understand and respond to user instructions to better meet user needs. Therefore, robots need to have the ability to learn knowledge and adapt to the ever-changing and expanding application scenarios by updating their own knowledge and skills. This task is the embodied learning task.

[0003] At present, reinforcement learning is widely used to train robot exploration strategies by sensing the differences in results. However, this method has its limitations. First, it requires a large amount of data for training, which undoubtedly increases the complexity and difficulty of the research. Second, this method may lead to redundancy in exploration paths, that is, the robot may conduct repeated explorations in certain areas. In addition, this method may also lead to a long-tail effect in the data set, that is, there is too little or even no data in certain categories.

[0004] The large model has scene-related common sense and planning capabilities, and can continuously adjust the planning goals during the exploration process to avoid the above problems. For example, the large model can prevent the robot from repeatedly exploring the room area that has already been explored by judging the room type. At the same time, the large model can also adjust the subsequent data collection goals by analyzing the current collected data to avoid the long tail effect of the data set. Therefore, the large model has great potential in solving these problems. At present, there is no related technology that uses large models for robot embodied learning. Summary of the invention

[0005] The present disclosure aims to solve one of the technical problems existing in the existing related technologies at least to a certain extent.

[0006] To this end, the present invention provides an embodied learning method, device and storage medium based on dynamic subtask planning, which enables the robot to collect new data or new knowledge that is conducive to improving its perception ability, and further improves the robot's understanding of the environment by learning the collected new data or new knowledge, thereby achieving rapid and effective improvement of the robot's perception ability with limited training data and training time.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] The first aspect of the present disclosure provides an embodied learning method based on dynamic subtask planning, comprising:

[0009] S1. Generate multiple subtasks using the task planning model based on the task instruction, sort the priorities of the subtasks based on the historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and use the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtask; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public data set; the environmental information is represented by the first-person perspective observation image captured by the robot;

[0010] S2. Construct a knowledge base for assisting the large model of posture prediction to evaluate the performance of the perception model to be improved, wherein the knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base, wherein the image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score;

[0011] S3. parse the current subtask into several executable high-level operations using the task parsing model. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. The target room type to be visited and the target semantic category set to be collected are determined based on the current subtask content.

[0012] S4, the robot performs the high-level operation of the analysis based on the low-level action plan, first navigates to the target room, then navigates to the target object in the target room in sequence, retrieves the relevant context from the knowledge base as the external knowledge prompt of the pose prediction large model, and guides the robot to collect data on the target object;

[0013] During the navigation process, the task parsing model dynamically adjusts the navigation order of the target object according to the environmental information collected by the robot in real time, so as to realize the dynamic planning of the current subtask content;

[0014] S5. After the current subtask is completed, the semantic category of the data collected when the robot performs the current subtask is fed back to the task planning model as the current subtask information. The task planning model will dynamically plan the next subtask according to the feedback information, task instructions, the performance of the perception model to be improved, and environmental information, and return to step S3 until all subtasks are completed;

[0015] S6. Using the data set collected during the navigation process to improve the perception performance of the perception model to be improved.

[0016] In some embodiments, in step S1, the task planning model divides several types of objects specified by the task instructions that need to improve perception performance into multiple subtasks according to room types based on the common sense of the association between room types and object categories, and the subtasks are represented by the target room type and the semantic category in the target room.

[0017] In some embodiments, in step S1, the priorities of the subtasks are sorted, specifically including:

[0018] Based on the collection of objects of various categories in the historical subtask information, the task planning model gives priority to collecting data in rooms where the semantic segmentation accuracy scores of the perception model to be improved are low and the object categories that have not been collected are located.

[0019] When the task planning model determines that the room the robot is currently in is the room type corresponding to one of the subtasks based on the current environmental information captured by the robot, the subtask corresponding to the current room is given priority.

[0020] In some embodiments, step S2 specifically includes:

[0021] S21. Data Collection

[0022] Collect a large number of perception models to be improved as specified in the task instructions. The data of the semantic category and the corresponding semantic truth value, for any semantic category Collect its observation image data Semantic category The observed image data of the i-th object in, n is the semantic category For each object, multiple observation image data are collected around the object, and the semantic truth value corresponding to each observation image data is obtained, which are recorded as and Semantic category The jth observed image of the i-th object in For the observed image The corresponding semantic truth value, in which the value of the pixel coordinate position corresponding to the semantics of object i is 1, and the rest are 0. The semantic category of the collection The total number of observed images of the i-th object in;

[0023] S22. Construction of image feature knowledge sub-base

[0024] Extract observation image Image features Use all extracted image features to build an image feature knowledge sub-base Database feat ;

[0025] S23. Construction of observation information knowledge sub-base

[0026] The observed image is obtained using the perception model to be improved Prediction semantic graph of The predicted semantics of the i-th object in the predicted semantic map corresponds to the pixel coordinate position value of 1, and the rest are 0; using the predicted semantic map Semantic truth value The inconsistency between them is used to calculate the score of the perception model to be improved on the observation The specific calculation method is as follows:

[0027]

[0028] Where W and H are the width and height of the observed image, respectively. Represents the predicted semantic value at the pixel coordinate position [w,h] in the observed image, Representative observation image The semantic truth value at the pixel coordinate position [w,h];

[0029] The inconsistency perception scores of all observed images of each object Sort from high to low and return the observed image number corresponding to the score The observed images corresponding to the inconsistency scores from high to low are denoted as So using the observed image Constructing the observation information knowledge sub-database obs .

[0030] In some embodiments, step S4 specifically includes:

[0031] S41. Target room navigation

[0032] The semantic information in the current observation image is extracted by using the perception model to be improved, and is input into the task parsing model. The room layout common sense possessed by the task parsing model is used to determine the type of the current room. When the task parsing model determines that the current room is not the target room, the room layout common sense is used to guide the robot to leave the current room, and continue to explore new rooms to find the target room. When the robot is in the target room, navigation to the target room is stopped. When it is determined that there is no room of the target type in the environment, the semantic category collection information is recorded in the historical subtask information, and the subtask priorities are reordered to determine the next subtask.

[0033] S42, Target Object Navigation

[0034] The semantic information in the current observation image is extracted by using the perception model to be improved, and the semantic information is input into the task parsing model together with the object corresponding to the current target semantic category in the target semantic category set, that is, the current target object. The task parsing model uses the common sense of the position distance between different objects and the current target object to calculate the association probability between the object and the current target object, and guides the robot to explore the area with high probability until the current target object is found.

[0035] When the task parsing model determines that there is no target object corresponding to the current target semantic category, it will dynamically adjust the current subtask content, set the object corresponding to the next target semantic category as the current target object, and continue searching; at the same time, the task parsing model will dynamically adjust the order of the target semantic categories in the target semantic category set based on the real-time observation image of the robot;

[0036] S43. Target object data collection

[0037] S431, target object retrieval

[0038] Given the target semantic category c in the current observation image, extract the current observation image obs c,curr Observational features of feat c,curr ; Calculate the observed features feat respectively c,curr The similarity simscore with each object image feature belonging to the target semantic category c in the image feature knowledge sub-base c,i , take the maximum similarity score as the similarity score sim of each object belonging to the target semantic category c c,i ; Calculate the number corresponding to the object with the highest object similarity score as the object retrieval number belonging to the target semantic category c

[0039] S432, Target object observation information retrieval

[0040] Search by object number The first K observation images of the object corresponding to the number are retrieved from the observation information knowledge sub-base as K comparison observation images to form a comparison observation image set.

[0041] S433, Navigation and observation data collection based on hierarchical targets

[0042] The pose prediction model is used to predict the control observation image set. Analyze each control observation image in the kth control observation image and predict the observation position pos of the object belonging to the target semantic category c in the kth control observation image k and observation angle rot k , set the observation positions and observation angles of the predicted K comparison observation images as sub-target sub;

[0043] Make the robot reach each observation position pos of the sub-goal sub by the shortest path k At the observation angle rot k Make observations and obtain the corresponding new observation images A total of K images are obtained, and each new observation image is obtained Corresponding semantic tags Construct a dataset data of objects belonging to the target semantic category c from K new observation images and their semantic labels c ;

[0044] S434, subtask data set generation

[0045] After collecting data about the current target semantic category, repeat steps S42 to S433 until collecting data about all semantic categories in the target semantic category set, and obtaining a data set data about the current subtask. Csub , and record the current subtask content and the semantic category of the data collected by the subtask into the historical subtask information.

[0046] In some embodiments, in step S431, cosine similarity is used to measure the similarity simscore c,i .

[0047] In some embodiments, in step S5, when the task planning large model dynamically plans the next subtask, it first determines the semantic category C for which data has not yet been collected based on the historical subtask information, and then generates the next subtask based on the environmental information, the perception model performance to be improved and the semantic category C to collect data related to the semantic category C, so as to ensure that data of all semantic categories are collected.

[0048] In some embodiments, in step S6, the perception model to be improved is fine-tuned or trained using the data set collected during the navigation process.

[0049] A second aspect of the present disclosure provides a device based on the embodied learning method described in any embodiment of the first aspect of the present disclosure, comprising:

[0050] The first module is configured to generate multiple subtasks based on the task instruction using the task planning model, sort the priorities of the subtasks based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and use the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtask; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public data set; the environmental information is represented by the first-person perspective observation image captured by the robot;

[0051] The second module is configured to construct a knowledge base for assisting the large model of pose prediction to evaluate the performance of the perception model to be improved, the knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base, the image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score;

[0052] The third module is configured to use the task parsing model to parse the current subtask into several executable high-level operations, wherein the high-level operations include three categories: target room navigation, target object navigation, and target object data collection, and determine the target room type to be visited and the target semantic category set to be collected based on the current subtask content;

[0053] The fourth module is configured to, when the robot performs the high-level operation of the analysis based on the underlying action plan, first enable the robot to navigate to the target room, then navigate to the target object in the target room in turn, retrieve relevant context from the knowledge base as an external knowledge prompt for the pose prediction large model, and guide the robot to collect data on the target object; during the navigation process, the task analysis large model dynamically adjusts the navigation order of the target object according to the environmental information collected by the robot in real time, so as to realize the dynamic planning of the current subtask content;

[0054] The fifth module is configured to feed back the semantic category of the data collected when the robot performs the current subtask as the current subtask information to the task planning model after the current subtask is completed. The task planning model will dynamically plan the next subtask according to the feedback information, task instructions, the performance of the perception model to be improved, and environmental information;

[0055] The sixth module is configured to improve the perception performance of the perception model to be improved by using the data set collected during the navigation process.

[0056] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the embodied learning method described in any embodiment of the first aspect of the present disclosure.

[0057] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0058] The present disclosure provides an embodied learning method based on dynamic subtask planning, which enables the robot to collect new data or new knowledge that is conducive to improving perception capabilities, and further improves the robot's understanding of the environment by learning the collected new data or new knowledge. Specifically: based on the room layout and object layout knowledge of the task parsing large model, subtasks are generated, and then the subtask priority is dynamically adjusted based on historical subtask information, the performance of the perception model to be improved, and the environmental information. In addition, during the completion of the subtask, the subtask content is dynamically adjusted according to the environmental information. The present disclosure collects multi-view observation data of a large number of object instances in advance, extracts data features to construct an image feature knowledge sub-base, and measures the perception results of the collected data based on a predefined semantic consistency perception score, sorts the data according to the perception score to construct an observation information knowledge sub-base; uses environmental information, combined with data with similar contexts retrieved from the constructed knowledge base, and uses the posture prediction large model to predict postures with poor perception performance. Therefore, the present disclosure can achieve effective environmental exploration without consuming a lot of data and time training; in addition, by dynamically adjusting the subtask priority and subtask content, efficient environmental exploration is achieved; and the data collected during the exploration process is used for learning to improve the performance of the perception model in the new environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an overall flow chart of the embodied learning method provided by the embodiment of the present disclosure.

[0060] Figure 2 It is a schematic diagram of the initial subtask planning in the embodied learning method provided in an embodiment of the present disclosure.

[0061] Figure 3It is a schematic diagram of constructing an image feature knowledge sub-base in the embodied learning method provided in an embodiment of the present disclosure.

[0062] Figure 4 It is a schematic diagram of constructing an observation information knowledge sub-base in the embodied learning method provided in an embodiment of the present disclosure.

[0063] Figure 5 It is a flowchart of subtask parsing in the embodied learning method provided in an embodiment of the present disclosure.

[0064] Figure 6 It is a flowchart of subtask data collection in the embodied learning method provided in an embodiment of the present disclosure.

[0065] Figure 7 It is a schematic diagram of subtask updating in the embodied learning method provided in an embodiment of the present disclosure.

[0066] Figure 8 It is a schematic diagram of the structure of an electronic device provided in the third aspect of the present disclosure. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] On the contrary, the present application covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present application as defined by the claims. Further, in order to make the public have a better understanding of the present application, some specific details are described in detail in the detailed description of the present application below. Those skilled in the art can fully understand the present application without the description of these details.

[0069] See also Figure 1 The embodiment of the present disclosure provides an embodied learning method based on dynamic subtask planning, which is used in indoor scenes. The method of this embodiment includes the following steps:

[0070] S1. Based on the task instructions, use the task planning model to generate multiple subtasks, and then based on the historical subtask information Info sub, the performance of the perception model to be improved and the environmental information captured by the robot are used to sort the priorities of the subtasks, and the subtask with the highest priority is used as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set in advance by the user, and the semantic category is represented by the object category name; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task of the semantic segmentation model pre-trained using the public data set (in this embodiment, the pre-trained model Mask R-CNN (Mask Region-based Convolutional Neural Network) is used in the new environment; the environmental information is represented by the first-person perspective observation image captured by the RGB camera equipped with the robot;

[0071] S2. Construct a knowledge base for assisting the large model of posture prediction to evaluate the performance of the perception model to be improved. The knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances. The data of each object instance is stored in descending order based on the perception score.

[0072] S3. The task parsing model parses the current subtask into several executable high-level operations. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the target room type to be visited and the target semantic category set to be collected are determined.

[0073] S4, the robot executes the high-level operation parsed in step S3 based on the low-level action plan, first navigates to the target room, then navigates to the target object in the target room in sequence, retrieves relevant context from the knowledge base constructed in step S2 as external knowledge prompts for the pose prediction large model, and guides the robot to collect data on the target object;

[0074] During the navigation process, the task parsing model dynamically adjusts the navigation order of the target object according to the environmental information collected by the robot in real time, and realizes the dynamic planning of the current subtask content;

[0075] S5. After the current subtask is completed, the semantic category of the data collected when the robot performs the current subtask is fed back to the task planning model as the current subtask information. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information, and return to step S3 until all subtasks are completed;

[0076] S6. Use the data set collected during navigation to fine-tune or train the perception model to improve its perception performance.

[0077] In some embodiments, see Figure 2 , step S1 specifically includes the following steps:

[0078] S11. Subtask generation

[0079] Get the task instructions set by the user. The task instructions need to specify the semantic category of the perception model to be improved. Semantic Category is the semantic category name of the total object that needs to improve perception performance. Since indoor scenes are often structured, the object categories in different types of rooms are often fixed. Based on the common sense of the association between room types and object categories in the task planning model, the task planning model is used to Class objects are divided into multiple subtasks according to room types. The subtasks are represented by the target room type and the semantic category in the room. It should be noted that different rooms may contain the same semantic category at the same time. For example, both the living room and the bedroom may contain chairs. Since there may be errors between the actual room layout and common sense, for example, there may be no chairs in the bedroom, all will be considered. The room types that may appear in the semantics of the class are considered, and the subtask content is dynamically adjusted in the subsequent steps to ensure that data for all semantic categories is collected. The class semantics include chair, sofa, plant, bed, toilet, and the task planning model will generate the following subtasks:

[0080] chair_sofa_in_livingroom();

[0081] toilet_in_bathroom();

[0082] bed_chair_plant_in_bedroom();

[0083] S12. Subtask Priority Sorting

[0084] The subtask priorities are sorted based on historical subtask information, the accuracy scores of each category in the perception model to be improved, and the current environment information. Specifically:

[0085] First, based on the historical subtask information Info subAccording to the data collection of each category, priority is given to collecting data in the rooms where the semantic categories of the perception model to be improved have low semantic segmentation accuracy scores and have not been collected. For example, the performance scores of the perception model on semantic categories that have not been collected are: chair (0.48), plant (0.56), sofa (0.66), bed (0.68), toilet (0.76). The subtasks corresponding to the categories with low collection scores are set to the highest priority, and the subtask priorities are sorted from high to low as follows:

[0086] bed_chair_plant_in_bedroom();

[0087] chair_sofa_in_livingroom();

[0088] toilet_in_bathroom();

[0089] In addition, the current environment information is input into the task planning model, which will estimate the current room type. If the robot is in a room type corresponding to one of the subtasks, this subtask will be given priority. For example, if the task planning model determines that the current room is a bathroom, the subtask priority will be adjusted to:

[0090] toilet_in_bathroom();

[0091] bed_chair_plant_in_bedroom();

[0092] chair_sofa_in_livingroom();

[0093] S13. Output current subtask

[0094] Task planning sorts subtasks according to their priorities and outputs the subtask with the highest current priority as the subtask that needs to be executed currently.

[0095] In some embodiments, step S2 specifically includes the following steps:

[0096] S21. Data Collection

[0097] Collect a large amount of The data of the class object and the corresponding semantic truth value. Collect its observation image data in Semantic category The observed image data of the i-th object in, n is the semantic category For each object, data is collected around the object, requiring that the object can be observed in the collected observation image, and the semantic truth value corresponding to the observation image is obtained by calling the environment API interface, which are recorded as and in Semantic category The jth observed image of the i-th object in For the observed image The corresponding semantic truth value, in which the value of the pixel coordinate position corresponding to the semantics of object i is 1, and the rest are The semantic category of the collection The total number of observed images of the i-th object in .

[0098] S22. Construction of image feature knowledge sub-base

[0099] See also Figure 3 , in order to build the image feature knowledge sub-base Database feat , using a visual model (in this embodiment, a pre-trained model CLIP (Contrastive Language-Image Pre-training) is used as a visual model) to extract the observed image Image features Recorded as Among them, φ() is the pre-trained model CLIP. The extracted image features are saved in the image feature knowledge sub-base, recorded as:

[0100]

[0101]

[0102]

[0103] in, Image feature knowledge sub-database feat The semantic category is The image features of all observed images are Image feature knowledge sub-database feat Medium semantic category The image features of all observed images of the i-th object in, Image feature knowledge sub-database feat Medium semantic category The image features of the jth observed image of the ith object in Semantic category The total number of image features of the observed image of the i-th object in .

[0104] S23. Construction of observation information knowledge sub-base

[0105] See also Figure 4 In order to build the observation information knowledge sub-base Database obs , use the perception model to be improved to predict the semantic graph, denoted as in, is the perception model to be improved, Semantic category The jth observed image of the ith object in The predicted semantic map of the ith object in the predicted semantic map has a pixel coordinate value of 1, and the others are 0. Semantic truth value In the semantic category The semantic inconsistency calculation on the observation is to improve the score of the perception model on the observation The specific calculation method is as follows:

[0106]

[0107] Where W and H are the width and height of the observed image, respectively. Represents the predicted semantic value at the pixel coordinate position [w,h] in the observed image, Represents the semantic truth value of the pixel coordinate position [w,h] in the observed image.

[0108] The inconsistency perception scores of all observed images of each object Sort from high to low and return the observed image number corresponding to the score Denoted as:

[0109]

[0110] Among them, des_sort() represents descending sort.

[0111] The observed images corresponding to the inconsistency scores from high to low are recorded as:

[0112]

[0113] Store the retrieved observation image into the observation information knowledge sub-base Database obs In, it is written as:

[0114]

[0115]

[0116]

[0117] in, Represents the observation information knowledge sub-base Database obs The semantic category is All observed image data, Represents the observation information knowledge sub-base Database obs The semantic category is All observed image data of the i-th object, Represents the observation information knowledge sub-base Database obs The semantic category is The jth observed image data of the ith object.

[0118] In some embodiments, see Figure 5 , step S3 specifically includes the following steps:

[0119] The content of the current subtask outputted in step S13 is parsed using the task parsing model. First, the executable high-level operations are determined. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. Among them, target room navigation is used to guide the robot to reach the target room, such as guiding the robot to reach the bedroom; target object navigation is used to guide the robot to reach the vicinity of the target object in the target room, and face the object so that the object is located in the observation image, such as guiding the robot to reach the bed in the bedroom; target object data collection is used to guide the robot to collect data with poor perception accuracy of the target object. For example, for a TV, the robot can recognize that it is a TV from the image collected from the front, but cannot accurately recognize that it is a TV from the image collected from the side. The target object data collection operation guides the robot to collect data from the side, that is, data with poor perception performance. Then, based on the current subtask content, determine the type of target room R that needs to be visited. target The target semantic category set for which data needs to be collected Where M is the total number of semantic categories that need to be collected for the current subtask. And C sub It may contain one or more semantic categories. Finally, the task parsing model is used to transform the subtask into a combination of multiple executable high-level operations.

[0120] In some embodiments, see Figure 5 , step S4 specifically includes the following steps:

[0121] S41. Target room navigation

[0122] Since the indoor environment is often a highly structured environment, that is, the room layout is usually the same, for example, the bedroom is next to the living room, and the object categories contained in different room types are often fixed. The semantic segmentation model (in this embodiment, the pre-trained model Mask R-CNN is used as the semantic segmentation model) is used to extract the semantic information in the current observation image, and the information is input into the task parsing model. The room layout common sense of the task parsing model is used to judge the current room type. When the task parsing model determines that the current room is not the target room R target When the robot is in the target room R, it will use the room layout common sense of the task parsing large model to guide the robot to leave the current room and continue to explore new rooms to find the target room. target It should be noted that due to the possible errors between the actual room layout and the common sense of the task parsing model, when it is determined that there is no room of the target type in the environment, the semantic category collection information (used to indicate whether the object corresponding to the semantic category has been collected) is recorded in the historical subtask information Info sub and jumps to step S12 to reorder the subtask priorities to determine the next subtask.

[0123] S42, Target Object Navigation

[0124] Since the layout of objects in indoor environments often has strong correlations, for example, tables are usually next to chairs, we can use common sense of object layout to guide the robot to quickly find the target semantic category set C sub The first target semantic category in Specifically, the semantic segmentation model is used to extract the semantic information in the current observation image, and the information and the current target object are input into the task parsing model. The task parsing model will calculate the probability of association between the object and the current target object based on the common sense of the position distance between different objects and the current target object, and guide the robot to explore the area with high probability until the current target object is found.

[0125] Furthermore, since there may be errors between the actual layout of objects in the room and the common sense possessed by the task parsing model, during the navigation process, the task parsing model will dynamically adjust the navigation order of the current target object according to the environmental information collected by the robot in real time, and realize the dynamic planning of the current subtask content. Specifically: when it is determined that there is no object in the room with the current semantic category When an object is found, the task parsing model will dynamically adjust the content of the current subtask and assign the next semantic category The object with the target object is set as the current target object and the search continues; in addition, the task parsing model will dynamically adjust the order of the target semantic categories in the target semantic category set based on the real-time observation image of the robot. For example, when navigating to the object with the semantic category In the process of finding the target object, the robot first observes the semantic category If the target object is a semantic object, the semantic category is collected first. data and will as the next target object.

[0126] S43. Target object data collection

[0127] See also Figure 6 Due to the lack of semantic truth information of the current environment, it is difficult to directly obtain the observed poses where the perception model performs poorly. In order to find observation data that can improve the performance of the perception model, a knowledge base is built in advance. By querying the knowledge base for observation images that are most similar to the current observation state and have a high perception model inconsistency score as external knowledge prompts for the pose prediction model, it is possible to infer the locations where the perception model may perform poorly in the current environment, thereby guiding the robot to explore these areas. The current target object data collection specifically includes the following steps:

[0128] S431, target object retrieval

[0129] Given the target semantic category c in the current observation image, for the current observation image obs c,curr , using the visual model (in this embodiment, the pre-trained model CLIP is used as the visual model) to extract the current observation image obs c,curr Observational features of feat c,curr :

[0130] feat c,curr =φ(obs c,curr )

[0131] Calculate the current observation features feat respectively c,curr The similarity simscore with each object image feature belonging to the target semantic category c in the image feature knowledge sub-base constructed in step S22 c,i :

[0132]

[0133]

[0134] in, is the current observation feature feat c,currThe similarity with the jth observed image feature of the i-th object belonging to the target semantic category c in the image feature knowledge sub-base constructed in step S22, similarity() calculates the cosine similarity between the input variables;

[0135] Take the maximum similarity score as the similarity score sim of the i-th object belonging to the target semantic category c c,i :

[0136] sim c,i =max(simscore c,i )

[0137] Calculate the number corresponding to the object with the highest object similarity score as the object retrieval number belonging to the target semantic category c

[0138]

[0139] S432, Target object observation information retrieval

[0140] The object retrieval number belonging to the target semantic category c obtained in step S431 Directly retrieve the first K observation images of the object corresponding to the number from the observation information knowledge sub-base constructed in step S23 as K reference observation images to form a reference observation image set

[0141]

[0142] in, is the first The comparison observation images of objects are sorted from high to low according to the inconsistency score, Represents the first The kth control observation image corresponding to the object.

[0143] S433, Navigation and observation data collection based on hierarchical targets

[0144] In order to guide the robot to reach and compare the observed image set The corresponding observation angle data is collected, and each of the K comparison observation images retrieved in step S432 is analyzed using the posture prediction large model to predict the observation position and observation angle of the object belonging to the target semantic category c in the kth comparison observation image, which is recorded as:

[0145]

[0146] Among them, pos k ,rot kare the predicted observation position and observation angle of the kth control observation image, respectively, and ψ() is the large model for pose prediction;

[0147] The predicted observation positions and observation angles of the K comparison observation images are set as sub-target sub, denoted as:

[0148] sub=[sub 1 ,sub 2 ,...,sub k ,...,sub K ]

[0149] sub k =[pos k ,rot k ]

[0150] Among them, sub k is the predicted observation pose of the kth reference observation image;

[0151] Use the underlying navigation model to plan the shortest path to guide the robot to take actions to reach the observation locations of the sub-goals Then adjust the observation angle of the robot camera to achieve the observation angle rot of the target object with semantic category c. k ;

[0152] The underlying navigation model guides the robot to continuously navigate in the environment to collect observation data on the position and orientation of the sub-targets. Finally, for objects belonging to the target semantic category c, new observation data about the object will be obtained:

[0153]

[0154] In addition, new observation data is obtained through the label generation module Corresponding semantic tags

[0155]

[0156] Among them, ρ() is the label generation module. It should be noted that the label can be obtained by calling the environment interface or generated by projecting the semantic point cloud.

[0157] Finally, we will get a dataset of labeled objects belonging to the target semantic category c:

[0158]

[0159] S434, subtask data set generation

[0160] After collecting information about the current target semantic category After the data of C is collected, steps S42 to S433 are repeated until all C sub In addition, due to the possible discrepancy between the layout of objects in the actual room and the common sense of the large model, when the navigation process is carried out, it is found that the object C that is not included in the subtask but belongs to C new When C new ∈C, Then, steps S431 to S433 are repeated to collect information about the semantic category C. new data, and the semantic category C new The collection information is written into the historical subtask information Info sub middle.

[0161] Finally, we will get the data set for the current subtask:

[0162]

[0163] in, Data collected for the mth subtask.

[0164] Record the current subtask content and the semantic category of the data collected by the current subtask to the historical subtask information Info sub middle.

[0165] In some embodiments, see Figure 7 , step S5 specifically includes the following steps:

[0166] After the current subtask is executed, the task instructions, environmental information, perception model performance to be improved, and historical subtask information generated during the subtask execution are stored in the Info file. sub The task planning model will input the data into the task planning model, which will determine the semantic categories of the data that have not been collected yet based on the historical subtask information. Then, based on the environmental information, the performance of the perception model to be improved, and the semantic category Generate the next subtask instructions to collect categories After the next subtask is determined, the process returns to step S3 to continue collecting the next subtask data until all subtasks are completed.

[0167] In some embodiments, step S6 specifically includes the following steps:

[0168] S61. Task Dataset Generation

[0169] After executing all subtasks, the task dataset will be obtained:

[0170]

[0171] S62, perception model performance update to be improved

[0172] Using the collected task data set data C Update the perception model performance to be improved, recorded as:

[0173]

[0174] Among them, θ per is the network parameter of the perception model to be improved, and π() is the model performance improvement method. It should be noted that there are many methods to improve model performance, such as retraining and fine-tuning, which can achieve the purpose of improving model performance. The network parameters of the perception model after using the task dataset.

[0175] It should be noted that the task planning big model, task analysis big model and posture prediction big model in this embodiment are all big models built based on the existing ChatGPT framework. In addition, any visual-language big model can also be used.

[0176] The second aspect of the present disclosure provides an embodied learning device based on dynamic subtask planning, comprising:

[0177] The first module is configured to generate multiple subtasks based on the task instruction using the task planning model, sort the priorities of the subtasks based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and use the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtask; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public data set; the environmental information is represented by the first-person perspective observation image captured by the robot;

[0178] The second module is configured to construct a knowledge base for assisting the large model of pose prediction to evaluate the performance of the perception model to be improved, the knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base, the image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score;

[0179] The third module is configured to use the task parsing model to parse the current subtask into several executable high-level operations, wherein the high-level operations include three categories: target room navigation, target object navigation, and target object data collection, and determine the target room type to be visited and the target semantic category set to be collected based on the current subtask content;

[0180] The fourth module is configured to, when the robot performs the high-level operation of the analysis based on the underlying action plan, first enable the robot to navigate to the target room, then navigate to the target object in the target room in turn, retrieve relevant context from the knowledge base as an external knowledge prompt for the pose prediction large model, and guide the robot to collect data on the target object; during the navigation process, the task analysis large model dynamically adjusts the navigation order of the target object according to the environmental information collected by the robot in real time, so as to realize the dynamic planning of the current subtask content;

[0181] The fifth module is configured to feed back the semantic category of the data collected when the robot performs the current subtask as the current subtask information to the task planning model after the current subtask is completed. The task planning model will dynamically plan the next subtask according to the feedback information, task instructions, the performance of the perception model to be improved, and environmental information;

[0182] The sixth module is configured to improve the perception performance of the perception model to be improved by using the data set collected during the navigation process.

[0183] It should be noted that the aforementioned explanation of an embodiment of an embodied learning method based on dynamic subtask planning is also applicable to an embodied learning device based on dynamic subtask planning in this embodiment, and will not be repeated here.

[0184] In order to implement the above embodiment, the embodiment of the present disclosure further proposes a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to execute the dynamic embodied scene generation method of the above embodiment.

[0185] Reference below Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that the electronic device in the embodiments of the present disclosure may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, servers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0186] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 to a random access memory (RAM) 103. In the RAM 103, various programs and data required for the operation of the electronic device are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0187] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0188] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0189] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0190] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0191] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the embodied learning method based on dynamic subtask planning.

[0192] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, python, and conventional procedural programming languages, such as "C-" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0193] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0194] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0195] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0196] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0197] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0198] A person of ordinary skill in the art may understand that all or part of the steps carried out in the above-mentioned embodiment method may be completed by instructing the relevant hardware through a program, and the developed program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0200] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An embodied learning method based on dynamic subtask planning, characterized in that: include: S1. Generate multiple subtasks using the task planning model based on the task instruction, sort the priorities of the subtasks based on the historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and use the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtask; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public data set; the environmental information is represented by the first-person perspective observation image captured by the robot; S2. Construct a knowledge base for assisting the large model of posture prediction to evaluate the performance of the perception model to be improved, wherein the knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base, wherein the image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score; S3. parse the current subtask into several executable high-level operations using the task parsing model. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. The target room type to be visited and the target semantic category set to be collected are determined based on the current subtask content. S4, the robot performs the high-level operation of the analysis based on the low-level action plan, first navigates to the target room, then navigates to the target object in the target room in sequence, retrieves the relevant context from the knowledge base as the external knowledge prompt of the pose prediction large model, and guides the robot to collect data on the target object; During the navigation process, the task parsing model dynamically adjusts the navigation order of the target object according to the environmental information collected by the robot in real time, so as to realize the dynamic planning of the current subtask content; S5. After the current subtask is completed, the semantic category of the data collected when the robot performs the current subtask is fed back to the task planning model as the current subtask information. The task planning model will dynamically plan the next subtask according to the feedback information, task instructions, the performance of the perception model to be improved, and environmental information, and return to step S3 until all subtasks are completed; S6. Using the data set collected during the navigation process to improve the perception performance of the perception model to be improved.

2. The embodied learning method according to claim 1, characterized in that: In step S1, the task planning model divides several types of objects specified by the task instructions that need to improve perception performance into multiple subtasks according to room types based on the common sense of the association between room types and object categories. The subtasks are represented by the target room type and the semantic category in the target room.

3. The embodied learning method according to claim 1, characterized in that: In step S1, the priorities of the subtasks are sorted, specifically including: Based on the collection of objects of various categories in the historical subtask information, the task planning model gives priority to collecting data in rooms where the semantic segmentation accuracy scores of the perception model to be improved are low and the object categories that have not been collected are located. When the task planning model determines that the room the robot is currently in is the room type corresponding to one of the subtasks based on the current environmental information captured by the robot, the subtask corresponding to the current room is given priority.

4. The embodied learning method according to claim 1, characterized in that: Step S2 specifically includes: S21. Data Collection Collect a large number of perception models to be improved as specified in the task instructions. The data of the semantic category and the corresponding semantic truth value, for any semantic category Collect its observation image data Semantic category The observed image data of the i-th object in, n is the semantic category For each object, multiple observation image data are collected around the object, and the semantic truth value corresponding to each observation image data is obtained, which are recorded as and Semantic category The jth observed image of the i-th object in For the observed image The corresponding semantic truth value, in which the value of the pixel coordinate position corresponding to the semantics of object i is 1, and the rest are 0. The semantic category of the collection The total number of observed images of the i-th object in; S22. Construction of image feature knowledge sub-base Extract observation image Image features Use all extracted image features to build an image feature knowledge sub-base Database feat ; S23. Construction of observation information knowledge sub-base The observed image is obtained using the perception model to be improved Prediction semantic graph of The predicted semantics of the i-th object in the predicted semantic map corresponds to the pixel coordinate position value of 1, and the rest are 0; using the predicted semantic map Semantic truth value The inconsistency between them is used to calculate the score of the perception model to be improved on the observation The specific calculation method is as follows: Where W and H are the width and height of the observed image, respectively. represents the predicted semantic value at the pixel coordinate position [w,h] in the observed image, Representative observation image The semantic truth value at the pixel coordinate position [w,h]; The inconsistency perception scores of all observed images of each object Sort from high to low and return the observed image number corresponding to the score The observed images corresponding to the inconsistency scores from high to low are denoted as So using the observed image Constructing the observation information knowledge sub-database obs .

5. The embodied learning method according to claim 4, characterized in that: Step S4 specifically includes: S41. Target room navigation The semantic information in the current observation image is extracted by using the perception model to be improved, and is input into the task parsing model. The room layout common sense possessed by the task parsing model is used to determine the type of the current room. When the task parsing model determines that the current room is not the target room, the room layout common sense is used to guide the robot to leave the current room, and continue to explore new rooms to find the target room. When the robot is in the target room, navigation to the target room is stopped. When it is determined that there is no room of the target type in the environment, the semantic category collection information is recorded in the historical subtask information, and the subtask priorities are reordered to determine the next subtask. S42, Target Object Navigation The semantic information in the current observation image is extracted by using the perception model to be improved, and the semantic information is input into the task parsing model together with the object corresponding to the current target semantic category in the target semantic category set, that is, the current target object. The task parsing model uses the common sense of the position distance between different objects and the current target object to calculate the association probability between the object and the current target object, and guides the robot to explore the area with high probability until the current target object is found. When the task parsing model determines that there is no target object corresponding to the current target semantic category, it will dynamically adjust the current subtask content, set the object corresponding to the next target semantic category as the current target object, and continue searching; at the same time, the task parsing model will dynamically adjust the order of the target semantic categories in the target semantic category set based on the real-time observation image of the robot; S43. Target object data collection S431, target object retrieval Given the target semantic category c in the current observation image, extract the current observation image obs c,curr Observational features of feat c,curr ; Calculate the observed features feat respectively c,curr The similarity simscore with each object image feature belonging to the target semantic category c in the image feature knowledge sub-base c,i , take the maximum value of the similarity score as the similarity score sim of each object belonging to the target semantic category c c,i ; Calculate the number corresponding to the object with the highest object similarity score as the object retrieval number belonging to the target semantic category c S432, Target object observation information retrieval Search by object number The first K observation images of the object corresponding to the number are retrieved from the observation information knowledge sub-base as K comparison observation images to form a comparison observation image set. S433, Navigation and observation data collection based on hierarchical targets The pose prediction model is used to predict the control observation image set. Analyze each control observation image in the kth control observation image and predict the observation position pos of the object belonging to the target semantic category c in the kth control observation image k and observation angle rot k , set the observation positions and observation angles of the predicted K comparison observation images as sub-target sub; Make the robot reach each observation position pos of the sub-goal sub by the shortest path k At the observation angle rot k Make observations and obtain the corresponding new observation images A total of K images are obtained, and each new observation image is obtained Corresponding semantic tags Construct a dataset data of objects belonging to the target semantic category c from K new observation images and their semantic labels c ; S434, subtask data set generation After the data on the current target semantic category is collected, steps S42 to S433 are repeated until the data on all semantic categories in the target semantic category set are collected to obtain the data set for the current subtask. The current subtask content and the semantic category of the data collected by the subtask are recorded in the historical subtask information.

6. The embodied learning method according to claim 5, characterized in that: In step S431, cosine similarity is used to measure the similarity simscore c,i .

7. The embodied learning method according to claim 1, characterized in that: In step S5, when the task planning model dynamically plans the next subtask, it first determines the semantic category for which data has not been collected according to the historical subtask information. Then, based on the environmental information, the performance of the perception model to be improved, and the semantic category Generate the next subtask to collect semantic categories Related data to ensure that data from all semantic categories are collected.

8. The embodied learning method according to claim 1, characterized in that: In step S6, the perception model to be improved is fine-tuned or trained using the data set collected during the navigation process.

9. A device based on the embodied learning method according to any one of claims 1 to 8, characterized in that: include: The first module is configured to generate multiple subtasks based on the task instruction using the task planning model, sort the priorities of the subtasks based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and use the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtask; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public data set; the environmental information is represented by the first-person perspective observation image captured by the robot; The second module is configured to construct a knowledge base for assisting the large model of pose prediction to evaluate the performance of the perception model to be improved, the knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base, the image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score; The third module is configured to use the task parsing model to parse the current subtask into several executable high-level operations, wherein the high-level operations include three categories: target room navigation, target object navigation, and target object data collection, and determine the target room type to be visited and the target semantic category set to be collected based on the current subtask content; The fourth module is configured to, when the robot performs the high-level operation of the analysis based on the underlying action plan, first enable the robot to navigate to the target room, then navigate to the target object in the target room in turn, retrieve relevant context from the knowledge base as an external knowledge prompt for the pose prediction large model, and guide the robot to collect data on the target object; during the navigation process, the task analysis large model dynamically adjusts the navigation order of the target object according to the environmental information collected by the robot in real time, so as to realize the dynamic planning of the current subtask content; The fifth module is configured to feed back the semantic category of the data collected when the robot performs the current subtask as the current subtask information to the task planning model after the current subtask is completed. The task planning model will dynamically plan the next subtask according to the feedback information, task instructions, the performance of the perception model to be improved, and environmental information; The sixth module is configured to improve the perception performance of the perception model to be improved by using the data set collected during the navigation process.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the embodied learning method according to any one of claims 1 to 8.

Citation Information

Patent Citations

  • Multi-robot online task allocation and execution method and device and storage medium

    CN115284288A

  • Intelligent task sequence planning method based on language vision large model and knowledge graph

    CN117874258A

  • Robot intelligent control method and system, electronic equipment and medium

    CN118123799A

  • Target navigation method and system based on hierarchical semantic map

    CN118189961A

  • Multi-robot cooperative control method and system based on large language model

    CN118386248A

Cited By

  • Land resource environment monitoring control system

    CN119165804A

  • Semantic navigation method based on large model, article taking and delivering method and robot

    CN119188791A

  • Navigation path planning method, device, equipment, robot, medium and product

    CN119347786A

  • Deformable object shape control method based on visual language model and historical data learning

    CN119820579A

  • Tool use command planning method based on large language model and fine tuning optimization

    CN119886200A