Embodied learning method, device and storage medium based on dynamic subtask planning

Abstract

This disclosure proposes an embodied learning method based on dynamic subtask planning, including: a task planning model generates multiple subtasks based on task instructions, the performance of the perception model to be improved, and the environmental information captured by the robot, and prioritizes them to determine the current subtask; a task parsing model parses the subtask content to guide the robot in the environment, and uses a pose prediction model to guide the robot in data collection. During navigation, the task parsing model dynamically adjusts the subtask content based on environmental information; after the current subtask is completed, the task planning model dynamically plans subsequent subtasks; after all subtasks are completed, the data set collected during navigation is used to improve the performance of the perception model. This invention enables the robot to collect new data or new knowledge that is beneficial to improving perception capabilities, and further improves the robot's understanding of the environment by learning the collected new data or new knowledge.

Description

Technical Field

[0001] The present disclosure relates to the fields of robot navigation and computer vision, and in particular to an embodied learning method, device, and storage medium based on dynamic subtask planning. Background Art

[0002] The diversity of application scenarios places high demands on robots' ability to learn knowledge. For example, in the military, robots must continuously learn combat strategies and tactics to adapt to changing war situations; in the medical field, robots must continuously learn diagnostic and treatment methods to provide better medical services; and in the field of service robots, robots must continuously learn how to better understand and respond to user commands to better meet user needs. Therefore, robots must possess the ability to learn knowledge and adapt to ever-changing and expanding application scenarios by updating their knowledge and skills. This task is embodied learning.

[0003] Currently, reinforcement learning is widely used to train robot exploration strategies based on differences in perception results. However, this approach has its limitations. First, it requires a large amount of data for training, which undoubtedly increases the complexity and difficulty of research. Second, this method may lead to redundant exploration paths, that is, the robot may repeatedly explore certain areas. Furthermore, this method may also lead to a long tail effect in the dataset, that is, too little or no data for certain categories.

[0004] Large models, equipped with scene-specific common sense and planning capabilities, can continuously adjust planning goals during exploration, thus avoiding the aforementioned issues. For example, large models can determine room type to prevent the robot from repeatedly exploring areas of a room it has already explored. Furthermore, large models can analyze currently collected data and adjust subsequent data collection goals, thus avoiding the long-tail effect in datasets. Therefore, large models have great potential for addressing these issues. Currently, there are no technologies that use large models for embodied learning in robots. Summary of the Invention

[0005] The present disclosure aims to solve one of the technical problems existing in the existing related technologies at least to a certain extent.

[0006] To this end, the present disclosure provides an embodied learning method, device and storage medium based on dynamic subtask planning, which enables the robot to collect new data or new knowledge that is conducive to improving its perception ability, and further improves the robot's understanding of the environment by learning the collected new data or new knowledge, thereby achieving a rapid and effective improvement of the robot's perception ability under limited training data and training time.

[0007] In order to achieve the above objectives, the present disclosure adopts the following technical solutions:

[0008] The first aspect of the present disclosure provides an embodied learning method based on dynamic subtask planning, comprising:

[0009] S1. Based on the task instruction, a task planning model is used to generate multiple subtasks. The priorities of the subtasks are sorted based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and the subtask with the highest priority is used as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public dataset; the environmental information is represented by the first-person perspective observation image captured by the robot;

[0010] S2. Construct a knowledge base for assisting a large pose prediction model in evaluating the performance of the perception model to be improved. The knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot. The observation information knowledge sub-base stores data of multiple first-person perspective observation images of different object instances. The data of each object instance is stored in descending order based on the perception score.

[0011] S3. Utilize the task parsing model to parse the current subtask into several executable high-level operations. These high-level operations fall into three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the target room type to be navigated to and the target semantic category set for data collection are determined.

[0012] S4, the robot performs the parsed high-level operations based on the low-level action plan, first navigating to the target room, then sequentially navigating to the target object in the target room, retrieving relevant context from the knowledge base as external knowledge prompts for the pose prediction large model, and guiding the robot to collect data on the target object;

[0013] During the navigation process, the task parsing model dynamically adjusts the navigation sequence of the target object based on the environmental information collected by the robot in real time, thereby achieving dynamic planning of the current subtask content.

[0014] S5. After the current subtask is completed, the semantic category of the data collected when the robot performs the current subtask is fed back to the task planning model as the current subtask information. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information, and return to step S3 until all subtasks are completed;

[0015] S6. Utilize the data set collected during the navigation process to improve the perception performance of the perception model to be improved.

[0016] In some embodiments, in step S1, the task planning model divides several types of objects specified in the task instructions that require improved perception performance into multiple subtasks according to room types based on the common sense of the association between room types and object categories. The subtasks are represented by the target room type and the semantic category within the target room.

[0017] In some embodiments, in step S1, the priorities of the subtasks are sorted, specifically including:

[0018] Based on the collection of objects of various categories in the historical subtask information, the large task planning model prioritizes data collection in rooms where the semantic segmentation accuracy of the perception model to be improved is low and the object categories that have not been collected are located.

[0019] When the task planning model determines that the room the robot is currently in is the room type corresponding to one of the subtasks based on the current environmental information captured by the robot, the subtask corresponding to the current room is given priority.

[0020] In some embodiments, step S2 specifically includes:

[0021] S21. Data Collection

[0022] Collect a large number of data containing the perception model to be improved specified in the task instructions The data of the semantic category and the corresponding semantic truth value, for any semantic category Collect its observation image data Semantic category The observed image data of the i-th object in, n is the semantic category For each object, multiple observation image data are collected around the object, and the semantic truth value corresponding to each observation image data is obtained, which are recorded as and Semantic category The jth observation image of the i-th object in, For the observed image The corresponding semantic truth value, in which the pixel coordinate position of the semantic object i is 1, and the rest are 0. The semantic categories collected The total number of observed images of the i-th object in ;

[0023] S22. Construction of image feature knowledge sub-base

[0024] Extract observation images Image features Utilize all extracted image features to build an image feature knowledge sub-base Database feat ;

[0025] S23. Construction of Observation Information Knowledge Sub-base

[0026] Use the perception model to be improved to obtain the observed image Prediction semantic graph of The pixel coordinate position value of the predicted semantics corresponding to the i-th object in the predicted semantic map is 1, and the others are 0; and semantic truth value The inconsistency between them is used to calculate the score of the perception model to be improved on the observation The specific calculation method is as follows:

[0027]

[0028] Where W and H are the width and height of the observed image respectively, Represents the predicted semantic value at the pixel coordinate position [w,h] in the observed image, Representative observation image The semantic truth value at the pixel coordinate position [w,h];

[0029] The inconsistency perception score of all observed images of each object Sort from high to low and return the observed image number corresponding to the score The observed images corresponding to the inconsistency scores from high to low are denoted as Using the observed image Constructing the observation information knowledge sub-database obs .

[0030] In some embodiments, step S4 specifically includes:

[0031] S41. Target room navigation

[0032] The perception model to be improved is used to extract semantic information from the current observation image and input it into the task parsing model. The room layout common sense possessed by the task parsing model is used to determine the current room type. When the task parsing model determines that the current room is not the target room, the room layout common sense is used to guide the robot to leave the current room and continue exploring new rooms to find the target room. When the robot is located in the target room, target room navigation is stopped. When it is determined that there is no room of the target type in the environment, the semantic category collection information is recorded in the historical subtask information, and the subtask priorities are reordered to determine the next subtask.

[0033] S42, Target Object Navigation

[0034] The perception model to be improved is used to extract semantic information from the current observation image, and inputs it together with the object corresponding to the current target semantic category in the target semantic category set, that is, the current target object, into the task parsing model. The task parsing model uses the common sense of the position and distance between different objects and the current target object to calculate the probability of association between the object and the current target object, and guides the robot to explore the area with high probability until the current target object is found.

[0035] When the task parsing model determines that there is no target object corresponding to the current target semantic category, it will dynamically adjust the current subtask content, set the object corresponding to the next target semantic category as the current target object, and continue searching; at the same time, the task parsing model will dynamically adjust the order of the target semantic categories in the target semantic category set based on the robot's real-time observation image;

[0036] S43. Target object data collection

[0037] S431, target object retrieval

[0038] Given the target semantic category c in the current observation image, extract the current observation image obs c,curr Observational features of feat c,curr ; Calculate the observation features feat respectively c,curr The similarity simscore with each object image feature belonging to the target semantic category c in the image feature knowledge sub-base c,i , take the maximum similarity score as the similarity score sim of each object belonging to the target semantic category c c,i ; Calculate the number corresponding to the object with the highest object similarity score as the object retrieval number belonging to the target semantic category c

[0039] S432, target object observation information retrieval

[0040] Search by object number Retrieve the first K observation images of the object corresponding to the number from the observation information knowledge sub-base as K comparison observation images to form a comparison observation image set

[0041] S433, Navigation and observation data collection based on hierarchical targets

[0042] The pose prediction model is used to predict the control observation image set Analyze each control observation image in the k-th control observation image and predict the observation position pos of the object belonging to the target semantic category c in the k-th control observation image k and observation angle rot k , set the observation position and observation angle of the predicted K comparison observation images as the sub-target sub;

[0043] Make the robot reach each observation position pos of the sub-goal sub by the shortest path k At the observation angle rot k Make observations and obtain the corresponding new observation images A total of K images are obtained, and each new observation image is obtained Corresponding semantic tags Construct a dataset data of objects belonging to the target semantic category c from K new observation images and their semantic labels c ;

[0044] S434. Subtask dataset generation

[0045] After collecting data about the current target semantic category, repeat steps S42 to S433 until all the semantic categories in the target semantic category set are collected to obtain the data set data for the current subtask. Csub , and record the current subtask content and the semantic category of the data collected by the subtask into the historical subtask information.

[0046] In some embodiments, in step S431, cosine similarity is used to measure the similarity simscore c,i .

[0047] In some embodiments, in step S5, when the task planning model dynamically plans the next subtask, it first determines the semantic category C for which data has not yet been collected based on the historical subtask information, and then generates the next subtask based on the environmental information, the perception model performance to be improved and the semantic category C to collect data related to the semantic category C to ensure that data of all semantic categories are collected.

[0048] In some embodiments, in step S6, the perception model to be improved is fine-tuned or trained using the data set collected during the navigation process.

[0049] A second aspect of the present disclosure provides a device based on the embodied learning method described in any embodiment of the first aspect of the present disclosure, comprising:

[0050] The first module is configured to generate multiple subtasks based on the task instruction using the task planning model, sort the priorities of the subtasks based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and select the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public dataset; and the environmental information is represented by the first-person perspective observation image captured by the robot;

[0051] The second module is configured to build a knowledge base for assisting the large pose prediction model in evaluating the performance of the perception model to be improved. The knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score;

[0052] The third module is configured to use the task parsing model to parse the current subtask into several executable high-level operations. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the target room type to be traveled to and the target semantic category set for data collection are determined.

[0053] The fourth module is configured to, when the robot executes the high-level parsed operation based on the low-level action plan, first navigate the robot to the target room, then sequentially navigate to the target object in the target room, retrieve relevant context from the knowledge base as external knowledge prompts for the pose prediction large model, and guide the robot to collect data on the target object; during the navigation process, the task parsing large model dynamically adjusts the navigation sequence of the target object based on the environmental information collected by the robot in real time, thereby realizing dynamic planning of the current subtask content;

[0054] The fifth module is configured to, after the current subtask is completed, feed back the semantic category of the data collected by the robot while executing the current subtask as the current subtask information to the task planning model. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information;

[0055] The sixth module is configured to improve the perception performance of the perception model to be improved by using the data set collected during the navigation process.

[0056] A third aspect of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the embodied learning method described in any embodiment of the first aspect of the present disclosure.

[0057] Compared with the prior art, the present invention has the following characteristics and beneficial effects:

[0058] This disclosure provides an embodied learning method based on dynamic subtask planning, enabling robots to collect new data or knowledge that enhances their perception capabilities. Learning from this new data or knowledge further improves the robot's understanding of the environment. Specifically, subtasks are generated based on the room and object layout knowledge of a large task parsing model. Subtask priorities are then dynamically adjusted based on historical subtask information, the performance of the perception model to be improved, and environmental information. Furthermore, during subtask completion, subtask content is dynamically adjusted based on environmental information. This method pre-collects multi-view observation data from a large number of object instances, extracts data features, and constructs an image feature knowledge sub-base. The collected data is then measured based on a predefined semantic consistency perception score, and the data is sorted by perception score to construct an observation information knowledge sub-base. The method then utilizes environmental information, combined with data with similar context retrieved from the constructed knowledge base, to predict poses with poor perception performance using a large pose prediction model. Therefore, this method achieves effective environmental exploration without requiring extensive data or time-consuming training. Furthermore, efficient environmental exploration is achieved by dynamically adjusting subtask priorities and content. Learning from the data collected during exploration improves the performance of the perception model in new environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an overall flow chart of the embodied learning method provided by the embodiment of the present disclosure.

[0060] Figure 2 Schematic diagram of initial subtask planning in the embodied learning method provided by an embodiment of the present disclosure.

[0061] Figure 3This is a schematic diagram of constructing an image feature knowledge sub-base in the embodied learning method provided in an embodiment of the present disclosure.

[0062] Figure 4 It is a schematic diagram of constructing an observation information knowledge sub-base in the embodied learning method provided in an embodiment of the present disclosure.

[0063] Figure 5 It is a flowchart of subtask parsing in the embodied learning method provided by an embodiment of the present disclosure.

[0064] Figure 6 This is a flowchart of subtask data collection in the embodied learning method provided by an embodiment of the present disclosure.

[0065] Figure 7 Schematic diagram of subtask updating in the embodied learning method provided by an embodiment of the present disclosure.

[0066] Figure 8 It is a structural diagram of the electronic device provided in the third aspect of the present disclosure. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0068] On the contrary, this application covers any alternatives, modifications, equivalents, and solutions made within the spirit and scope of this application as defined by the claims. Furthermore, to facilitate a better understanding of this application, certain specific details are described in detail below in the detailed description of this application. Those skilled in the art will be able to fully understand this application without these details.

[0069] See also Figure 1 The embodiment of the present disclosure provides an embodied learning method based on dynamic subtask planning for indoor scenes. The method of this embodiment includes the following steps:

[0070] S1, based on the task instructions, use the task planning model to generate multiple subtasks, and then based on the historical subtask information Info sub, the performance of the perception model to be improved and the environmental information captured by the robot are used to sort the priorities of the subtasks, and the subtask with the highest priority is used as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set in advance by the user, and the semantic category is represented by the object category name; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task of the semantic segmentation model pre-trained using the public dataset (in this embodiment, the pre-trained model Mask R-CNN (Mask Region-based Convolutional Neural Network) is used in the new environment; the environmental information is represented by the first-person perspective observation image captured by the RGB camera equipped with the robot;

[0071] S2. Build a knowledge base for assisting the large pose prediction model in evaluating the performance of the perception model to be improved. This knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores the image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores the data of multiple first-person perspective observation images of different object instances. The data of each object instance is stored in descending order based on the perception score.

[0072] S3. The task parsing model parses the current subtask into several executable high-level operations. These high-level operations fall into three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the model determines the target room type and the target semantic category set for data collection.

[0073] S4: The robot executes the high-level operations parsed in step S3 based on the low-level action plan. It first navigates to the target room, then sequentially navigates to the target object in the target room. It retrieves relevant context from the knowledge base built in step S2 as external knowledge prompts for the pose prediction model, guiding the robot to collect data on the target object.

[0074] During the navigation process, the task parsing model dynamically adjusts the navigation sequence of the target object based on the environmental information collected by the robot in real time, thus achieving dynamic planning of the current subtask content;

[0075] S5. After the current subtask is completed, the semantic category of the data collected by the robot when performing the current subtask is fed back to the task planning model as the current subtask information. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information, and return to step S3 until all subtasks are completed;

[0076] S6. Use the dataset collected during navigation to fine-tune or train the perception model to improve its perception performance.

[0077] In some embodiments, see Figure 2 , step S1 specifically includes the following steps:

[0078] S11. Subtask generation

[0079] Get the task instructions set by the user, which must specify the semantic category of the perception model to be improved Semantic Category is the semantic category name of the total object that needs to improve perception performance. Since indoor scenes are often structured, the object categories in different types of rooms are often fixed. Based on the common sense of the association between room types and object categories in the task planning model, the task planning model is used to Class objects are divided into multiple subtasks based on room types. The subtasks are characterized by the target room type and the semantic category within the room. It should be noted that different rooms may contain the same semantic category at the same time. For example, both the living room and the bedroom may have chairs. Since the actual room layout may have errors with common sense, for example, there may be no chairs in the bedroom, all will be considered. The room types that may appear in the semantics of the class are considered, and the subtask content is dynamically adjusted in the subsequent steps to ensure that data for all semantic categories are collected. The class semantics include chair, sofa, plant, bed, and toilet. The task planning model will generate the following subtasks:

[0080] chair_sofa_in_livingroom();

[0081] toilet_in_bathroom();

[0082] bed_chair_plant_in_bedroom();

[0083] S12. Subtask Priority Sorting

[0084] The subtask priorities are sorted based on historical subtask information, the accuracy scores of each category in the perception model to be improved, and the current environment information. Specifically:

[0085] First, based on the historical subtask information Info subConsidering the data collection status of each category, we prioritize data collection in rooms with low semantic segmentation accuracy scores for the perception model to be improved and that have not been collected. For example, the performance scores of the perception model for semantic categories that have not been collected are: chair (0.48), plant (0.56), sofa (0.66), bed (0.68), and toilet (0.76). We set the subtasks corresponding to the categories with low collection scores to the highest priority, and then sort the subtask priorities from highest to lowest as follows:

[0086] bed_chair_plant_in_bedroom();

[0087] chair_sofa_in_livingroom();

[0088] toilet_in_bathroom();

[0089] In addition, the current environment information is input into the task planning model, which estimates the current room type. If the robot is in a room type corresponding to a subtask, this subtask will be given priority. For example, if the task planning model determines that the current room is a bathroom, the subtask priority will be adjusted to:

[0090] toilet_in_bathroom();

[0091] bed_chair_plant_in_bedroom();

[0092] chair_sofa_in_livingroom();

[0093] S13. Output current subtask

[0094] Task planning sorts subtasks according to their priorities and outputs the subtask with the highest current priority as the subtask that needs to be executed currently.

[0095] In some embodiments, step S2 specifically includes the following steps:

[0096] S21. Data Collection

[0097] Collect a large amount of The data of the class object and the corresponding semantic truth value. Collect its observation image data in Semantic category The observed image data of the i-th object in, n is the semantic category For each object, data is collected around the object, requiring that the object can be observed in the collected observation image, and the semantic truth value corresponding to the observation image is obtained by calling the environment API interface, which is recorded as and in Semantic category The jth observation image of the i-th object in, For the observed image The corresponding semantic truth value, in which the pixel coordinate position of the semantic object i is 1, and the rest are The semantic categories collected The total number of observed images of the i-th object in .

[0098] S22. Construction of image feature knowledge sub-base

[0099] See also Figure 3 , in order to build the image feature knowledge sub-base Database feat , using a visual model (in this embodiment, a pre-trained model CLIP (Contrastive Language-Image Pre-training) is used as a visual model) to extract the observed image Image features Record Where φ() is the pre-trained model CLIP. The extracted image features are saved in the image feature knowledge sub-base, which is recorded as:

[0100]

[0101]

[0102]

[0103] in, Image feature knowledge sub-database feat The semantic category is The image features of all observed images, Image feature knowledge sub-database feat Middle semantic category The image features of all observed images of the i-th object in, Image feature knowledge sub-database feat Middle semantic category The image features of the jth observation image of the i-th object in, Semantic category The total number of image features of the observed image of the i-th object in .

[0104] S23. Construction of Observation Information Knowledge Sub-base

[0105] See also Figure 4 , in order to build the observation information knowledge sub-base Database obs , use the perception model to be improved to predict the semantic graph, denoted as in, is the perception model to be improved, Semantic category The jth observation image of the i-th object in The predicted semantic map of the i-th object has a pixel coordinate value of 1, and the rest are 0. and semantic truth value In semantic categories The semantic inconsistency calculation on the observation score of the perception model to be improved The specific calculation method is as follows:

[0106]

[0107] Where W and H are the width and height of the observed image respectively, Represents the predicted semantic value at the pixel coordinate position [w,h] in the observed image, Represents the semantic truth value of the pixel coordinate position [w,h] in the observed image.

[0108] The inconsistency perception score of all observed images of each object Sort by high to low and return the observed image number corresponding to the score Denoted as:

[0109]

[0110] Among them, des_sort() represents descending sort.

[0111] The observed images corresponding to the inconsistency scores from high to low are recorded as:

[0112]

[0113] Store the extracted observation image into the observation information knowledge sub-base Database obs In Chinese, it is recorded as:

[0114]

[0115]

[0116]

[0117] in, Represents the observation information knowledge sub-base Database obs The semantic category is All observation image data, Represents the observation information knowledge sub-base Database obs The semantic category is All observed image data of the i-th object, Represents the observation information knowledge sub-base Database obs The semantic category is The jth observation image data of the i-th object.

[0118] In some embodiments, see Figure 5 , step S3 specifically includes the following steps:

[0119] The content of the current subtask outputted in step S13 is parsed using the task parsing model. First, the executable high-level operations are determined. The high-level operations include three categories: target room navigation, target object navigation, and target object data acquisition. Among them, target room navigation is used to guide the robot to reach the target room, such as guiding the robot to reach the bedroom; target object navigation is used to guide the robot to reach the vicinity of the target object in the target room, and face the object so that the object is located in the observation image, such as guiding the robot to reach the bed in the bedroom; target object data acquisition is used to guide the robot to collect data with poor target object perception accuracy. For example, for a TV, the robot can recognize that it is a TV from the image collected from the front, but cannot accurately recognize that it is a TV from the image collected from the side. The target object data acquisition operation guides the robot to collect data from the side, that is, data with poor perception performance. Then, based on the current subtask content, determine the target room type R that needs to be visited. target The target semantic category set for which data needs to be collected Where M is the total number of semantic categories that need to be collected for the current subtask. And C sub It may contain one or more semantic categories. Finally, the task parsing model is used to transform the subtask into a combination of multiple executable high-level operations.

[0120] In some embodiments, see Figure 5 , step S4 specifically includes the following steps:

[0121] S41. Target room navigation

[0122] Since indoor environments are often highly structured, that is, the room layouts are usually the same, such as bedrooms next to living rooms, and the object categories contained in different room types are often fixed. A semantic segmentation model is used (in this embodiment, a pre-trained model Mask R-CNN is used as a semantic segmentation model) to extract semantic information from the current observation image, and this information is input into the task parsing model. The room layout common sense of the task parsing model is used to determine the current room type. When the task parsing model determines that the current room is not the target room R target When the robot is in the target room R, it will use the room layout common sense of the task parsing large model to guide the robot to leave the current room and continue to explore the new room to find the target room. target It should be noted that due to the possible discrepancy between the actual room layout and the common sense of the task parsing model, when it is determined that there is no room of the target type in the environment, the semantic category collection information (used to indicate whether the object corresponding to the semantic category has been collected) is recorded in the historical subtask information Info sub and jumps to step S12 to reorder the subtask priorities to determine the next subtask.

[0123] S42, Target Object Navigation

[0124] Since the layout of objects in indoor environments often has strong correlations, for example, tables are often next to chairs, we can use common sense of object layout to guide the robot to quickly find the target semantic category set C sub The first target semantic category in The corresponding object is the current target object. Specifically, the semantic segmentation model is used to extract semantic information from the current observed image and input this information along with the current target object into the task parsing model. The task parsing model calculates the probability of association between the object and the current target object based on common sense about the position and distance between different objects and the current target object. The robot is guided to explore areas with high probability until it finds the current target object.

[0125] Furthermore, since there may be errors between the layout of objects in the actual room and the common sense possessed by the task parsing model, during the navigation process, the task parsing model will dynamically adjust the navigation order of the current target object according to the environmental information collected by the robot in real time, and realize the dynamic planning of the current subtask content. Specifically: when it is determined that there is no object in the room with the current semantic category When the object is found, the task parsing model will dynamically adjust the content of the current subtask and assign the next semantic category The object with the target object is set as the current target object and the search continues; in addition, the task parsing model will dynamically adjust the order of the target semantic categories in the target semantic category set based on the real-time observation image of the robot. For example, when navigating to the object with the semantic category In the process of the target object, the robot first observes the semantic category target objects, the semantic categories are collected first. data and will as the next target object.

[0126] S43. Target object data collection

[0127] See also Figure 6 Due to the lack of semantic truth information about the current environment, it is difficult to directly obtain observation poses where the perception model performs poorly. In order to find observation data that can improve the performance of the perception model, a knowledge base is constructed in advance. By querying the knowledge base for observation images that are most similar to the current observation state and have high perception model inconsistency scores as external knowledge prompts for the pose prediction model, it is possible to infer locations in the current environment where the perception model may perform poorly, thereby guiding the robot to explore these areas. The current target object data collection specifically includes the following steps:

[0128] S431, target object retrieval

[0129] Given the target semantic category c in the current observation image, for the current observation image obs c,curr , using the visual model (in this embodiment, the pre-trained model CLIP is used as the visual model) to extract the current observation image obs c,curr Observational features of feat c,curr :

[0130] feat c,curr =φ(obs c,curr )

[0131] Calculate the current observation features feat separately c,curr The similarity simscore of each object image feature belonging to the target semantic category c in the image feature knowledge sub-base constructed in step S22 c,i :

[0132]

[0133]

[0134] in, feat is the current observation feature c,currThe similarity between the jth observed image feature of the i-th object belonging to the target semantic category c in the image feature knowledge sub-base constructed in step S22 is calculated using similarity() to calculate the cosine similarity between the input variables.

[0135] Take the maximum similarity score as the similarity score sim of the i-th object belonging to the target semantic category c c,i :

[0136] sim c,i =max(simscore c,i )

[0137] Calculate the number corresponding to the object with the highest object similarity score as the object retrieval number belonging to the target semantic category c

[0138]

[0139] S432, target object observation information retrieval

[0140] The object retrieval number of the target semantic category c obtained in step S431 Directly retrieve the first K observation images of the object corresponding to the number from the observation information knowledge sub-base constructed in step S23 as K reference observation images to form the reference observation image set

[0141]

[0142] in, is the first The control observation images of objects are sorted from high to low according to the inconsistency score, Represents the first The kth control observation image corresponding to the object.

[0143] S433, Navigation and observation data collection based on hierarchical targets

[0144] In order to guide the robot to reach the reference observation image set The corresponding observation angle data is collected, and the pose prediction model is used to analyze each of the K reference observation images retrieved in step S432 to predict the observation position and observation angle of the object belonging to the target semantic category c in the k-th reference observation image, which is recorded as:

[0145]

[0146] Among them, pos k ,rot kare the predicted observation position and observation angle of the kth control observation image, respectively, and ψ() is the pose prediction model;

[0147] The observation positions and observation angles of the predicted K comparison observation images are set as sub-targets sub, which are recorded as:

[0148] sub=[sub 1 ,sub 2 ,...,sub k ,...,sub K ]

[0149] sub k =[pos k ,rot k ]

[0150] Among them, sub k is the predicted observation pose of the kth control observation image;

[0151] Use the underlying navigation model to plan the shortest path to guide the robot to take action to reach the observation locations of the sub-goals Then adjust the observation angle of the robot camera to achieve the observation angle rot of the target object with semantic category c. k ;

[0152] The underlying navigation model guides the robot to continuously navigate in the environment to collect observation data on the position and orientation of the sub-target Finally, for an object belonging to the target semantic category c, new observation data about the object will be obtained:

[0153]

[0154] In addition, new observation data is obtained through the label generation module Corresponding semantic tags

[0155]

[0156] Among them, ρ() is the label generation module. It should be noted that labels can be obtained by calling the environment interface or generated by projecting semantic point clouds.

[0157] Finally, we will get a dataset of labeled objects belonging to the target semantic category c:

[0158]

[0159] S434. Subtask dataset generation

[0160] After collecting the information about the current target semantic category After the data of C is collected, repeat steps S42 to S433 until all sub In addition, due to the possible discrepancy between the layout of objects in the actual room and the common sense of the large model, when the navigation process is carried out, it is found that the object C that is not included in the subtask but belongs to C new When C new ∈C, Then repeat steps S431 to S433 to collect information about the semantic category C new data, and the semantic category C new The collection information is written into the historical subtask information Info sub middle.

[0161] Finally, we will get the dataset for the current subtask:

[0162]

[0163] in, Data collected for the mth subtask.

[0164] Record the current subtask content and the semantic category of the data collected by the current subtask to the historical subtask information Info sub middle.

[0165] In some embodiments, see Figure 7 , step S5 specifically includes the following steps:

[0166] After the current subtask is executed, the task instructions, environmental information, the perception model performance to be improved, and the historical subtask information generated during the subtask execution are stored in Info. sub The data is input into the task planning model, which will determine the semantic categories of the data that have not been collected yet based on the historical subtask information. Then, based on the environmental information, the performance of the perception model to be improved, and the semantic category Generate the next subtask instructions to collect categories After the next subtask is determined, the process returns to step S3 to continue collecting data for the next subtask until all subtasks are completed.

[0167] In some embodiments, step S6 specifically includes the following steps:

[0168] S61. Task dataset generation

[0169] After executing all subtasks, the task dataset will be obtained:

[0170]

[0171] S62, performance update of perception model to be improved

[0172] Using the collected task dataset data C Update the performance of the perception model to be improved, recorded as:

[0173]

[0174] Among them, θ per is the network parameter of the perception model to be improved, and π() is the model performance improvement method. It should be noted that there are many methods to improve model performance, such as retraining and fine-tuning, which can achieve the purpose of improving model performance. These are the network parameters of the perception model after using the task dataset to improve it.

[0175] It should be noted that the task planning model, task analysis model and posture prediction model in this embodiment are all large models built based on the existing ChatGPT framework. In addition, any visual-language model can also be used.

[0176] The second aspect of the present disclosure provides an embodied learning device based on dynamic subtask planning, comprising:

[0177] The first module is configured to generate multiple subtasks based on the task instruction using the task planning model, sort the priorities of the subtasks based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and select the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public dataset; and the environmental information is represented by the first-person perspective observation image captured by the robot;

[0178] The second module is configured to build a knowledge base for assisting the large pose prediction model in evaluating the performance of the perception model to be improved. The knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score;

[0179] The third module is configured to use the task parsing model to parse the current subtask into several executable high-level operations. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the target room type to be traveled to and the target semantic category set for data collection are determined.

[0180] The fourth module is configured to, when the robot executes the high-level parsed operation based on the low-level action plan, first navigate the robot to the target room, then sequentially navigate to the target object in the target room, retrieve relevant context from the knowledge base as external knowledge prompts for the pose prediction large model, and guide the robot to collect data on the target object; during the navigation process, the task parsing large model dynamically adjusts the navigation sequence of the target object based on the environmental information collected by the robot in real time, thereby realizing dynamic planning of the current subtask content;

[0181] The fifth module is configured to, after the current subtask is completed, feed back the semantic category of the data collected by the robot while executing the current subtask as the current subtask information to the task planning model. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information;

[0182] The sixth module is configured to improve the perception performance of the perception model to be improved by using the data set collected during the navigation process.

[0183] It should be noted that the above explanation of an embodiment of an embodied learning method based on dynamic subtask planning is also applicable to an embodied learning device based on dynamic subtask planning in this embodiment, and will not be repeated here.

[0184] In order to implement the above embodiment, the embodiment of the present disclosure further proposes a computer-readable storage medium on which a computer program is stored. The program is executed by a processor to execute the dynamic embodied scene generation method of the above embodiment.

[0185] Reference below Figure 8 , which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure. It should be noted that the electronic devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs, desktop computers, and servers. Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0186] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 102 or a program loaded from a storage device 108 into a random access memory (RAM) 103. Various programs and data required for the operation of the electronic device are also stored in the RAM 103. The processing device 101, the ROM 102, and the RAM 103 are connected to each other via a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104.

[0187] Typically, the following devices may be connected to the I / O interface 105: an input device 106 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, etc.; an output device 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 108 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 109. The communication device 109 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 8 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0188] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, this embodiment includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 109, or installed from the storage device 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0189] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0190] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0191] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the embodied learning method based on dynamic subtask planning.

[0192] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C-" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0193] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0194] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0195] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0196] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing it in a suitable manner if necessary, and then storing it in a computer memory.

[0197] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0198] Those skilled in the art will understand that all or part of the steps carried out in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the developed program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0199] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0200] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. An embodied learning method based on dynamic subtask planning, characterized in that: include: S1. Based on the task instruction, a task planning model is used to generate multiple subtasks. The priorities of the subtasks are sorted based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and the subtask with the highest priority is used as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public dataset; the environmental information is represented by the first-person perspective observation image captured by the robot; S2. Construct a knowledge base for assisting a large pose prediction model in evaluating the performance of the perception model to be improved. The knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot. The observation information knowledge sub-base stores data of multiple first-person perspective observation images of different object instances. The data of each object instance is stored in descending order based on the perception score. S3. Utilize the task parsing model to parse the current subtask into several executable high-level operations. These high-level operations fall into three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the target room type to be navigated to and the target semantic category set for data collection are determined. S4, the robot performs the parsed high-level operations based on the low-level action plan, first navigating to the target room, then sequentially navigating to the target object in the target room, retrieving relevant context from the knowledge base as external knowledge prompts for the pose prediction large model, and guiding the robot to collect data on the target object; During the navigation process, the task parsing model dynamically adjusts the navigation sequence of the target object based on the environmental information collected by the robot in real time, thereby achieving dynamic planning of the current subtask content. S5. After the current subtask is completed, the semantic category of the data collected when the robot performs the current subtask is fed back to the task planning model as the current subtask information. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information, and return to step S3 until all subtasks are completed; S6. Utilize the data set collected during the navigation process to improve the perception performance of the perception model to be improved.

2. The embodied learning method according to claim 1, characterized in that In step S1, the task planning model divides the several types of objects specified in the task instructions that need to improve perception performance into multiple subtasks according to the room type based on the common sense association between room type and object category. The subtasks are represented by the target room type and the semantic category within the target room.

3. The embodied learning method according to claim 1, characterized in that In step S1, the priorities of the subtasks are sorted, specifically including: Based on the collection of objects of various categories in the historical subtask information, the large task planning model prioritizes data collection in rooms where the semantic segmentation accuracy of the perception model to be improved is low and the object categories that have not been collected are located. When the task planning model determines that the room the robot is currently in is the room type corresponding to one of the subtasks based on the current environmental information captured by the robot, the subtask corresponding to the current room is given priority.

4. The embodied learning method according to claim 1, wherein: Step S2 specifically includes: S21. Data Collection Collect a large number of data containing the perception model to be improved specified in the task instructions The data of the semantic category and the corresponding semantic truth value, for any semantic category Collect its observation image data Semantic category The observed image data of the i-th object in, n is the semantic category For each object, multiple observation image data are collected around the object, and the semantic truth value corresponding to each observation image data is obtained, which are recorded as and Semantic category The jth observation image of the i-th object in, For the observed image The corresponding semantic truth value, in which the pixel coordinate position of the semantic object i is 1, and the rest are 0. The semantic categories collected The total number of observed images of the i-th object in ; S22. Construction of image feature knowledge sub-base Extract observation images Image features Utilize all extracted image features to build an image feature knowledge sub-base Database feat ; S23. Construction of Observation Information Knowledge Sub-base Use the perception model to be improved to obtain the observed image Prediction semantic graph of The pixel coordinate position value of the predicted semantics corresponding to the i-th object in the predicted semantic map is 1, and the others are 0; and semantic truth value The inconsistency between them is used to calculate the score of the perception model to be improved on the observation The specific calculation method is as follows: Where W and H are the width and height of the observed image respectively, Represents the predicted semantic value at the pixel coordinate position [w,h] in the observed image, Representative observation image The semantic truth value at the pixel coordinate position [w,h]; The inconsistency perception score of all observed images of each object Sort from high to low and return the observed image number corresponding to the score The observed images corresponding to the inconsistency scores from high to low are denoted as Using the observed image Constructing the observation information knowledge sub-database obs .

5. The embodied learning method according to claim 4, characterized in that Step S4 specifically includes: S41. Target room navigation The perception model to be improved is used to extract semantic information from the current observation image and input it into the task parsing model. The room layout common sense possessed by the task parsing model is used to determine the current room type. When the task parsing model determines that the current room is not the target room, the room layout common sense is used to guide the robot to leave the current room and continue exploring new rooms to find the target room. When the robot is located in the target room, target room navigation is stopped. When it is determined that there is no room of the target type in the environment, the semantic category collection information is recorded in the historical subtask information, and the subtask priorities are reordered to determine the next subtask. S42, Target Object Navigation The perception model to be improved is used to extract semantic information from the current observation image, and inputs it together with the object corresponding to the current target semantic category in the target semantic category set, that is, the current target object, into the task parsing model. The task parsing model uses the common sense of the position and distance between different objects and the current target object to calculate the probability of association between the object and the current target object, and guides the robot to explore the area with high probability until the current target object is found. When the task parsing model determines that there is no target object corresponding to the current target semantic category, it will dynamically adjust the current subtask content, set the object corresponding to the next target semantic category as the current target object, and continue searching; at the same time, the task parsing model will dynamically adjust the order of the target semantic categories in the target semantic category set based on the robot's real-time observation image; S43. Target object data collection S431, target object retrieval Given the target semantic category c in the current observation image, extract the current observation image obs c,curr Observational features of feat c,curr ; Calculate the observation features feat respectively c,curr The similarity simscore with each object image feature belonging to the target semantic category c in the image feature knowledge sub-base c,i , take the maximum similarity score as the similarity score sim of each object belonging to the target semantic category c c,i ; Calculate the number corresponding to the object with the highest object similarity score as the object retrieval number belonging to the target semantic category c S432, target object observation information retrieval Search by object number Retrieve the first K observation images of the object corresponding to the number from the observation information knowledge sub-base as K comparison observation images to form a comparison observation image set S433, Navigation and observation data collection based on hierarchical targets The pose prediction model is used to predict the control observation image set Analyze each control observation image in the k-th control observation image and predict the observation position pos of the object belonging to the target semantic category c in the k-th control observation image k and observation angle rot k , set the observation position and observation angle of the predicted K comparison observation images as the sub-target sub; Make the robot reach each observation position pos of the sub-goal sub by the shortest path k At the observation angle rot k Make observations and obtain the corresponding new observation images A total of K images are obtained, and each new observation image is obtained Corresponding semantic tags Construct a dataset data of objects belonging to the target semantic category c from K new observation images and their semantic labels c ; S434. Subtask dataset generation After collecting data about the current target semantic category, repeat steps S42 to S433 until all semantic categories in the target semantic category set are collected to obtain the data set for the current subtask. The current subtask content and the semantic category of the data collected by the subtask are recorded in the historical subtask information.

6. The embodied learning method according to claim 5, characterized in that In step S431, cosine similarity is used to measure the similarity simscore c,i .

7. The embodied learning method according to claim 1, characterized in that In step S5, when the task planning model dynamically plans the next subtask, it first determines the semantic category for which data has not yet been collected based on the historical subtask information. Then, based on the environmental information, the performance of the perception model to be improved, and the semantic category Generate the next subtask to collect semantic categories Related data to ensure that data of all semantic categories are collected.

8. The embodied learning method according to claim 1, characterized in that In step S6, the perception model to be improved is fine-tuned or trained using the data set collected during the navigation process.

9. A device based on the embodied learning method according to any one of claims 1 to 8, characterized in that: include: The first module is configured to generate multiple subtasks based on the task instruction using the task planning model, sort the priorities of the subtasks based on historical subtask information, the performance of the perception model to be improved, and the environmental information captured by the robot, and select the subtask with the highest priority as the current subtask; wherein the task instruction is the semantic category of the perception model to be improved set by the user, and the semantic category is represented by the name of the object category; the historical subtask information is represented by the collection of objects of each category in the historical subtasks; the performance of the perception model to be improved is represented by the accuracy index of the semantic segmentation task in the new environment of the semantic segmentation model pre-trained using the public dataset; and the environmental information is represented by the first-person perspective observation image captured by the robot; The second module is configured to build a knowledge base for assisting the large pose prediction model in evaluating the performance of the perception model to be improved. The knowledge base includes an image feature knowledge sub-base and an observation information knowledge sub-base. The image feature knowledge sub-base stores image features of multiple first-person perspective observation images of different object instances captured by the robot; the observation information knowledge sub-base stores multiple first-person perspective observation image data of different object instances, and the data of each object instance is stored in descending order based on the perception score; The third module is configured to use the task parsing model to parse the current subtask into several executable high-level operations. The high-level operations include three categories: target room navigation, target object navigation, and target object data collection. Based on the current subtask content, the target room type to be traveled to and the target semantic category set for data collection are determined. The fourth module is configured to, when the robot executes the high-level parsed operation based on the low-level action plan, first navigate the robot to the target room, then sequentially navigate to the target object in the target room, retrieve relevant context from the knowledge base as external knowledge prompts for the pose prediction large model, and guide the robot to collect data on the target object; during the navigation process, the task parsing large model dynamically adjusts the navigation sequence of the target object based on the environmental information collected by the robot in real time, thereby realizing dynamic planning of the current subtask content; The fifth module is configured to, after the current subtask is completed, feed back the semantic category of the data collected by the robot while executing the current subtask as the current subtask information to the task planning model. The task planning model will dynamically plan the next subtask based on the feedback information, task instructions, the performance of the perception model to be improved, and environmental information; The sixth module is configured to improve the perception performance of the perception model to be improved by using the data set collected during the navigation process.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the embodied learning method according to any one of claims 1 to 8.

Images