Robot multi-task control model training method and device, robot, medium and program product
By generating extended task labels and selecting samples across pools, the problem of insufficient generalization ability of robot multi-task control models is solved, and knowledge sharing and training effect improvement are achieved among multiple tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GOERTEK INC
- Filing Date
- 2025-12-12
- Publication Date
- 2026-04-17
AI Technical Summary
The robot's multi-task control model has insufficient generalization ability, resulting in poor training performance and weak sharing ability among multiple tasks.
By generating extended task labels that match the current task labels, a retrieval-enhanced generation algorithm is used to determine the relevant target experience replay pool from all experience replay pools, and sample experience data is selected for model training.
It enhances the task-sharing capability of the robot's multi-task control model, thereby improving the model's generalization and training effectiveness.
Smart Images
Figure CN121880923A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of robot multi-task control model training technology, and in particular to a robot multi-task control model training method, device, robot, medium and program product. Background Technology
[0002] In training multi-task control models for robots, one-hot encoding or predefined fixed task labels are typically used to label each training task, and different training tasks are configured with corresponding experience replay pools. Thus, sample experience data can be directly selected from the corresponding experience replay pools based on one-hot encoding or fixed task labels to perform model training.
[0003] However, one-hot encoding or predefined fixed task labels can lead to a single information dimension in the training task and a weak ability to share information between multiple tasks. This results in poor generalization of the policies learned by the robot's multi-task control model and poor training performance. Summary of the Invention
[0004] The main purpose of this application is to provide a method, device, robot, medium and program product for training a robot multi-task control model, which aims to solve the technical problem of insufficient generalization ability of robot multi-task control models.
[0005] To achieve the above objectives, this application proposes a method for training a robot multi-task control model, which includes: Determine extended task labels that match the current task label of the current training task; wherein the current training task is one of several different types of preset tasks in the robot multi-task control model; From all the experience replay pools of the robot multi-task control model, determine at least one target experience replay pool associated with the extended task label; each training task has a corresponding experience replay pool. Select sample experience data from the target experience replay pool to obtain the sample set for the current training task; The multi-task control model of the robot is trained based on the sample set.
[0006] In some embodiments, determining an extended task label that matches the current task label of the current training task includes: Based on the current task label and the preset task corpus, an extended task label matching the current task label is generated using a retrieval-enhanced generation algorithm; the preset task corpus contains corpus information associated with multiple training tasks.
[0007] In some embodiments, based on the current task label and a preset task corpus, an enhanced retrieval generation algorithm is used to generate extended task labels that match the current task label, including: Based on the environmental state information of the current training task, the current task label, and the preset task corpus, an extended task label matching the current task label is generated using a retrieval-enhanced generation algorithm.
[0008] In some embodiments, determining at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot multi-task control model includes: Based on the correlation between the extended task tags and the task tags corresponding to each experience replay pool, at least one target experience replay pool associated with the extended task tags is identified.
[0009] In some embodiments, selecting sample experience data from the target experience replay pool to obtain the sample set for the current training task includes: For each target experience replay pool, the sample allocation information is determined based on the correlation between the extended task labels and the corresponding task labels of the target experience replay pool; For each target experience replay pool, sample experience data is selected from the target experience replay pool based on the sample allocation information; Based on the sample experience data from all target experience replay pools, the sample set for the current training task is obtained.
[0010] In some embodiments, determining at least one target experience replay pool associated with an extended task tag based on the correlation between the extended task tag and the task tags corresponding to each experience replay pool includes: Calculate the cosine similarity between the semantic vector of the extended task label and the semantic vector of the task label corresponding to each experience replay pool to determine the correlation between the extended task label and the task label corresponding to each experience replay pool. The experience replay pool with a cosine similarity greater than a preset threshold is identified as the target experience replay pool. For each target experience replay pool, based on the correlation between the extended task tags and the task tags corresponding to the target experience replay pool, sample allocation information is determined, including: Based on the cosine similarity of each target's experience replay pool, the sample allocation information between each target's experience replay pool is determined; the sample allocation information is positively correlated with the corresponding cosine similarity.
[0011] Furthermore, to achieve the above objectives, this application also provides a robot multi-task control model training device, which includes: The extended task label determination module is used to determine extended task labels that match the current task label of the current training task; wherein, the current training task is one of the various types of preset tasks in the robot multi-task control model; The target experience replay pool determination module is used to determine at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot multi-task control model; each training task has a corresponding experience replay pool. The sample set determination module is used to select sample experience data from the target experience replay pool to obtain the sample set for the current training task. The model training module is used to train the robot's multi-task control model based on the sample set.
[0012] In addition, to achieve the above objectives, this application also provides a robot, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot multi-task control model training method described above.
[0013] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the robot multi-task control model training method described above.
[0014] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the robot multi-task control model training method described above.
[0015] One or more technical solutions proposed in this application have at least the following technical effects: This application can generate extended task labels based on the current task labels, breaking the limitation that one-hot encoding or fixed labels can only represent a single task identity. Extended task labels can carry more task-related information. Furthermore, by using extended task labels, relevant target experience replay pools are determined from all experience replay pools. This overcomes the isolation limitation of extracting sample experience data from only the unique experience replay pool corresponding to the current training task. This allows the robot's multi-task control model to learn more common knowledge among multiple tasks, strengthens the sharing ability between tasks, and solves the problem of poor model training generalization caused by fixed labels. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0017] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0018] Figure 1 This is a schematic diagram of a robot multi-task control model training scheme in related technologies; Figure 2 This is a flowchart illustrating an embodiment of the robot multi-task control model training method of this application. Figure 3 A flowchart illustrating some specific implementation methods of the robot multi-task control model training method of this application; Figure 4 This is a detailed flowchart of step S300 in some specific embodiments of the robot multi-task control model training method of this application. Figure 5 A simplified schematic diagram of the robot multi-task control model training method provided in Embodiment 1 of this application; Figure 6 This is a schematic diagram of the module structure of the robot multi-task control model training device according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware operating environment involved in the robot multi-task control model training method in the embodiments of this application.
[0019] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0022] The main solution of this application embodiment is as follows: determining an extended task label that matches the current task label of the current training task; wherein, the current training task is one of all preset tasks of the robot multi-task control model, and all preset tasks include at least one preset action task and at least two of obstacle avoidance tasks; determining at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot multi-task control model; each training task has a corresponding experience replay pool; selecting sample experience data from the target experience replay pool to obtain the sample set of the current training task; and training the robot multi-task control model based on the sample set.
[0023] In training multi-task control models for robots, one-hot encoding or predefined fixed task labels are typically used to label each training task, while corresponding experience replay pools are configured for different training tasks. Thus, based on one-hot encoding or fixed task labels, sample experience data is selected only from the corresponding experience replay pool to perform model training.
[0024] However, one-hot encoding or predefined fixed task labels can lead to a single information dimension in the training task and a weak ability to share information between multiple tasks. This results in poor generalization of the policies learned by the robot's multi-task control model and poor training performance.
[0025] Therefore, this application provides a solution that can generate extended task labels based on the current task labels, breaking the limitation that one-hot encoding or fixed labels can only represent a single task identity. Extended task labels can carry more task-related information. Furthermore, by using extended task labels, relevant target experience replay pools are determined from all experience replay pools. This overcomes the isolation limitation of extracting sample experience data from only the unique experience replay pool corresponding to the current training task, enabling the robot's multi-task control model to learn more common knowledge among multiple tasks, strengthening the sharing ability between tasks, and solving the problem of poor model training generalization caused by fixed labels.
[0026] To better understand and explain the solutions of the embodiments of this application, some technical terms involved in the embodiments of this application will be briefly explained below.
[0027] Robot multi-task control model: Essentially a multi-task reinforcement learning model (MTL), it is a single machine learning model capable of learning and optimizing multiple tasks simultaneously. Its core idea is to improve the performance, learning efficiency, and generalization ability of each task by allowing the model to process multiple tasks in parallel on shared underlying representations or parameters, leveraging the correlations and shared information between tasks.
[0028] Actor-Critic Networks: A common architecture for multi-task robot control models, combining policy gradient methods and value function estimation. The Actor network learns the optimal action policy, maximizing cumulative reward by continuously adjusting policy parameters. The Critic network evaluates the effectiveness of the current policy, providing the Actor with a value function estimate to help it update policy parameters more efficiently. Figure 1 As shown, Figure 1 This diagram illustrates the training of a robot multi-task control model based on the Actor-Critic network architecture in related technologies. In the training of this model, the agent continuously interacts with the environment to generate quintuples of experience data (s, a, r, s', task); where s is the current state, a is the action performed by the agent, r is the reward, s' is the state after the action, and task is the task label. Different experience replay pools (buffer1, buffer2, ..., bufferern) are set up according to different task types. Experience data can be placed into the corresponding experience replay pool based on the task in the quintuple; then, small batches of random samples are periodically collected from the experience replay pools for training. During training, the Critic network first calculates the Temporal Difference Error (TD) and the advantage value. The Actor network then updates the policy using the policy gradient (such as PPO shearing or entropy regularization) and simultaneously performs a soft update on the target network. This process is repeated until the average reward converges, resulting in the trained target Actor network and target Critic network. Thus, the training of the robot's multi-task control model is completed. Figure 1 In the middle, s t （i） For the current training task （i） The corresponding current state at time t, s t+1 （i） Let a be the state at time t (i.e., the state after the action). t+1 （i） This refers to the action that the agent needs to perform at time t+1.
[0029] Experience replay pool: In reinforcement learning, it is a cache or database used to store experiential data generated by the interaction between the agent and the environment; experiential data is usually stored in the form of tuples such as (s, a, r, s', task).
[0030] Retrieval-Augmented Generation (RAG) is a hybrid artificial intelligence approach that combines information retrieval with text generation. Its core idea is to first retrieve the most relevant information fragments to the current query from an external knowledge source (such as a database or document set) when generating content. Then, this retrieved information is used as context or evidence to enhance and guide the large language model to generate more accurate and reliable information.
[0031] The solutions provided in this application can be applied to application scenarios such as service robots, warehouse robots, or drones.
[0032] To better understand the solution provided in the embodiments of this application, the solution will be described below in conjunction with a specific application scenario.
[0033] Taking warehouse robots as an example, under the control of the warehouse robot multi-task control model, they perform preset action tasks such as picking, grasping, and handling of goods of various specifications and materials, as well as obstacle avoidance and fault recovery tasks.
[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems will be described in detail below through specific embodiments.
[0035] Based on this, embodiments of this application provide a method for training a robot multi-task control model, referring to... Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the robot multi-task control model training method of this application.
[0036] In this embodiment, the robot multi-task control model training method includes steps S100~S400: Step S100: Determine the extended task label that matches the current task label of the current training task.
[0037] The current training task is one of several different types of preset tasks in the robot multi-task control model.
[0038] Step S200: From all the experience replay pools of the robot multi-task control model, determine at least one target experience replay pool associated with the extended task label; each training task has a corresponding experience replay pool.
[0039] Step S300: Select sample experience data from the target experience replay pool to obtain the sample set for the current training task.
[0040] Step S400: Train the robot multi-task control model based on the sample set.
[0041] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer or personal computer, or an electronic device or robot capable of performing the above functions. The following description uses a robot as an example to illustrate this embodiment and the subsequent embodiments.
[0042] Task labels are typically predefined symbols used to uniquely identify and distinguish different training tasks. Training task labels, also known as static task labels, represent the inherent attributes of the training task and do not change with external conditions (such as the task execution environment). However, since task labels are usually defined for task types and cover relatively limited information dimensions, directly using task labels for selecting sample empirical data and training subsequent robot multi-task control models (hereinafter referred to as models) will cause the model to focus only on a single task objective, limiting the model's learning scope.
[0043] Therefore, for the current training task of the model, an extended task label can be determined based on the current task label corresponding to the current training task. The extended task label is an expansion of the task label in terms of information and semantics. Compared with the task label, the extended task label has more dimensions and feature information, and it deepens and supplements the details of the task label.
[0044] The expanded task label can be determined using semantic association information of the current training task. Alternatively, in some specific implementations, step S100 specifically includes: Step S110: Based on the current task tags and the preset task corpus, use the retrieval enhancement generation algorithm to generate extended task tags that match the current task tags.
[0045] The pre-defined task corpus contains corpus information associated with multiple training tasks.
[0046] Taking the training tasks of warehouse robots as an example, the multiple training tasks of warehouse robots include obstacle avoidance, sorting, grasping, and handling. The preset task corpus contains corpus information associated with multiple training tasks; it can be some task-related knowledge text, such as: operating rules: "When grasping an object, the center of the gripper should be aligned with the center of gravity of the object", "When grasping a regular cube, the gripper should be aligned with the geometric center", "When stacking objects in a disorderly manner, the grasping order should be planned to avoid collisions", "When abnormal fluctuations in gripping force are detected, a compliant retreat should be performed and the attempt should be repeated", "When handling heavy objects, the center of gravity should be lowered and the acceleration of movement should be kept stable"; physical common sense: "The static friction coefficient of smooth surfaces (such as plastic and metal) is low", "The moment of inertia of an object increases with the square of the distance of the mass distribution from the axis", "Flexible objects (such as packages) will deform when grasped", "For fragile items, compliant grasping should be used and the end speed should be controlled"; task objective descriptions: "The core evaluation indicators of sorting tasks are success rate and number of operations per hour", "Obstacle avoidance tasks must maximize movement efficiency under the premise of zero collisions", and so on. This corpus information can be converted into semantic vectors by a text encoder and stored.
[0047] When the current task label of the warehouse robot is "sorting", the RAG algorithm can convert sorting into a corresponding query vector and query relevant semantic vectors in the aforementioned preset task corpus to determine the corresponding corpus information. For example, it can query relevant corpus information such as "For fragile items, compliant gripping should be used and the end effector speed should be controlled", "In poor lighting conditions, tactile or depth sensors should be used to assist in positioning", and "When items are stacked haphazardly, the gripping order should be planned to avoid collisions". Based on this corpus information, an extended task label matching the current task label is generated. In one example, the extended task label might be "In a cluttered and poorly lit environment, use a compliant control strategy to sort fragile items and pay attention to planning the gripping order", which contains more potential information than the current task label "sorting".
[0048] In one specific implementation, such as Figure 3 As shown, step S110 can specifically include step S111: based on the environmental state information of the current training task, the current task label, and the preset task corpus, use the retrieval enhancement generation algorithm to generate extended task labels that match the current task label.
[0049] The environmental state information of the current training task refers to the data about the external world and its own state perceived by the warehouse robot at the current moment through its sensors (such as cameras, LiDAR, inertial sensors, etc.). Therefore, in order to improve the accuracy of extended task labels, extended task labels can be generated by combining the environmental information of the current training task. This allows the extended task labels to be more suitable for the agent's current task execution scenario, providing a more accurate direction for the selection of sample experience data for subsequent models.
[0050] For example, in one scenario, the current task label of the warehouse robot is "sorting," and its camera has captured environmental information as "target object: large plastic box with a smooth surface; environmental conditions: low light." In this case, the RAG algorithm can be used to generate an extended task label by combining the aforementioned environmental information and the current task label. The generated extended task label might be in the form of "sorting large plastic boxes with a smooth surface under insufficient light conditions; adopting anti-slip gripping strategies (such as increasing the contact area) and incorporating force feedback to compensate for visual uncertainty." This extended task label is a dynamic description that changes with the environment in real time, rather than an inherent static attribute of the current task, ensuring that subsequent model strategy learning can more accurately adapt to the specific scenario.
[0051] After determining the extended task label corresponding to the current training task, sample experience data can be further selected from the experience replay pool for model training. In related technologies, to reduce interference between training tasks, the experience replay pool corresponding to each training task is usually designed to be isolated; that is, each training task has its own separate task label to distinguish the training tasks. When training the model, each training task only selects sample experience data from its corresponding experience replay pool for training. This hierarchical design of the experience replay pool leads to low utilization of sample experience data in the experience replay pool, resulting in resource waste. Therefore, in this embodiment, after determining the extended task label corresponding to the current training task, at least one target experience replay pool associated with the extended task label can be determined from all experience replay pools of the robot multi-task control model based on the extended task label.
[0052] In some specific implementations, step S200 includes: Step S210: Based on the correlation between the extended task tag and the task tags corresponding to each experience replay pool, determine at least one target experience replay pool associated with the extended task tag.
[0053] Understandably, both the extended task label and the task label can be feature vectors stored by the agent. The correlation between two vectors can be represented by their similarity or distance in vector space, thus determining the task label associated with the extended task label. Since the extended task label contains more information related to the current task, the expanded information can effectively improve the correlation between various training tasks. Therefore, based on the correlation between the extended task label and the task labels corresponding to each experience replay pool, at least one target experience replay pool with a strong correlation to the extended task label of the current training task can be selected from all experience replay pools. This at least one target experience replay pool can include the experience replay pool corresponding to the current training task itself, as well as the experience replay pools corresponding to other training tasks.
[0054] Understandably, the correlation between the extended task tags and the task tags corresponding to each experience replay pool can be measured using methods such as Euclidean distance and Pearson correlation coefficient. Alternatively, in a specific embodiment, step S210 includes: calculating the cosine similarity between the semantic vector of the extended task tag and the semantic vector of the task tag corresponding to each experience replay pool to determine the correlation between the extended task tag and the task tag corresponding to each experience replay pool; and identifying the experience replay pool with a cosine similarity greater than a preset threshold as the target experience replay pool.
[0055] The extended task labels may also be in the form of a natural language description. To facilitate relevance calculation, the semantic vectors corresponding to the extended task labels can be determined. By calculating the cosine similarity between the semantic vectors of the extended task labels and the semantic vectors of the task labels corresponding to each experience replay pool, the relevance between the extended task labels and the task labels in the experience replay pools corresponding to each training task can be determined. The higher the cosine similarity, the stronger the relevance. The relevance representation can refer to the following expression:
[0056] in, For the current training task Extended task tags, For the first j The task tags corresponding to each experience replay pool To expand the correlation between task tags and the corresponding task tags in the experience replay pool, Calculate the cosine similarity of the semantic vectors corresponding to the tags.
[0057] After determining the cosine similarity of each experience replay pool, the experience replay pools with a cosine similarity greater than a preset threshold are designated as target experience replay pools. The sample experience data in these target experience replay pools may be related to the current training task. Selecting sample experience data from these target experience replay pools yields the sample set for the current training task. Thus, the construction of the sample set is not limited to the experience replay pool corresponding to the current training task; sample experience data corresponding to any training task related to the current training task can be included in model training. This achieves cross-task sample selection, avoids missing potentially relevant sample experience data, effectively utilizes sample experience data, and avoids resource waste.
[0058] When selecting sample experience data, a random number of sample experience data can be drawn from these target experience replay pools. The number of sample experience data drawn from each target experience replay pool can be the same or different. Alternatively, in some specific implementations, sample experience data can be drawn based on correlation, such as... Figure 4 As shown, step S300 may include: Step S310: For each target experience replay pool, determine the sample allocation information based on the correlation between the extended task label and the corresponding task label of the target experience replay pool.
[0059] Step S320: For each target experience replay pool, select sample experience data from the target experience replay pool based on the sample allocation information.
[0060] Step S330: Based on the sample experience data of all target experience replay pools, obtain the sample set of the current training task.
[0061] Specifically, the sample allocation information can be the specific number of sample experience data to be extracted from each target experience replay pool, or it can be the extraction ratio of sample experience data. The higher the correlation between the extended task label and the task label of the target experience replay pool, the more sample experience data can be extracted from that target experience replay pool; if the correlation between the task label of the target experience replay pool and the extended task label is low, fewer sample experience data will be extracted from that target experience replay pool.
[0062] In one specific implementation, step S310 may include: determining sample allocation information between each target experience replay pool based on the cosine similarity corresponding to each target experience replay pool; the sample allocation information is positively correlated with the corresponding cosine similarity. That is, the larger the cosine similarity, the more sample experience data is selected from the corresponding experience replay pool; the smaller the cosine similarity, the fewer sample experience data is selected from the corresponding experience replay pool.
[0063] For example, if the total number of empirical data samples in the sample set is 1000, and the target empirical replay pool is ordered by relevance: buffer1 (the empirical replay pool corresponding to the current training task) > buffer3 > buffer2. In one example, sampling can be performed according to random proportions a, b, c (where a > b > c, a + b + c = 1), with buffer1 sampling 1000 × a empirical data samples, buffer2 sampling 1000 × b empirical data samples, and buffer3 sampling 1000 × a empirical data samples. Alternatively, in another example, a fixed proportion x (e.g., 50%, 80%, etc.) of empirical data samples can be selected from the empirical replay pool corresponding to the task label with the highest relevance. For the remaining proportions 1-x of empirical data samples, they can be selected from buffer2 and buffer3 according to random or fixed proportions (y, z); y is the sampling proportion of buffer2, and z is the sampling proportion of buffer3, but this random proportion also satisfies y > z.
[0064] Based on the above sample allocation information, corresponding sample experience data can be selected from the target experience replay pool, and all selected sample experience data can be summarized to obtain the sample set of the current training task. The sample set can be used to train the robot multi-task control model.
[0065] Figure 5 This is a schematic diagram illustrating the overall process of training the robot's multi-task control model in this embodiment. Figure 5 As shown, the environmental state information s for the current training task is determined. （i） and the current task tag （i） Using RAG technology to generate task tags that match the current task. （i） Matching extended task tag 'task' （i） Then, based on task' （i） The correlation between task labels and multiple experience replay pools (buffer1, buffer2, ..., bufferern) is used to determine multiple sample experience data from these pools, thus constructing a sample set. The Actor-Critic architecture robot multi-task control model can be trained using the sample experience data in the sample set. Extended task labels generated by RAG can enhance the correlation between training tasks and improve the model's generalization ability. Furthermore, selecting sample experience data across pools based on the correlation between extended task labels and task labels can effectively improve sample utilization and, to some extent, ensure the generalization ability of the learned strategy. It should be noted that this example is only for understanding this application and does not constitute a limitation on the robot multi-task control model training method of this application. Any simple modifications based on this technical concept are within the scope of protection of this application.
[0066] It is easy to see that the robot multi-task control model training method provided in this application generates extended task labels based on the current task label. These extended task labels can carry more task-related information, breaking the limitation of one-hot encoding or fixed labels in related technologies that can only represent a single task identity. By using extended task labels to determine the relevant target experience replay pool from all experience replay pools, the isolation limitation of extracting sample experience data from only the unique experience replay pool corresponding to the current training task is overcome. This allows the robot multi-task control model to learn more common knowledge among multiple tasks, strengthens the sharing ability between tasks, and solves the problem of poor generalization of model training caused by fixed labels.
[0067] This application also provides a robot multi-task control model training device, please refer to... Figure 6 The robot multi-task control model training device includes: The extended task label determination module 10 is used to determine an extended task label that matches the current task label of the current training task; wherein, the current training task is one of the various types of preset tasks in the robot multi-task control model; The target experience replay pool determination module 20 is used to determine at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot multi-task control model; each training task has a corresponding experience replay pool. The sample set determination module 30 is used to select sample experience data from the target experience replay pool to obtain the sample set for the current training task. The model training module 40 is used to train the robot's multi-task control model based on the sample set.
[0068] The robot multi-task control model training device provided in this application, employing the robot multi-task control model training method in the above embodiments, can solve the technical problem of insufficient generalization ability of robot multi-task control models. Compared with related technologies, the beneficial effects of the robot multi-task control model training device provided in this application are the same as those of the robot multi-task control model training method provided in the above embodiments, and other technical features in the above robot multi-task control model training device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0069] This application provides a robot, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the robot multi-task control model training method in Embodiment 1 above.
[0070] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the embodiments of this application. The robot in the embodiments of this application may include, but is not limited to, mobile terminals such as laptops, PDAs (Personal Digital Assistants), PADs (Portable Application Description), etc., and fixed terminals such as desktop computers. Figure 7 The robot shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of this application.
[0071] like Figure 7 As shown, the robot may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for robot operation, such as a robot multi-task control model. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the robot to communicate wirelessly or wiredly with other devices to exchange data. Although a robot with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.
[0072] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0073] The robot provided in this application, employing the robot multi-task control model training method described in the above embodiments, can solve the technical problem of insufficient generalization ability of the robot multi-task control model. Compared with related technologies, the beneficial effects of the robot multi-task control model training device provided in this application are the same as those of the robot multi-task control model training method provided in the above embodiments, and other technical features in this robot multi-task control model training device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0074] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0075] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0076] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the robot multi-task control model training method in the above embodiments.
[0077] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0078] The aforementioned computer-readable storage medium may be included in the robot; or it may exist independently and not be assembled into the robot.
[0079] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the robot, cause the robot to: determine an extended task label that matches the current task label of the current training task; wherein the current training task is one of several different types of preset tasks in the robot's multi-task control model; determine at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot's multi-task control model; each training task has a corresponding experience replay pool; select sample experience data from the target experience replay pool to obtain a sample set for the current training task; and train the robot's multi-task control model based on the sample set.
[0080] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0082] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0083] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described robot multi-task control model training method, thereby solving the technical problem of insufficient generalization ability of the robot multi-task control model. Compared with related technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the robot multi-task control model training method provided in the above embodiments, and will not be repeated here.
[0084] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the robot multi-task control model training method described above.
[0085] The computer program product provided in this application can solve the technical problem of insufficient generalization ability of robot multi-task control models. Compared with related technologies, the beneficial effects of the computer program product provided in this application are the same as those of the robot multi-task control model training method provided in the above embodiments, and will not be repeated here.
[0086] The above description is only a part of the embodiments of this application and does not limit the scope of protection. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the scope of protection.
Claims
1. A method for training a robot multi-task control model, characterized in that, The robot multi-task control model training method includes: Determine extended task labels that match the current task label of the current training task; wherein, the current training task is one of several different types of preset tasks in the robot multi-task control model; From all the experience replay pools of the robot multi-task control model, at least one target experience replay pool associated with the extended task label is determined; wherein each training task has a corresponding experience replay pool. Sample experience data is selected from the target experience replay pool to obtain the sample set for the current training task; The robot multi-task control model is trained based on the sample set.
2. The robot multi-task control model training method as described in claim 1, characterized in that, The process of determining the extended task label that matches the current task label of the current training task includes: Based on the current task label and the preset task corpus, an extended task label matching the current task label is generated using a retrieval enhancement generation algorithm; the preset task corpus contains corpus information associated with multiple training tasks.
3. The robot multi-task control model training method as described in claim 2, characterized in that, The step of generating extended task tags that match the current task tags based on the current task tags and a preset task corpus using a retrieval-enhanced generation algorithm includes: Based on the environmental state information of the current training task, the current task label, and the preset task corpus, an extended task label matching the current task label is generated using a retrieval-enhanced generation algorithm.
4. The robot multi-task control model training method as described in claim 1, characterized in that, The step of determining at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot multi-task control model includes: Based on the correlation between the extended task tag and the task tag corresponding to each of the experience replay pools, at least one target experience replay pool associated with the extended task tag is determined.
5. The robot multi-task control model training method as described in claim 4, characterized in that, The step of selecting sample experience data from the target experience replay pool to obtain the sample set for the current training task includes: For each of the target experience replay pools, sample allocation information is determined based on the correlation between the extended task tags and the task tags corresponding to the target experience replay pools; For each of the target experience replay pools, sample experience data is selected from the target experience replay pools based on the sample allocation information; Based on the sample experience data of all the target experience replay pools, the sample set of the current training task is obtained.
6. The robot multi-task control model training method as described in claim 5, characterized in that, The step of determining at least one target experience replay pool associated with the extended task tag based on the correlation between the extended task tag and the task tags corresponding to each of the experience replay pools includes: Calculate the cosine similarity between the semantic vector of the extended task tag and the semantic vector of the task tag corresponding to each experience replay pool, so as to determine the correlation between the extended task tag and the task tag corresponding to each experience replay pool; The experience replay pool with a cosine similarity greater than a preset threshold is identified as the target experience replay pool. For each of the target experience replay pools, the sample allocation information is determined based on the correlation between the extended task tags and the task tags corresponding to the target experience replay pools, including: Based on the cosine similarity corresponding to each of the target experience replay pools, the sample allocation information between each of the target experience replay pools is determined; the sample allocation information is positively correlated with the corresponding cosine similarity.
7. A robot multi-task control model training device, characterized in that, The robot multi-task control model training device includes: An extended task label determination module is used to determine an extended task label that matches the current task label of the current training task; wherein, the current training task is one of a variety of preset tasks of different types in the robot multi-task control model; The target experience replay pool determination module is used to determine at least one target experience replay pool associated with the extended task label from all experience replay pools of the robot multi-task control model; each training task has a corresponding experience replay pool. The sample set determination module is used to select sample experience data from the target experience replay pool to obtain the sample set of the current training task; The model training module is used to train the robot multi-task control model based on the sample set.
8. A robot, characterized in that, The robot includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the robot multi-task control model training method as described in any one of claims 1 to 6.
9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the robot multi-task control model training method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the robot multi-task control model training method as described in any one of claims 1 to 6.