A strategy model training method, device and equipment applied to multi-task learning

Abstract

This invention relates to the field of artificial intelligence and discloses a method, apparatus, and device for training a policy model for multi-task learning. The method for training a policy model for multi-task learning includes: acquiring a set of target tasks; training a first policy model on each target task to obtain a second policy model and a first sub-training result; sequentially adjusting the first parameters of the second policy model to obtain a third policy model and a second sub-training result; sequentially adjusting the second parameters of the third policy model to obtain a fourth policy model; and determining the fourth policy model as the policy model for multi-task learning. This reduces the impact of gradient differences between tasks on the policy model during multi-task learning, resulting in more even training progress across tasks and improved training efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence, and in particular to a method, apparatus and device for training a policy model applied to multi-task learning. Background Technology

[0002] With the development of reinforcement learning technology, users are no longer satisfied with the learning ability of policy models for single tasks, and their demand for policy models to learn multiple tasks is also gradually increasing.

[0003] Most existing training methods for policy models facing multiple tasks employ alternating training. Specifically, this involves first inputting at least two different tasks into the policy model, and then adjusting the corresponding parameters of the policy model for each task until the training result of the policy model for that task more closely approximates the expected value.

[0004] However, since the gradients of each task relative to the policy model are different, the policy model may be influenced by different gradients, resulting in biased training for certain tasks. This can lead to incomplete learning of a particular task during the policy model's learning process, thus affecting the final learning result. In the aforementioned alternating training method, the policy model is optimized only based on the training results of each task, without addressing the gradients of the task relative to the policy model. To ensure the completeness and uniformity of task training, the policy model may undergo multiple trial-and-error processes during training. This results in lower actual training efficiency. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a policy model training method, apparatus and device for multi-task learning, so as to solve the problem of low efficiency caused by the different training processes of multiple tasks in the existing policy model learning multi-task learning process.

[0006] In a first aspect, the present invention proposes a method for training a policy model applied to multi-task learning, the method comprising:

[0007] Obtain a target task set, wherein the target task set includes at least two target tasks;

[0008] The first strategy model is invoked to train each target task in the target task set in sequence to obtain the second strategy model and at least two first sub-training results, wherein the at least two first sub-training results correspond one-to-one with the at least two target tasks.

[0009] Based on each first sub-training result, the second policy model is sequentially adjusted with the first parameter to obtain a third policy model and at least two second sub-training results. The at least two first sub-training results correspond one-to-one with the at least two second sub-training results. Each time the second sub-training result obtained after the first parameter adjustment is better than the first sub-training result before the corresponding adjustment.

[0010] Based on the preset gradient constraint set and the training results of each second sub-sub ...

[0011] The fourth strategy model is determined to be a strategy model applied to multi-task learning.

[0012] In some possible implementations, the target task set may be represented as a set of target tasks and / or a set of target task distributions. Thus, the application scenarios for policy models in multi-task learning include situations where the tasks to be learned can be a continuous task distribution or dispersed, independent tasks.

[0013] In some possible implementations, the gradient constraint set is obtained from the weights corresponding to each target task and a preset advantage function. This gradient constraint set is used to characterize the weighted performance improvement lower bound for each target task. Thus, by imposing range constraints on the performance of the policy model for each target task, the adjustable range for each target task during gradient adjustment is determined.

[0014] In some possible implementations, the dominance function is used to characterize the performance change of the third strategy model when the second parameter is adjusted, and the difference between the performance of the fourth strategy model and the performance of the third strategy model is proportional to the dominance function. Thus, the expected dominance function under the nearest future state distribution can be determined based on the above relationship.

[0015] In some possible implementations, the step of sequentially adjusting the second parameter of the third strategy model further includes:

[0016] Obtain the first state distribution of the third strategy model, whereby the first state distribution is used to characterize the model state of the third strategy model after the current second parameter adjustment.

[0017] According to preset rules, the first state distribution is approximated as the model state corresponding to the next adjustment of the second parameter of the third strategy model. This allows for the confirmation of the expected advantage function under the nearest future state distribution, which can then be used to adjust the parameters of the current target task in subsequent iterations or for the remaining target tasks.

[0018] In some possible implementations, the model state includes: parameters corresponding to the third strategy model, environmental interaction results, and empirical data. This multi-type data reference ensures the accuracy of subsequent parameter adjustments.

[0019] In some possible implementations, the weight corresponding to each target task is used to adjust the learning speed of the corresponding target task. This allows the progress of multi-task learning to remain average or meet the user's preset requirements.

[0020] Secondly, the present invention also proposes a policy model training device for multi-task learning, the device comprising:

[0021] The acquisition module is used to acquire a target task set, wherein the target task set includes at least two target tasks;

[0022] The calling module is used to call the first strategy model to train each target task in the target task set in sequence, so as to obtain the second strategy model and at least two first sub-training results, wherein the at least two first sub-training results correspond one-to-one with the at least two target tasks.

[0023] The first parameter adjustment module is used to adjust the first parameter of the second strategy model in sequence according to each first sub-training result to obtain a third strategy model and at least two second sub-training results. The at least two first sub-training results correspond one-to-one with the at least two second sub-training results. The second sub-training result obtained after each first parameter adjustment is better than the first sub-training result before the corresponding adjustment.

[0024] The second parameter adjustment module is used to adjust the second parameters of the third strategy model in sequence according to the preset gradient constraint set and the training result of each second sub-training to obtain the fourth strategy model, wherein the gradient of each target task in the fourth strategy model meets the preset requirements.

[0025] The output module is used to determine that the fourth strategy model is a strategy model applied to multi-task learning.

[0026] Thirdly, the present invention also proposes an electronic device, which includes: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0027] The memory is used to store executable instructions, which, when executed, cause the processor to execute the policy model training method in either the first aspect or the second aspect.

[0028] Fourthly, the present invention also proposes a computer-readable storage medium storing executable instructions, which, when executed, cause a computing device to perform the policy model training method in any possible implementation of the first or second aspect.

[0029] This invention proposes a policy model training method for multi-task learning. In this scheme, firstly, a target task set is obtained, which includes at least two target tasks. Then, a first policy model is sequentially trained on each target task in the target task set to obtain a second policy model and at least two first sub-training results, each corresponding one-to-one with the at least two target tasks. Next, based on each first sub-training result, the second policy model is sequentially adjusted with first parameters to obtain a third policy model and at least two second sub-training results, each corresponding one-to-one with the at least two second sub-training results. Each second sub-training result obtained after adjusting the first parameters is better than the corresponding first sub-training result before adjustment. Then, based on a preset gradient constraint set and each second sub-training result, the third policy model is sequentially adjusted with second parameters to obtain a fourth policy model, where the gradient of each target task in the fourth policy model meets preset requirements. Finally, the fourth policy model is determined to be the policy model applied to multi-task learning. As can be seen, after optimizing and training the policy model for each target task separately, the gradient of each target task relative to the policy model is constrained according to a preset gradient constraint set to mitigate the gradient differences between each target task. In this way, the policy model is less affected by the gradient differences between multiple tasks during multi-task learning, resulting in a more even training progress across tasks and improved training efficiency. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the training method for a strategy model applied to multi-task learning proposed in this invention.

[0031] Figure 2 This is a schematic diagram of the strategy model training device proposed in this invention;

[0032] Figure 3 This is a schematic diagram of the electronic device structure for training the strategy model proposed in this invention. Detailed Implementation

[0033] The terminology used in the following embodiments of the present invention is for the purpose of describing alternative embodiments and is not intended to be limiting of the invention. As used in the specification and appended claims of the present invention, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well. It should also be understood that although the terms first, second, etc., may be used in the following embodiments to describe a class of objects, the objects are not limited to these terms. These terms are used to distinguish specific objects of that class of objects. For example, other classes of objects that may be described using the terms first, second, etc. in the following embodiments are similarly described and will not be repeated here.

[0034] With the development of reinforcement learning technology, users are no longer satisfied with the learning ability of policy models for single tasks, and their demand for policy models to learn multiple tasks is also gradually increasing.

[0035] Most existing training methods for policy models facing multiple tasks employ alternating training. Specifically, this involves first inputting at least two different tasks into the policy model, and then adjusting the corresponding parameters of the policy model for each task until the training result of the policy model for that task more closely approximates the expected value.

[0036] However, since the gradient of each task relative to the policy model is different—understandably, in a neural network, the gradient is a vector indicating the direction in which the function value increases the fastest at a given point—the policy model may be influenced by different gradients, resulting in biased training for certain tasks. This can lead to incomplete learning of a particular task by the policy model, thus affecting the final learning result. In the aforementioned alternating training method, the policy model is optimized only based on the training results of each task, without processing the gradient of the task relative to the policy model. To ensure the completeness and uniformity of task training, the policy model may undergo multiple trial-and-error processes during training. This results in lower actual training efficiency.

[0037] This application specifically relates to machine learning in artificial intelligence. Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formulaic learning. In this application, model training is primarily achieved through machine learning.

[0038] Of course, since the model training method provided in this application can be applied to different fields, it may also involve other specific technologies in artificial intelligence. For example, if applied to fields such as facial expression recognition and facial recognition, it may also involve computer vision. Computer vision (CV) is a science that studies how to make machines "see." More specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, follow, and measure targets, and further performs image processing to make the computer-processed images more suitable for human observation or transmission to instruments for detection. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0039] The policy model training method for multi-task learning provided in this application can be executed by a single electronic device or by a computer cluster. The computer cluster includes at least two electronic devices that support the policy model training method of this application, and any one of these electronic devices can implement the policy model training function described in this application through the policy model training method.

[0040] Any electronic device designed in the embodiments of this application can be an electronic device such as a mobile phone, tablet computer, wearable device (e.g., smartwatch, smart bracelet, etc.), laptop computer, desktop computer, and in-vehicle device. The electronic device is pre-installed with a policy model training application. It is understood that the embodiments of this application do not impose any limitations on the specific type of electronic device.

[0041] The following is a description of several exemplary embodiments, illustrating the technical solutions of the present invention and the technical effects produced by the technical solutions of the present invention.

[0042] Figure 1 This is a schematic diagram of the training process for a policy model applied to multi-task learning proposed in this invention. Figure 1 As shown, the policy model training method for multi-task learning proposed in this invention includes the following steps:

[0043] S100, Obtain a target task set, wherein the target task set includes at least two target tasks;

[0044] S200, the first strategy model is called to train each target task in the target task set in sequence to obtain the second strategy model and at least two first sub-training results, wherein the at least two first sub-training results correspond one-to-one with the at least two target tasks;

[0045] S300, based on each first sub-training result, the second strategy model is sequentially adjusted with the first parameter to obtain a third strategy model and at least two second sub-training results. The at least two first sub-training results correspond one-to-one with the at least two second sub-training results. Each time the second sub-training result obtained after the first parameter adjustment is better than the first sub-training result before the corresponding adjustment.

[0046] S400, based on the preset gradient constraint set and the training results of each second sub-sub ...

[0047] S500, the fourth strategy model is determined to be a strategy model applied to multi-task learning.

[0048] For example, in a given scenario where a policy model is trained for a multi-functional robotic arm, the trained policy model applied to multiple tasks is ultimately applied to the manipulation of the multi-functional robotic arm.

[0049] Suppose the user requires the robotic arm to perform 50 independent tasks, such as opening and closing drawers, turning on a faucet, and moving items to a designated location. These tasks are independent of each other and may not be sequential in practice, but each independent task requires continuous motion control.

[0050] To address this, we first need to define the target task set for the policy model to be trained. Based on the task requirements for the robotic arm described above, we can construct a uniform distribution containing all of the above tasks based on these 50 independent tasks, meaning that each task has an equal probability of being selected, which is 1 / 50 (i.e., completing the construction of the target task set).

[0051] The maximum entropy reinforcement learning algorithm (Soft Actor Critic, SAC) is used here as the reinforcement learning algorithm.

[0052] Understandably, this method is not limited by the distribution of the task and can be applied to common probability distributions in mathematics, such as Gaussian distribution, uniform distribution, Bernoulli distribution, etc.

[0053] Understandably, the choice of reinforcement learning algorithm here is not limited to SAC, but depends on the discrete or continuous nature of the tasks in the constructed target task set. For example, when the target tasks in the constructed target task set are continuous, the algorithms that can be selected include DDPG, TD3, and SAC; when the target tasks in the target task set are discrete, the algorithms that can be selected include DQN, PPO, and A2C.

[0054] Furthermore, after the construction of the target task set is completed, the target task set is obtained first, i.e., S100 is executed.

[0055] Furthermore, after obtaining the target task set, the policy model to be learned (i.e., the first policy model mentioned above) is invoked to learn each target task in the target task set. After learning, an initially learned model (i.e., the second policy model mentioned above) and the learning results for each target task (i.e., the first sub-training results mentioned above) are obtained, such as the learning results for the drawer opening task and the learning results for the drawer closing task. This completes step S200.

[0056] Specifically, assuming that the tasks in the above target task set are labeled as z1, z2, ..., z50 respectively, the target task set can be represented as {z1, z2, ..., z50}. The task randomly selected for training each time can be labeled as z, and the selection process is represented as z ~ {z1, z2, ..., z50}.

[0057] Optionally, for S200, after invoking the current first policy model (represented by π) and interacting with the current task z, the resulting data is recorded in a preset experience collection Bz. The obtained second policy model is used... express.

[0058] It is understandable that the strategy model to be learned here can be a blank model or a general model to be learned that meets the user's preset requirements.

[0059] Furthermore, after obtaining the learning results for each target task, since the difficulty of training the first policy model for each target task varies, the quality of the learning results for each target task (i.e., the degree to which they meet the preset expectations) also varies. Therefore, to make the training results for each target task better meet the preset expectations, it is necessary to optimize the parameters of the second policy model for each target task (parameter optimization here also includes policy adjustment), that is, to execute the process described in S300, perform the first parameter adjustment, and obtain the third policy model and at least two second sub-training results.

[0060] It is understandable that, since the S300 adjustment process involves sequential parameter optimization for each target task, it is possible that after adjusting for a previous target task, the second policy model may perform worse on subsequent target tasks compared to before the adjustment (i.e., with each parameter optimization, the second policy model has a negative impact on the learning progress of other tasks).

[0061] Specifically, based on the above, the optimization method using the SAC algorithm is as follows:

[0062]

[0063]

[0064]

[0065] in, and These are the action-value network and policy network in the SAC algorithm, corresponding to θ and The action-value network Q and policy network in the SAC algorithm The parameter Bz represents the experience replay dataset for task z. This represents the transfer data in the experience recovery set. This represents the mean of a value function. Indicates learning efficiency. Indicates the discount factor. Temperature sparsity is used to control the balance between algorithm exploration and utilization.

[0066] Furthermore, after adjusting the first parameter for each target task, in order to make the subsequent policy model applicable to all target tasks and ensure consistent training efficiency and performance for all target tasks, the S400 process needs to be executed, that is, to adjust the second parameter (including policy optimization and correction) of the third policy model to obtain the fourth policy model.

[0067] It is understandable that, in order to ensure consistent efficiency and performance for training all target tasks, this embodiment adopts the method of imposing corresponding constraints on the third strategy model (i.e., one of the implementation methods of adjusting the second parameter) to mitigate the negative impact of gradient interference between target tasks.

[0068] Specifically, the constraints on the third strategy model are implemented using a preset set of gradient constraints.

[0069] Optionally, the preset gradient constraint set is formed by: obtaining it from the weights corresponding to each target task and a preset advantage function, wherein the gradient constraint set is used to characterize the weighted performance improvement lower limit of each target task.

[0070] Specifically, the weight ω corresponding to each objective task is represented as follows:

[0071]

[0072] in, It is a sparse vector composed of the temperatures corresponding to each target task. It is a hyperparameter that controls the smoothness of the output distribution.

[0073] Optionally, the gradient constraint set is constructed based on the task weights and advantage functions corresponding to each target task, as follows:

[0074]

[0075] in, As task weight, Let A represent the current policy model, and let A represent the advantage function. denoted by , where c represents the discount factor and c represents the hyperparameter used to control the size of the solution set.

[0076] Optionally, the advantage function is used to characterize the performance change of the third strategy model when the second parameter is adjusted, and the difference between the performance of the fourth strategy model and the performance of the third strategy model is proportional to the advantage function.

[0077] Optionally, the step of sequentially adjusting the second parameter of the third strategy model further includes:

[0078] Obtain the first state distribution of the third strategy model, whereby the first state distribution is used to characterize the model state of the third strategy model after the current second parameter adjustment.

[0079] According to preset rules, the first state distribution is approximately calculated as the model state corresponding to the next time the second parameter of the third strategy model is to be adjusted.

[0080] Optionally, the model state includes: parameters corresponding to the third strategy model, environmental interaction results, and empirical data.

[0081] Specifically, the second parameter of the third strategy model is adjusted sequentially to obtain the fourth strategy model, which is expressed as follows:

[0082]

[0083]

[0084] in, Indicates the first Iterative strategy Let t represent the intermediate strategy (i.e., the second strategy model) at step t. This represents the KL divergence.

[0085] Furthermore, for S500, the fourth strategy model is used as a strategy model trained on a multi-objective task and applied to multi-task learning, which is the strategy model that can be applied to the robotic arm.

[0086] Optionally, the entire training process requires several iterations, with each iteration including steps S200 and S300. This allows the model training results to tend towards the preset expectation.

[0087] Optionally, if the number of iterations in the training process is less than the preset number of steps, or if the fourth strategy model cannot meet the preset requirements (such as the training results of the target task not meeting the corresponding preset expectations or the gradients between each target task not meeting the corresponding preset requirements), the process will continue to return to S200 and S300.

[0088] As can be seen, after optimizing and training the policy model for each target task separately, the gradient of each target task relative to the policy model is constrained according to a preset gradient constraint set to mitigate the gradient differences between each target task. In this way, the policy model is less affected by the gradient differences between multiple tasks during multi-task learning, resulting in a more even training progress across tasks and improved training efficiency.

[0089] The above embodiments describe various implementations of the strategy model training method for multi-task learning proposed in this invention from the perspectives of the action logic executed by the electronic device and the learning algorithm processing, including the acquisition method of the target task set, the training of the first strategy model, the adjustment of the first parameter, and the adjustment of the second parameter. It should be understood that, corresponding to the processing steps of acquiring the target task set, adjusting the first parameter, and adjusting the second parameter, the embodiments of this invention can implement the above functions in hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0090] For example, the functionality achieved by the above implementation steps can also be implemented using a policy model training device applied to multi-task learning. Figure 2 This is a schematic diagram of the strategy model training device proposed in this invention. Figure 2As shown, the policy model training device may include:

[0091] The calling module is used to call the first strategy model to train each target task in the target task set in sequence, so as to obtain the second strategy model and at least two first sub-training results, wherein the at least two first sub-training results correspond one-to-one with the at least two target tasks.

[0092] The first parameter adjustment module is used to adjust the first parameter of the second strategy model in sequence according to each first sub-training result to obtain a third strategy model and at least two second sub-training results. The at least two first sub-training results correspond one-to-one with the at least two second sub-training results. The second sub-training result obtained after each first parameter adjustment is better than the first sub-training result before the corresponding adjustment.

[0093] The second parameter adjustment module is used to adjust the second parameters of the third strategy model in sequence according to the preset gradient constraint set and the training result of each second sub-training to obtain the fourth strategy model, wherein the gradient of each target task in the fourth strategy model meets the preset requirements.

[0094] The output module is used to determine that the fourth strategy model is a strategy model applied to multi-task learning.

[0095] It is understandable that the above division of modules / units is merely a logical functional division. In actual implementation, the functions of each module can be integrated into hardware entities. For example, the acquisition module, the calling module, the first parameter adjustment module, and the second parameter adjustment module can be integrated into the processor. The programs and instructions that implement the functions of each module can be maintained in memory. For example, Figure 3 This is a schematic diagram of the electronic device structure for training the strategy model proposed in this invention. (See diagram below.) Figure 3 As shown, the electronic device includes a processor, a transceiver, and a memory. The transceiver is used to acquire information related to the first and second strategy models in the strategy model training method. The memory can store relevant battle information and strategy model information generated during the battle learning process, and can also store code for the processor to execute. When the processor runs the code stored in the memory, the electronic device performs some or all of the operations of the strategy model training method described above.

[0096] For details on the specific implementation process, please refer to the embodiments illustrated in the above method diagram, which will not be described in detail here.

[0097] In a specific implementation, corresponding to the aforementioned electronic device, this embodiment of the invention also provides a computer storage medium. The computer storage medium disposed in the electronic device can store a program, which, when executed, can implement some or all of the steps in various embodiments of the training method for the policy model. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0098] One or more of the above modules or units can be implemented by software, hardware, or a combination of both. When any of the above modules or units is implemented by software, the software exists as computer program instructions and is stored in memory. The processor can be used to execute the program instructions and implement the above method flow. The processor can include, but is not limited to, at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor, etc., and various computing devices that run software. Each computing device may include one or more cores for executing software instructions to perform calculations or processing. The processor can be built into a SoC (System-on-a-Chip) or an application-specific integrated circuit (ASIC), or it can be a separate semiconductor chip. In addition to the cores within the processor for executing software instructions to perform calculations or processing, it may further include necessary hardware accelerators, such as field-programmable gate arrays (FPGAs), PLDs (programmable logic devices), or logic circuits that implement dedicated logic operations.

[0099] When the above modules or units are implemented in hardware, the hardware can be any one or any combination of CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, special purpose digital circuit, hardware accelerator or non-integrated discrete device, which can run the necessary software or perform the above method flow independently of software.

[0100] Furthermore, Figure 3The system may also include a bus interface, which can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors (represented by the processor) and memory (represented by the memory). The bus interface can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. The transceiver provides the unit for communicating with various other devices over the transmission medium. The processor is responsible for managing the bus architecture and general processing, and the memory can store the data used by the processor during operation.

[0101] When the above modules or units are implemented using software, they can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0102] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments.

[0103] The various parts of this specification are described in a progressive manner. Similar or identical parts between the different embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant details can be found in the description of the method embodiments.

[0104] Although alternative embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0105] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a policy model applied to multi-task learning, characterized in that, The method, executed in an electronic device including a processor, memory, and communication interface, is used to train a robotic arm with a multi-task control strategy. The method includes: Obtain a set of robotic arm control tasks, wherein the set of robotic arm control tasks includes at least two independent robotic arm control tasks. The first strategy model is called to train each robotic arm control task in the robotic arm control task set in sequence to obtain a second strategy model and at least two first sub-training results, wherein the at least two first sub-training results correspond one-to-one with the at least two robotic arm control tasks. Based on each first sub-training result, the second policy model is sequentially adjusted with the first parameter to obtain a third policy model and at least two second sub-training results. The at least two first sub-training results correspond one-to-one with the at least two second sub-training results. Each time the second sub-training result obtained after the first parameter adjustment is better than the first sub-training result before the corresponding adjustment. Based on the preset gradient constraint set and the training results of each second sub-training, the second parameters of the third strategy model are adjusted sequentially to obtain the fourth strategy model. The gradient of each robotic arm control task in the fourth strategy model meets the preset requirements. The gradient constraint set is obtained by the weights corresponding to each robotic arm control task and the preset advantage function. The gradient constraint set is used to characterize the lower limit of the weighted performance improvement of each robotic arm control task. The fourth strategy model is determined to be a strategy model applied to multi-task learning.

2. The method as described in claim 1, characterized in that, The robotic arm control task set can be represented in the form of: a set of robotic arm control tasks and / or a set of robotic arm control task distributions.

3. The method as described in claim 1, characterized in that, The advantage function is used to characterize the performance change of the third strategy model when the second parameter is adjusted. The difference between the performance of the fourth strategy model and the performance of the third strategy model is proportional to the advantage function.

4. The method as described in claim 1, characterized in that, The step of sequentially adjusting the second parameter of the third strategy model also includes: Obtain the first state distribution of the third strategy model, whereby the first state distribution is used to characterize the model state of the third strategy model after the current second parameter adjustment. According to preset rules, the first state distribution is approximately calculated as the model state corresponding to the next time the second parameter of the third strategy model is to be adjusted.

5. The method as described in claim 4, characterized in that, The model state includes: the parameters corresponding to the third strategy model, the environmental interaction results, and the empirical data.

6. The method as described in claim 1, characterized in that, The weights corresponding to each robotic arm control task are used to adjust the learning speed of the corresponding robotic arm control task.

7. A policy model training device for multi-task learning, characterized in that, The device includes: An acquisition module is used to acquire a set of robotic arm control tasks, wherein the set of robotic arm control tasks includes at least two independent robotic arm control tasks. The calling module is used to call the first strategy model to train each robotic arm control task in the robotic arm control task set in sequence, so as to obtain the second strategy model and at least two first sub-training results, wherein the at least two first sub-training results correspond one-to-one with the at least two robotic arm control tasks. The first parameter adjustment module is used to adjust the first parameter of the second strategy model in sequence according to each first sub-training result to obtain a third strategy model and at least two second sub-training results. The at least two first sub-training results correspond one-to-one with the at least two second sub-training results. The second sub-training result obtained after each first parameter adjustment is better than the first sub-training result before the corresponding adjustment. The second parameter adjustment module is used to adjust the second parameters of the third strategy model in sequence according to the preset gradient constraint set and the training results of each second sub-project, so as to obtain the fourth strategy model. The gradient of each robotic arm control task in the fourth strategy model meets the preset requirements. The gradient constraint set is obtained by the weights corresponding to each robotic arm control task and the preset advantage function. The gradient constraint set is used to characterize the lower limit of the weighted performance improvement of each robotic arm control task. The output module is used to determine that the fourth strategy model is a strategy model applied to multi-task learning.

8. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store executable instructions, which, when executed, cause the processor to perform the policy model training method for multi-task learning as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The storage medium stores executable instructions, which, when executed, cause the computing device to perform the policy model training method for multi-task learning as described in any one of claims 1-6.

Images