Intelligent agent decision-making method and device based on continuous strategy combination and electronic equipment

Abstract

The invention provides an intelligent agent decision-making method and device based on continuous strategy combination and electronic equipment. The method comprises the following steps: acquiring a current state of an intelligent agent; based on a pre-trained weight distribution network, determining weight values corresponding to a plurality of sub-strategies according to the current state of the intelligent agent; according to the multiple sub-strategies and the weight values corresponding to the sub-strategies, a behavior strategy of the intelligent agent for the target task is obtained through combination; wherein the weight distribution network is obtained by training and optimizing a target task training sample set based on a pre-training strategy network, the pre-training strategy network comprises a plurality of sub-strategy networks with isolated parameters, and the sub-strategy networks comprise a target task strategy network and a basic task strategy network. The plurality of sub-policies are correspondingly generated by the plurality of sub-policy networks. According to the method, an existing single strategy is fully utilized, according to the current state of the intelligent body, a strategy which is stronger and more adaptive to the environment is flexibly combined, the coping capacity of the intelligent body for dynamic changes of different environments is enhanced, partial interpretability is given to the system, and meanwhile, the system can be used for improving the dynamic change of the intelligent body. Only a small number of training samples of the target task need to be collected in the process, and the data collection and labeling cost is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of robot learning technology, and in particular to an intelligent agent decision-making method, device and electronic equipment based on continuous strategy combination. Background Art

[0002] Current approaches to robot learning rely on large datasets to train general robotic policies. These datasets come from different sources, such as simulated environments, real robot experiments, and videos of human demonstrations.

[0003] However, there are huge costs and technical challenges in collecting and compiling heterogeneous data (such as color, depth, tactile, and proprioceptive information) for different tasks in the real world. The complexity brought by this heterogeneous data makes it difficult for a single strategy to cope with the diversity of all tasks and domains.

[0004] Traditional training methods usually concentrate all training data of the same type, such as training a policy using only color and depth data in a simulated environment. However, this limits the generalization ability of the policy and cannot effectively handle the differences in data patterns between different fields and tasks, resulting in poor performance of the policy in some areas.

[0005] Therefore, how to solve the problem that existing general robot strategy learning methods require huge costs and have poor strategy learning effects is an important issue that needs to be urgently solved in the field of robot learning. Summary of the invention

[0006] The present invention provides an intelligent agent decision-making method, device and electronic device based on continuous strategy combination, which are used to overcome the defects of existing general robot strategy learning methods that require huge costs and have poor strategy learning effects. Only a small number of training samples of target tasks are needed to achieve more robust performance in different scenarios and tasks.

[0007] On the one hand, the present invention provides an agent decision-making method based on continuous strategy combination, including: obtaining the current state of the agent; determining the weight values ​​corresponding to multiple sub-strategies according to the current state of the agent based on a pre-trained weight allocation network; combining the multiple sub-strategies and their corresponding weight values ​​to obtain the agent's behavior strategy for a target task; wherein the weight allocation network is obtained by training and optimizing a pre-trained strategy network based on a target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with isolated parameters, the sub-strategy networks include a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated correspondingly by the multiple sub-strategy networks.

[0008] Furthermore, the method of obtaining the current state of the intelligent agent includes: determining a target task and multiple basic tasks to be used; wherein the target task is a new task that needs to be learned currently, and is split into multiple basic tasks; constructing a target task training sample set corresponding to the target task, and a basic task training sample set corresponding to multiple basic tasks; according to the target task training sample set and the multiple basic task training sample sets, continuously learning a single strategy through a parameter-isolated low-rank network to obtain a pre-trained strategy network; wherein the single strategy includes a sub-strategy corresponding to the target task and a sub-strategy corresponding to multiple basic tasks.

[0009] Furthermore, the low-rank network with parameter isolation continuously learns a single strategy to obtain a pre-trained strategy network, including: independently maintaining a set of weight matrices for each sub-strategy network when continuously learning multiple single strategies, and using orthogonality regularization to orthogonalize the weight matrices corresponding to multiple sub-strategy networks to ensure that the low-rank network maintains the ability to learn tasks while adapting to new tasks.

[0010] Furthermore, the training and optimization of the weight allocation network includes: obtaining a target task training sample set, each training sample in the target task training sample set includes a state sample and an action sample of the intelligent agent; based on multiple sub-strategy networks, obtaining correspondingly generated multiple decision actions according to the state samples; based on the weight allocation network, obtaining weight allocation values ​​corresponding to multiple decision actions according to the state samples; linearly combining multiple decision actions and their corresponding weight allocation values, using the difference between the control amount obtained by the linear combination and the action sample as the training loss, and iteratively optimizing the weight allocation network.

[0011] Furthermore, the pre-trained weight allocation network determines the weight values ​​corresponding to multiple sub-strategies according to the current state of the agent, including: encoding the current state of the agent to obtain a low-dimensional representation of the current state; inputting the low-dimensional representation of the current state into the pre-trained weight allocation network to obtain the weight values ​​corresponding to multiple sub-strategies.

[0012] Furthermore, it also includes: based on a pre-trained weight allocation network, obtaining a weight combination value sequence according to the actual motion state of the intelligent agent; determining the number of steps of abnormal state of the intelligent agent according to the actual motion state of the intelligent agent and its corresponding weight combination value sequence, so as to realize the verification of the intelligent agent decision-making method.

[0013] In the second aspect, the present invention also provides an intelligent agent decision-making device based on continuous strategy combination, including: an intelligent agent state acquisition module, used to obtain the current state of the intelligent agent; a strategy weight allocation module, used to determine the weight values ​​corresponding to multiple sub-strategies according to the current state of the intelligent agent based on a pre-trained weight allocation network; a strategy weight combination module, used to combine the intelligent agent's behavior strategy for the target task according to multiple sub-strategies and their corresponding weight values; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, the sub-strategy network includes a target task policy network and a basic task policy network, and multiple sub-strategies are generated accordingly by multiple sub-strategy networks.

[0014] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the intelligent agent decision-making method based on continuous strategy combination as described in any one of the above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described intelligent agent decision-making methods based on continuous strategy combination.

[0016] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described intelligent agent decision-making methods based on continuous strategy combination.

[0017] The agent decision method based on continuous strategy combination provided by the present invention obtains the current state of the agent, and based on the pre-trained weight allocation network, determines the weight values ​​corresponding to multiple sub-strategies according to the current state of the agent, and then combines the agent's behavior strategy for the target task according to the multiple sub-strategies and their corresponding weight values; wherein the weight allocation network is obtained by training and optimizing based on the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, the sub-strategy network includes a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated by multiple sub-strategy networks. This method makes full use of the existing single strategy, and flexibly combines stronger and more environmentally adaptable strategies according to the current state of the agent itself, which not only enhances the agent's ability to cope with dynamic changes in different environments, but also gives the system partial interpretability. At the same time, this process only needs to collect a small amount of training samples of the target task, effectively reducing the cost of data collection and annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of an intelligent agent decision-making method based on continuous strategy combination provided in an embodiment of the present invention.

[0020] Figure 2 It is a training optimization diagram of a weight distribution network provided in an embodiment of the present invention.

[0021] Figure 3 It is a complete flow chart of the intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention.

[0022] Figure 4 It is a verification diagram of the intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention.

[0023] Figure 5 It is a schematic diagram of the effects of the intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention and the existing decision-making method.

[0024] Figure 6 It is a structural diagram of an intelligent agent decision-making device based on continuous strategy combination provided by an embodiment of the present invention.

[0025] Figure 7 It is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that training a general robotics policy based on heterogeneous data from different tasks is a major challenge. Current approaches usually collect and aggregate all the data in a domain and train a single policy to handle this heterogeneity in tasks and domains, but this approach is costly and difficult.

[0028] In this regard, the prior art proposes a combination strategy method, but its combination weights are often static and require a lot of manual parameter adjustment, which cannot adapt to dynamic and complex scenarios.

[0029] In view of this, the present invention proposes an agent decision-making method based on continuous strategy combination, specifically, Figure 1 A flow chart of an intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the method includes steps S110-S130, and steps S110-S130 and related steps will be described in detail below.

[0031] S110, obtaining the current state of the agent.

[0032] It is easy to understand that the current state of the agent refers to all relevant information of the agent at the current moment, including but not limited to environmental perception information, internal state information, and historical information.

[0033] Environmental perception information refers to the current environmental information obtained by the agent through various sensors or other perception methods. For example, in a robot, environmental perception information may be images obtained by a visual camera or distance measurements obtained by a lidar.

[0034] Internal state information refers to the state information of the agent itself, such as the agent's current position coordinates and orientation, the agent's energy state (battery charge or fuel level), the state of the task the agent is performing, and the state of the agent's available resources (computing resources, storage space, etc.).

[0035] Historical information refers to the agent’s past experiences, including the tasks it has completed, the situations it has encountered, and the knowledge it has learned from these experiences.

[0036] The current state of the agent can be directly measured by sensors or inferred through the agent's behavior patterns or other indirect information, which will not be elaborated here.

[0037] It should be noted that the current state of the agent will be different for different application scenarios. For example, in the scenario of an autonomous vehicle, the current state of the agent includes the speed of the vehicle, the distance of surrounding objects, and the state of traffic signals; in the application of a virtual assistant, the current state of the agent includes the user's preferences and the context of the current conversation.

[0038] Based on obtaining the current state of the agent in step S110, further, step S120 is executed.

[0039] S120, based on a pre-trained weight allocation network, determine the weight values ​​corresponding to multiple sub-strategies according to the current state of the intelligent body; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, the sub-strategy network includes a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated correspondingly by the multiple sub-strategy networks.

[0040] In this step, the weight assignment network is a pre-trained neural network model that can receive the current state of the agent as input and output a set of weight values, which are used to adjust the importance of the sub-strategies adopted by the agent in different tasks and scenarios. In other words, the role of the weight assignment network is to determine how much weight each sub-strategy should be assigned based on the current state of the agent, thereby guiding the agent to take the optimal action / strategy.

[0041] Specifically, after determining the target task, the target task is first split into multiple basic tasks. These multiple basic tasks are currently available single strategies, and their training sample data is usually large. For example, in a specific embodiment, the target task is a task of an intelligent agent doing somersaults, and the multiple basic tasks obtained by splitting the target task are standing, walking, squatting, jumping, and flipping.

[0042] Then, use the training sample sets corresponding to each basic task to continuously learn each basic task (train the corresponding basic task policy network), and collect a small number of training samples under the target task, and use this small number of samples to continuously learn the target task (train the target task policy network), so that a pre-trained policy network can be obtained.

[0043] The training sample sets of the target task and each basic task at least include states and corresponding actions. The input of the basic task strategy network and the target task strategy network is the state, and the output is the sub-strategy.

[0044] Next, based on a small number of collected training samples of the target task, multiple decision actions output by the pre-trained strategy network and weight allocation values ​​corresponding to multiple decision actions output by the weight allocation network can be obtained. Thus, the difference between the value obtained by linearly combining multiple decision actions and their corresponding weight allocation values ​​and the actions in a small number of training samples is used as the training loss, and iterative optimization is performed to obtain a trained weight allocation network.

[0045] Finally, in actual application, the current state of the intelligent agent is obtained by inputting it into the trained weight allocation network to obtain a set of output weight values. This set of weight values ​​corresponds to multiple sub-strategy networks and also corresponds to the sub-strategies generated by multiple sub-strategy networks.

[0046] In a specific embodiment, after obtaining the current state of the agent, the current state of the agent is encoded / compressed to obtain a low-dimensional representation of the current state, which is more convenient for subsequent processing. Then, the low-dimensional representation of the current state is input into a pre-trained weight allocation network to obtain weight values ​​corresponding to multiple sub-strategies.

[0047] It is worth mentioning that this embodiment allows the agent to adapt to new tasks or environments by adjusting the weights of different sub-strategies without retraining the entire network. In addition, since the sub-strategy network is parameter-isolated, it means that even if a new sub-strategy is added, it will not affect the existing sub-strategy network, which can effectively improve the flexibility and scalability of the agent's decision-making.

[0048] It should also be noted that the pre-trained strategy network in this embodiment can use a low-rank network (Lora) with fewer parameters to continuously learn and update a single strategy, which can not only reduce the computational cost but also improve the efficiency of strategy learning.

[0049] After determining the weight values ​​corresponding to the plurality of sub-strategies according to the current state of the agent based on the pre-trained weight allocation network in step S120, step S130 is executed.

[0050] S130, combining the multiple sub-strategies and their corresponding weight values ​​to obtain a behavior strategy of the agent for the target task.

[0051] It is easy to understand that after obtaining multiple sub-strategies and its corresponding weight value After that, the sub-strategies and their corresponding weight values ​​are linearly combined to obtain the weighted combination value That is, the agent’s behavioral strategy for the target task.

[0052] Among them, the behavior strategy includes the probability distribution of the action / action that the agent should take in the current state.

[0053] In this embodiment, by obtaining the current state of the agent, and based on the pre-trained weight allocation network, the weight values ​​corresponding to multiple sub-strategies are determined according to the current state of the agent, and then the behavior strategy of the agent for the target task is combined according to the multiple sub-strategies and their corresponding weight values; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, the sub-strategy network includes the target task strategy network and the basic task strategy network, and multiple sub-strategies are generated by the multiple sub-strategy networks. This method makes full use of the existing single strategy, and flexibly combines stronger and more environmentally adaptable strategies according to the current state of the agent itself, which not only enhances the ability of the agent to cope with dynamic changes in different environments, but also gives the system partial interpretability. At the same time, this process only requires the collection of a small number of training samples of the target task, effectively reducing the cost of data collection and annotation.

[0054] On the basis of the above embodiments, the training process of the pre-training strategy network will be described in detail below.

[0055] Obtaining the current state of the intelligent agent, which includes: determining a target task and multiple basic tasks to be used; wherein the target task is a new task that needs to be learned currently, and is split into multiple basic tasks; constructing a target task training sample set corresponding to the target task, and a basic task training sample set corresponding to each of the multiple basic tasks; according to the target task training sample set and the multiple basic task training sample sets, continuously learning a single strategy through a parameter-isolated low-rank network to obtain a pre-trained strategy network; wherein the single strategy includes a sub-strategy corresponding to the target task and a sub-strategy corresponding to each of the multiple basic tasks.

[0056] It is understandable that at the very beginning, it is necessary to determine the target task that the agent currently needs to perform / learn, and the available data for the target task in the real world is usually scarce. On this basis, the target task can be split to obtain multiple corresponding basic tasks, and the available data for the basic tasks in the real world is usually abundant.

[0057] After determining the target task and its corresponding basic task, a small number of target task training sample sets corresponding to the target task and basic task training sample sets corresponding to multiple basic tasks are collected. The training samples in the target task training sample set and the basic task training sample set both include the agent state and its corresponding action. The target task training sample set is used to learn the target task (train the target task policy network), and the basic task training sample set is used to learn the basic task (train the basic task policy network).

[0058] After constructing the training sample set, a low-rank network with fewer parameters is used to continuously learn and update a single strategy (the sub-strategies corresponding to multiple basic tasks and the sub-strategies corresponding to the target task), so that the final pre-trained strategy network can simultaneously have the ability to learn the target task and multiple basic tasks.

[0059] When continuously learning any single strategy, a set of weight matrices is independently maintained for each sub-strategy network, and the weight matrices corresponding to multiple sub-strategy networks are orthogonalized using orthogonality regularization to ensure that the low-rank network maintains the ability to learn learned tasks (such as basic tasks) while adapting to new tasks (such as target tasks), avoiding the catastrophic forgetting problem of continuous learning.

[0060] Specifically, in the process of continuous learning, the agent needs to continuously learn new tasks or strategies, and each sub-policy network is responsible for handling a specific single task or sub-task. In order to ensure that each sub-policy network can focus on its specific function without interfering with each other, each sub-policy network maintains its own independent set of weight matrices. This means that each sub-policy network is an independent model with its own set of parameters, which are only used for the training and reasoning of the sub-policy network.

[0061] When the weight matrices of multiple sub-policy networks are orthogonal, it means that they are independent of each other to some extent, reducing the mutual influence between different sub-policy networks. Doing so helps prevent the negative impact on previously learned knowledge when learning new tasks, that is, avoiding the problem of catastrophic forgetting.

[0062] To make the weight matrices of multiple sub-policy networks orthogonal, a regularization term can be added to the loss function of multiple sub-policy networks to encourage the weight matrices to maintain orthogonality. The loss function with the regularization term added can be seen in the following formula (1).

[0063] (1).

[0064] in, represents the low-rank network i layer, represents the number of layers contained in each low-rank network, and represents the low-rank weight matrix, i and j Used to distinguish different low-rank weight matrices.

[0065] Low-rank networks (Lora) are used to fine-tune large-scale pre-trained models to adapt to new tasks or fields while reducing the impact on the performance of the original tasks. In the Lora framework, the weight parameters do not directly include the pre-trained weights corresponding to the target task and the basic task. Instead, the pre-trained model is updated by introducing additional low-rank matrices, which are multiplied or added to the original weight matrix to achieve fine-tuning of the model. In this way, not only can the target task be learned, but the knowledge of the original basic task can also be retained at the same time.

[0066] It should be noted that, in this embodiment, multiple sub-strategy networks (including a target task strategy network and multiple basic task strategy networks) coexist in the pre-trained strategy network through parameter isolation. Each sub-strategy network will not affect the learning process of other sub-strategy networks during learning. Even if a new sub-strategy network is added, it will not affect the existing sub-strategy network, thereby effectively improving the flexibility and scalability of the intelligent agent's decision-making.

[0067] In this embodiment, by determining the target task and multiple basic tasks to be used, wherein the target task is a new task that needs to be learned at present, and multiple basic tasks are obtained by splitting; and constructing a target task training sample set corresponding to the target task, and a basic task training sample set corresponding to multiple basic tasks, and then continuously learning a single strategy through a parameter-isolated low-rank network according to the target task training sample set and the multiple basic task training sample sets, a pre-trained strategy network is obtained; wherein the single strategy includes a sub-strategy corresponding to the target task and a sub-strategy corresponding to multiple basic tasks. This method not only reduces the computational cost but also improves the efficiency of strategy learning by continuously learning and updating a single strategy using a low-rank network with fewer parameters.

[0068] On the basis of the above embodiments, the training optimization process of the weight allocation network will be described in detail below.

[0069] The training and optimization weight allocation network includes: obtaining a target task training sample set, each training sample in the target task training sample set includes a state sample and an action sample of an intelligent agent; based on multiple sub-strategy networks, obtaining correspondingly generated multiple decision actions according to the state samples; based on the weight allocation network, obtaining weight allocation values ​​corresponding to multiple decision actions according to the state samples; linearly combining multiple decision actions and their corresponding weight allocation values, using the difference between the control amount obtained by the linear combination and the action sample as the training loss, and iteratively optimizing the weight allocation network.

[0070] It can be understood that the target task training sample set is first obtained, each of which includes a state sample and an action sample of the agent. The state sample is used to describe the state information of the agent at a certain moment, and the action sample is used to describe the action actually performed by the agent in a certain state.

[0071] The target task training sample set may be obtained through the behavior records of intelligent agents in a simulation environment or an actual deployment environment, or may be directly obtained by calling a public database, and is not specifically limited here.

[0072] Then, the state samples in the target task training sample set are used as the input of multiple sub-strategy networks to obtain multiple decision actions as output. At the same time, the state samples in the target task training sample set are used as the input of the weight distribution network to obtain the weight distribution values ​​corresponding to the multiple decision actions as output, that is, the weight distribution values ​​corresponding to the multiple sub-strategy networks.

[0073] Next, the decision actions output by multiple sub-strategy networks are linearly combined with their corresponding weight assignment values. Specifically, each decision action is multiplied by its corresponding weight assignment value, and then all weighted decision actions are added together to obtain the final control amount.

[0074] In order to optimize the weight distribution network, a loss function can be defined to measure the difference between the control amount obtained by the linear combination and the actual action sample executed later, and then the parameters in the weight distribution network are updated through the back propagation algorithm to minimize the aforementioned loss function. The loss function can be seen in the following formula (2).

[0075] (2).

[0076] In formula (2), Represents the overall network model, represents the pre-trained model, Indicates the number of LoRA, Indicates i LoRA, represents the weight distribution network, Indicates the status, represents the trainable network parameters, represents a low-rank network model, , represents a low-rank weight matrix.

[0077] The above process is iterated repeatedly until the loss function converges to a smaller value, indicating that the weight allocation network has learned how to assign appropriate weights to decision actions generated by different sub-strategy networks according to the state of the agent.

[0078] Figure 2 A schematic diagram of training optimization of a weight distribution network provided by an embodiment of the present invention is shown.

[0079] exist Figure 2 In the above example, State-aware module represents the weight distribution network, States represents the state sample, A and B represent two sub-strategy networks, B1-Bn and A1-An represent multiple different sub-strategies, Represents the pre-trained weights (the weight assignment values ​​of the weight assignment network prediction output).

[0080] like Figure 2 As shown, Figure 2 In the lower right corner, orthogonality regularization is used to orthogonalize the weight matrices corresponding to the sub-strategy networks A and B to ensure that the low-rank network can adapt to new tasks while maintaining the ability to learn the tasks. For the pre-trained policy network, after it is trained, its weights are frozen (such as Figure 2 for practical applications.

[0081] When training the weight distribution network, "States" is used as the model input. is the model output, They correspond to B1-Bn and A1-An respectively, and can be linearly combined. Then, the difference between the control amount obtained by the linear combination and the actual action sample is used as the training loss, and the weight allocation network is iteratively optimized to obtain a trained weight allocation network.

[0082] In this embodiment, by obtaining a target task training sample set, each training sample in the target task training sample set includes a state sample and an action sample of the agent; and based on multiple sub-strategy networks, according to the state samples, multiple decision actions corresponding to the generated are obtained, and then based on the weight allocation network, according to the state samples, the weight allocation values ​​corresponding to the multiple decision actions are obtained, so as to linearly combine multiple decision actions and their corresponding weight allocation values, and use the difference between the control amount obtained by the linear combination and the action sample as the training loss, iteratively optimize the weight allocation network, and obtain a trained weight allocation network. This method optimizes the weight allocation network by training, so that the trained weight allocation network can dynamically utilize the advantages of each single strategy according to the current state of the agent, and combine a stronger behavior strategy to complete the target task.

[0083] In some embodiments, Figure 3 A complete flow chart of the intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention is shown.

[0084] exist Figure 3In the above example, State-aware module represents the weight distribution network, States represents the current state of the agent, B1-Bn and A1-An represent multiple different sub-strategies. Represents the pre-trained weights (the weight values ​​corresponding to multiple sub-strategies predicted and output by the weight distribution network).

[0085] like Figure 3 As shown in Figure 2, the weight distribution network uses the current state of the agent as the model input and outputs the weight values ​​corresponding to multiple sub-strategies. Then, multiple sub-strategies and their corresponding weight values ​​are linearly combined to obtain the final behavior strategy Actions.

[0086] In some other embodiments, the effect of the intelligent agent decision-making method based on continuous strategy combination provided by the embodiments of the present invention is verified.

[0087] Based on the pre-trained weight allocation network, a weight combination value sequence is obtained according to the actual motion state of the intelligent agent; according to the actual motion state of the intelligent agent and its corresponding weight combination value sequence, the number of steps of abnormal state of the intelligent agent is determined to realize the verification of the decision-making method of the intelligent agent.

[0088] Among them, the actual motion state of the agent includes the dynamic state changes of the agent as it performs operations over time, which can be understood as a dynamic motion video of the agent.

[0089] It is easy to understand that by inputting the actual motion state of the agent at multiple consecutive moments into the pre-trained weight distribution network, the corresponding output weight combination value sequence can be obtained. At the same time, according to the actual motion state of the agent at these multiple consecutive moments, the moment when the agent needs to make operational adjustments can be known, that is, the number of steps when the agent's state is abnormal. The number of steps when the agent's state is abnormal, for example, the moment when the agent falls or is about to fall.

[0090] Figure 4 A verification schematic diagram of the intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention is shown.

[0091] like Figure 4 As shown, by comparing the weight combination value sequence output by the weight allocation network and the actual motion state of the agent at multiple consecutive moments, it can be found that for the steps with abnormal state of the agent, the weight combination values ​​in the corresponding weight combination value sequence also fluctuate greatly. This shows that according to the agent decision-making method based on continuous strategy combination provided by the embodiment of the present invention, it is necessary to perform strategy combination to adjust the operation performed by the agent at the step with abnormal state of the agent, and this is exactly what the embodiment of the present invention does.

[0092] in addition, Figure 5A schematic diagram showing the effects of the intelligent agent decision-making method based on continuous strategy combination provided by an embodiment of the present invention and the existing decision-making method.

[0093] like Figure 5 As shown, the "adaptive combination of stand strategy and run strategy" provided by the present invention is compared with the "only run strategy" and "combination of stand strategy and run strategy according to a fixed ratio" provided by the prior art. Figure 5 It can be seen that the adaptive combination method provided by the present invention performs best, followed by the fixed ratio combination method, and finally the method of running only a single strategy.

[0094] according to Figure 5 It can also be seen that directly combining strategies in a fixed ratio may destroy the performance of a single strategy, while adaptively combining strategies can achieve better results, making the agent's actions smoother and more natural while ensuring performance.

[0095] Corresponding to the agent decision-making method based on continuous strategy combination described in the above embodiments, the present invention also provides an agent decision-making device based on continuous strategy combination. Specifically, Figure 6 A schematic diagram of the structure of an intelligent agent decision-making device based on continuous strategy combination provided by an embodiment of the present invention is shown.

[0096] like Figure 6 As shown, the device includes: an agent state acquisition module 610, used to obtain the current state of the agent; a strategy weight allocation module 620, used to determine the weight values ​​corresponding to multiple sub-strategies according to the current state of the agent based on a pre-trained weight allocation network; a strategy weight combination module 630, used to combine the agent's behavior strategy for the target task based on multiple sub-strategies and their corresponding weight values; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, the sub-strategy network includes a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated correspondingly by multiple sub-strategy networks.

[0097] In this embodiment, the current state of the agent is obtained by the agent state acquisition module 610, and the strategy weight allocation module 620 determines the weight values ​​corresponding to multiple sub-strategies based on the pre-trained weight allocation network according to the current state of the agent, and then the strategy weight combination module 630 combines the multiple sub-strategies and their corresponding weight values ​​to obtain the agent's behavior strategy for the target task; wherein, the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, and the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, and the sub-strategy network includes a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated by multiple sub-strategy networks. The device makes full use of the existing single strategy and flexibly combines stronger and more environmentally adaptable strategies according to the current state of the agent itself, which not only enhances the ability of the agent to cope with dynamic changes in different environments, but also gives the system partial interpretability. At the same time, this process only requires the collection of a small number of training samples of the target task, effectively reducing the cost of data collection and annotation.

[0098] It should be noted that the intelligent agent decision-making device based on continuous strategy combination provided in the embodiment of the present invention and the intelligent agent decision-making method based on continuous strategy combination described in the above embodiments can be referred to each other, and will not be described in detail here.

[0099] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the agent decision method based on the continuous strategy combination, the method comprising: obtaining the current state of the agent; determining the weight values ​​corresponding to the plurality of sub-strategies according to the current state of the agent based on the pre-trained weight allocation network; combining the plurality of sub-strategies and their corresponding weight values ​​to obtain the agent's behavior strategy for the target task; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network comprises a plurality of sub-strategy networks with parameter isolation, the sub-strategy network comprises a target task strategy network and a basic task strategy network, and the plurality of sub-strategies are correspondingly generated by the plurality of sub-strategy networks.

[0100] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0101] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the intelligent agent decision-making method based on continuous strategy combination provided by the above-mentioned methods, and the method includes: obtaining the current state of the intelligent agent; based on a pre-trained weight allocation network, determining the weight values ​​corresponding to multiple sub-strategies according to the current state of the intelligent agent; combining the multiple sub-strategies and their corresponding weight values ​​to obtain the intelligent agent's behavior strategy for the target task; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to the target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with parameter isolation, the sub-strategy network includes a target task policy network and a basic task policy network, and multiple sub-strategies are generated accordingly by multiple sub-strategy networks.

[0102] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the intelligent agent decision-making method based on continuous strategy combination provided by the above-mentioned methods, the method comprising: obtaining the current state of the intelligent agent; determining the weight values ​​corresponding to multiple sub-strategies according to the current state of the intelligent agent based on a pre-trained weight allocation network; combining the multiple sub-strategies and their corresponding weight values ​​to obtain the intelligent agent's behavior strategy for a target task; wherein the weight allocation network is obtained by training and optimizing the pre-trained strategy network according to a target task training sample set, the pre-trained strategy network comprises multiple sub-strategy networks with parameter isolation, the sub-strategy network comprises a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated correspondingly by multiple sub-strategy networks.

[0103] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An agent decision-making method based on continuous strategy combination, characterized in that: include: Get the current state of the agent; Based on the pre-trained weight distribution network, the weight values ​​corresponding to multiple sub-strategies are determined according to the current state of the agent; Based on multiple sub-strategies and their corresponding weight values, the agent’s behavior strategy for the target task is obtained by combining them; Among them, the weight allocation network is obtained by training and optimizing based on a pre-trained strategy network according to a target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with isolated parameters, the sub-strategy networks include a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated correspondingly by multiple sub-strategy networks.

2. The agent decision-making method based on continuous strategy combination according to claim 1, characterized in that: The obtaining of the current state of the agent previously includes: Determine a target task and multiple basic tasks to be used; wherein the target task is a new task that needs to be learned at present, and is split into multiple basic tasks; Constructing a target task training sample set corresponding to the target task and basic task training sample sets corresponding to a plurality of basic tasks respectively; Based on the target task training sample set and multiple basic task training sample sets, a single strategy is continuously learned through a parameter-isolated low-rank network to obtain a pre-trained strategy network; The single strategy includes a sub-strategy corresponding to the target task and sub-strategies corresponding to multiple basic tasks.

3. The agent decision-making method based on continuous strategy combination according to claim 2, characterized in that: The single strategy is continuously learned through the parameter-isolated low-rank network to obtain a pre-trained strategy network, including: When continuously learning multiple single strategies, a set of weight matrices is independently maintained for each sub-strategy network, and the weight matrices corresponding to multiple sub-strategy networks are orthogonalized by orthogonality regularization to ensure that the low-rank network maintains its ability to learn tasks while adapting to new tasks.

4. The agent decision-making method based on continuous strategy combination according to claim 1, characterized in that: Training and optimizing the weight distribution network, including: Acquire a target task training sample set, wherein each training sample in the target task training sample set includes a state sample and an action sample of the agent; Based on multiple sub-strategy networks, multiple decision actions are generated correspondingly according to state samples; Based on the weight distribution network, the weight distribution values ​​corresponding to multiple decision actions are obtained according to the state samples; Linearly combine multiple decision actions and their corresponding weight assignment values, use the difference between the control amount obtained by linear combination and the action sample as the training loss, and iteratively optimize the weight assignment network.

5. The agent decision-making method based on continuous strategy combination according to claim 1, characterized in that: The pre-trained weight distribution network determines the weight values ​​corresponding to the multiple sub-strategies according to the current state of the agent, including: Encode the current state of the agent to obtain a low-dimensional representation of the current state; The low-dimensional representation of the current state is input into a pre-trained weight allocation network to obtain weight values ​​corresponding to multiple sub-strategies.

6. The agent decision-making method based on continuous strategy combination according to any one of claims 1 to 5, characterized in that: Also includes: Based on the pre-trained weight distribution network, a weight combination value sequence is obtained according to the actual motion state of the agent; According to the actual motion state of the agent and its corresponding weight combination value sequence, the number of steps of abnormal state of the agent is determined to verify the decision-making method of the agent.

7. An intelligent agent decision-making device based on continuous strategy combination, characterized in that: include: The agent state acquisition module is used to obtain the current state of the agent; The strategy weight allocation module is used to determine the weight values ​​corresponding to multiple sub-strategies based on the pre-trained weight allocation network and the current state of the agent; The strategy weight combination module is used to combine multiple sub-strategies and their corresponding weight values ​​to obtain the agent's behavior strategy for the target task; Among them, the weight allocation network is obtained by training and optimizing based on a pre-trained strategy network according to a target task training sample set, the pre-trained strategy network includes multiple sub-strategy networks with isolated parameters, the sub-strategy networks include a target task strategy network and a basic task strategy network, and multiple sub-strategies are generated correspondingly by multiple sub-strategy networks.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the intelligent agent decision-making method based on continuous strategy combination as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the intelligent agent decision-making method based on continuous strategy combination as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the intelligent agent decision-making method based on continuous strategy combination as described in any one of claims 1 to 6 is implemented.

Images