Intelligent agent skill extraction, training and decision making method and system, electronic equipment, storage medium and program product

Abstract

The invention provides a skill extraction, training and decision making method and system of an intelligent agent, electronic equipment, a storage medium and a program product. The skill extraction method comprises the steps that a training sample is obtained, the training sample comprises all segments of sub-tracks obtained by segmenting a complete track according to H steps of the preset length, and the complete track is composed of the state of all the steps completing a task and the actually-adopted action; inputting the state change of each section of sub-track in the training sample into a skill encoder to obtain a skill encoding vector of each section of sub-track; for each skill coding vector, determining a skill characterization vector most similar to the skill coding vector from M skill characterization vectors in a skill characterization vector library; calculating the skill extraction loss based on the difference between each skill coding vector and the most similar skill representation vector; and on the basis of the skill extraction loss, updating parameters of a skill encoder and skill characterization vectors in a skill characterization vector library.

Description

Technical Field

[0001] The present disclosure generally relates to the field of artificial intelligence technology, and more specifically, to a method, system, electronic device, storage medium, and program product for skill extraction, training, and decision-making of an intelligent agent. Background Art

[0002] Currently, multi-task reinforcement learning is a key research area in the field of artificial intelligence. Traditional reinforcement learning methods rely on real-time interaction with the environment to learn, but this poses challenges such as high data acquisition costs and high environmental interaction risks. Consequently, offline reinforcement learning has emerged. Offline reinforcement learning utilizes pre-collected static datasets to learn policies, avoiding direct interaction with the environment.

[0003] Existing offline multi-task reinforcement learning methods typically use methods based on network parameter sharing, such as sharing neural network structures, task conditional representations, or specially designed optimization processes to achieve multi-task learning. Although these methods improve the generalization ability of policies to a certain extent, their knowledge sharing is usually at a lower level, that is, action-level imitation learning. Summary of the Invention

[0004] An exemplary embodiment of the present disclosure is to provide a method, system, electronic device, storage medium and program product for skill extraction, training and decision-making of an intelligent agent, which can solve at least one of the above-mentioned problems existing in the prior art.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for offline skill extraction of an intelligent agent for performing multiple tasks is provided, the method comprising: obtaining training samples, wherein the training samples comprise: sub-trajectories obtained by segmenting a complete trajectory according to a preset length H steps, the complete trajectory consisting of the states of each step for completing a task and the actual actions taken; inputting the state changes of each sub-trajectory in the training sample into a skill encoder to obtain a skill coding vector for each sub-trajectory, wherein the state change of each sub-trajectory is the change of the end state of the sub-trajectory relative to the starting state of the sub-trajectory; for each skill coding vector, determining the skill representation vector that is most similar to the skill coding vector from among M skill representation vectors in a skill representation vector library; calculating the skill extraction loss based on the difference between each skill coding vector and the skill representation vector that is most similar to it; and updating the parameters of the skill encoder and the skill representation vectors in the skill representation vector library based on the skill extraction loss.

[0006] According to a second aspect of an embodiment of the present disclosure, there is provided an offline training method for an intelligent agent for performing multiple tasks, wherein the intelligent agent comprises: a skill representation vector library, a skill strategy network for deciding the skills to be taken, and an action strategy network for deciding the actions to be taken according to the skills taken, wherein the offline training method comprises: obtaining a trained skill encoder and a skill representation vector library by executing the skill offline extraction method as described above; obtaining training samples; inputting the state changes of each sub-trajectory in the training samples into the trained skill encoder to obtain the skill encoding vector of each sub-trajectory; determining the target skill representation vector corresponding to each sub-trajectory, and encoding the target skill representation vector of the target skill encoder; obtaining the target skill representation vector of the target skill encoder; obtaining the target skill representation vector of the target skill encoder; obtaining the target skill representation vector of the target skill encoder; and obtaining the target skill representation vector of the target skill encoder. The index of the target skill representation vector is used as the skill label of the sub-trajectory, wherein the target skill representation vector is the skill representation vector that is most similar to the skill encoding vector of the sub-trajectory in the trained skill representation vector library; for each sub-trajectory, the states and target skill representation vectors corresponding to the sub-trajectory before the sub-trajectory, as well as the initial state of the sub-trajectory are input into the skill strategy network to obtain the predicted index of the skill adopted by the sub-trajectory; based on the skill index prediction results and skill labels corresponding to each sub-trajectory, the loss of the skill strategy network is calculated; based on the loss of the skill strategy network, the parameters of the skill strategy network are updated.

[0007] Optionally, it also includes: for each sub-trajectory in the training sample, based on the sub-trajectory and its corresponding target skill representation vector, predicting the action taken at each step in the sub-trajectory through the action strategy network; based on the action prediction results corresponding to each step in each sub-trajectory and the actual action taken, calculating the loss of the action strategy network; based on the loss of the action strategy network, updating the parameters of the action strategy network.

[0008] Optionally, for each sub-trajectory in the training sample, based on the sub-trajectory and its corresponding target skill representation vector, the action taken at each step in the sub-trajectory is predicted by the action strategy network, including: for each step in each sub-trajectory, the target skill representation vector corresponding to the sub-trajectory, the state of the step and before the step in the sub-trajectory, and the action actually taken before the step in the sub-trajectory are input into the action strategy network to obtain the action prediction result of the step.

[0009] According to a third aspect of an embodiment of the present disclosure, a skill guidance decision-making method for an intelligent agent performing multiple tasks is provided, the skill guidance decision-making method comprising: entering the current skill window after completing H steps in the previous skill window, inputting the skill representation vectors of the current state and the historical state, and the skills taken in the historical skill window into a trained skill strategy network, and deciding the index of the skill taken in the current skill window; obtaining a skill representation vector that matches the index of the skill to be taken from a trained skill representation vector library; based on the obtained skill representation vector, the state after entering the current skill window, and the action taken, deciding in sequence the actions to be taken in the H steps in the current skill window through a trained action strategy network; wherein, the intelligent agent is trained by executing the offline training method as described above.

[0010] According to a fourth aspect of an embodiment of the present disclosure, there is provided an offline skill extraction device for an intelligent agent performing multiple tasks, the offline skill extraction device comprising: a sample acquisition unit configured to acquire training samples, wherein the training samples comprise: sub-trajectories obtained by segmenting a complete trajectory according to a preset length H steps, wherein the complete trajectory consists of the states and actual actions taken for completing each step of a task; a skill encoding unit configured to input the state changes of each sub-trajectory in the training sample into a skill encoder to obtain a skill encoding vector for each sub-trajectory, wherein the state change of each sub-trajectory is the change of the end state of the sub-trajectory relative to the starting state of the sub-trajectory; a representation determination unit configured to determine, for each skill encoding vector, the skill representation vector that is most similar to the skill encoding vector from among M skill representation vectors in a skill representation vector library; a skill extraction loss calculation unit configured to calculate the skill extraction loss based on the difference between each skill encoding vector and the skill representation vector that is most similar to it; and a skill extraction update unit configured to update the parameters of the skill encoder and the skill representation vectors in the skill representation vector library based on the skill extraction loss.

[0011] According to a fifth aspect of the embodiments of the present disclosure, there is provided an offline training system for an intelligent agent for performing multiple tasks, the intelligent agent comprising: a skill representation vector library, a skill policy network for deciding which skills to adopt, and an action policy network for deciding which actions to adopt based on the adopted skills. The offline training system comprises: the above-described offline skill extraction device, the skill policy network training device, and the action policy network training device;

[0012] Wherein, the skill strategy network training device includes: a sample acquisition unit, configured to acquire training samples; a skill encoding unit, configured to input the state changes of each sub-trajectory in the training sample into the trained skill encoder to obtain the skill encoding vector of each sub-trajectory; a skill label determination unit, configured to determine the target skill representation vector corresponding to each sub-trajectory, and use the index of the target skill representation vector as the skill label of the sub-trajectory, wherein the target skill representation vector is the skill representation vector most similar to the skill encoding vector of the sub-trajectory in the trained skill representation vector library; a skill prediction unit, configured to input the state and target skill representation vector corresponding to each sub-trajectory before the sub-trajectory, as well as the initial state of the sub-trajectory into the skill strategy network for each sub-trajectory, to obtain the predicted index of the skill taken by the sub-trajectory; a skill loss calculation unit, configured to calculate the loss of the skill strategy network based on the skill index prediction results and skill labels corresponding to each sub-trajectory; a skill strategy network updating unit, configured to update the parameters of the skill strategy network based on the loss of the skill strategy network;

[0013] Among them, the action strategy network training device includes: an action prediction unit, which is configured to predict the actions taken at each step in each sub-trajectory in the training sample based on the sub-trajectory and its corresponding target skill representation vector through the action strategy network; an action loss calculation unit, which is configured to calculate the loss of the action strategy network based on the action prediction results corresponding to each step in each sub-trajectory and the actual actions taken; and an action strategy network updating unit, which is configured to update the parameters of the action strategy network based on the loss of the action strategy network.

[0014] According to a sixth aspect of an embodiment of the present disclosure, a skill guidance decision system for an intelligent agent performing multiple tasks is provided, wherein the skill guidance decision system comprises: a skill decision unit, configured to enter a current skill window after completing H steps in a previous skill window, input the skill representation vectors of the current state and the historical state, and the skills taken in the historical skill window into a trained skill strategy network, and decide the index of the skill taken in the current skill window; a skill representation vector acquisition unit, configured to acquire a skill representation vector that matches the index of the skill to be decided from a trained skill representation vector library; an action decision unit, configured to decide the actions to be taken in the H steps in the current skill window in sequence through a trained action strategy network based on the acquired skill representation vector, the state after entering the current skill window, and the action taken; wherein the intelligent agent is trained by executing the offline training method as described above.

[0015] According to the seventh aspect of an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, which, when executed by a processor of an electronic device, enables the electronic device to execute the offline skill extraction method and / or offline training method and / or skill guidance decision method as described above.

[0016] According to an eighth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the offline skill extraction method and / or offline training method and / or skill guidance decision method as described above.

[0017] According to the ninth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer-executable instructions, which, when executed by at least one processor, implement the offline skill extraction method and / or offline training method and / or skill guidance decision method as described above.

[0018] According to the method, system, electronic device, storage medium and program product for skill extraction, training and decision-making of intelligent agents of the exemplary embodiments of the present disclosure, reusable skills can be effectively abstracted from offline data sets and these skills can be used to complete different tasks; by realizing cross-task abstraction and reuse of skills, the generalization ability of strategies is significantly improved; and by adopting a hierarchical strategy learning method, flexible combination and efficient application of skills are achieved.

[0019] In the following description, some aspects and / or advantages of the general inventive concept of the present disclosure will be set forth, and some aspects and / or advantages will be known through the following description or implementation of the general inventive concept of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] These and / or other aspects and advantages of the present application will become more clear and easier to understand from the following detailed description of the embodiments of the present application in conjunction with the accompanying drawings, in which:

[0021] Figure 1 A flowchart illustrating a method for offline skill extraction of an intelligent agent for performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0022] Figure 2 An example of an offline skill extraction method for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 3 A flowchart illustrating an offline training method for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 4 An example of an offline training method of a skill policy network according to an exemplary embodiment of the present disclosure is shown;

[0025] Figure 5 A flowchart illustrating an offline training method of an action policy network according to an exemplary embodiment of the present disclosure;

[0026] Figure 6 An example of an offline training method of an action policy network according to an exemplary embodiment of the present disclosure is shown;

[0027] Figure 7 A flowchart illustrating a skill-guided decision-making method for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0028] Figure 8 A structural block diagram of an offline skill extraction apparatus for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0029] Figure 9 A structural block diagram of an offline training system for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0030] Figure 10 A structural block diagram of a skill guidance decision system for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown;

[0031] Figure 11 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0032] Reference will now be made in detail to the embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like parts throughout. The embodiments are described below with reference to the drawings in order to explain the present disclosure.

[0033] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0034] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0035] Figure 1 A flowchart of a method for offline skill extraction of an agent for performing multiple tasks according to an exemplary embodiment of the present disclosure is shown.

[0036] The agent can perform a variety of tasks, each of which can be accomplished by using a set of skills. The skills used in different tasks may be completely different or partially the same, and each skill is achieved by performing a series of actions. As an example only, the agent can be used to control a robotic arm to perform a variety of tasks.

[0037] The intelligent agent includes at least: a skill representation vector library having M skill representation vectors, a skill strategy network for deciding the skills to be adopted, and an action strategy network for deciding the actions to be taken according to the adopted skills.

[0038] As an example, since a skill causes the state to change from the current state to the desired target state, a skill can be defined as the state change from the starting state to the ending state of a trajectory segment of a preset length (represented by H steps).

[0039] As an example, the skill representation vector library (i.e., the skill codebook) can be expressed as One of the skill representation vectors Used to represent a skill, different skill representation vectors are used to represent different skills, and the same skill can be reused for different tasks. By executing this offline skill extraction method, a constructed skill representation vector library (i.e., a trained skill representation vector library) can be obtained.

[0040] The skill policy network, as a high-level policy, selects skills from a skill representation vector library (i.e., a discrete skill space) to solve specific tasks. The action policy network, as a low-level policy, interacts with the environment based on the selected skills.

[0041] As an example, the network architecture type of the skill strategy network can be Transformer, RNN, LSTM, etc., and the network architecture type of the action strategy network can be Transformer, RNN, LSTM, etc. It should be understood that the present disclosure does not limit the architecture type and specific structure of the skill strategy network and the action strategy network.

[0042] Reference Figure 1 , in step S101, a training sample is obtained.

[0043] The training samples include: sub-trajectories obtained by segmenting a complete trajectory τ according to a preset length H steps. The complete trajectory consists of the state s of each step (i.e., each time point t) to complete a task and the actual action a.

[0044] Since completing a task requires the use of at least one skill, and it is assumed that one skill is completed every H steps, a complete trajectory (s0, a0, s1, a1, ...) can be segmented into H steps to obtain sub-trajectories corresponding to each skill. That is, a sub-trajectory is a segment of the trajectory when a skill is used. For example, the first sub-trajectory obtained by segmentation is: (s0, a0, ..., s H-1 ,a H-1 ), and so on, no longer elaborate.

[0045] As an exemplary embodiment, an offline data set may be pre-built, and then training samples may be obtained from the offline data set. As an example, multiple training samples may be obtained from the offline data set each time.

[0046] Regarding the construction of offline datasets, the trajectory of successfully completing a task is taken as a training sample, and multiple training samples involving different types of tasks are added to the offline dataset.

[0047] In step S102, the state change of each sub-trajectory in the training sample is input into a skill encoder to obtain a skill encoding vector of each sub-trajectory.

[0048] The state change of each sub-track segment is the change of the end state of the sub-track segment relative to the start state of the sub-track segment.

[0049] As an exemplary embodiment, the skill encoder G converts the sub-track (s t ,a t ,…,s t+H-1 ,a t+H-1 ) is mapped to a potential embedding space Z, and the skill encoding vector z of the sub-trajectory is obtained. t,H =G(s t+H-1 -s t ), where s t+H-1Indicates the end state of the sub-trajectory, s t Indicates the starting state of the sub-track.

[0050] According to an exemplary embodiment of the present disclosure, the skill encoder is task-independent, i.e., it does not rely on task-specific reward information and only captures trajectory-level state dynamics.

[0051] In step S103 , for each skill encoding vector, a skill representation vector that is most similar to the skill encoding vector is determined from among the M skill representation vectors in the skill representation vector library.

[0052] In step S104 , a skill extraction loss is calculated based on the difference between each skill encoding vector and the skill representation vector most similar to the skill encoding vector.

[0053] In step S105 , based on the skill extraction loss, the parameters of the skill encoder and the skill representation vectors in the skill representation vector library are updated.

[0054] As an exemplary embodiment, refer to Figure 2 In order to construct a finite and discrete skill representation vector library, vector quantization (VQ) technology can be used. Given the skill encoding vector z of a sub-trajectory t,H , can be discretized by the VQ module:

[0055]

[0056] in, Representing skill representation vector library The discrete skill representation vector in , Represents the skill representation vector library and the skill encoding vector z t,H The most similar skill representation vector.

[0057] As an exemplary embodiment, the training loss of the VQ module (i.e., skill extraction loss) is:

[0058]

[0059] Here, sg[·] means stopping gradient propagation, and α is used to balance the contribution between the two terms.

[0060] It should be understood that after executing step S105, you can return to execute step S101 to obtain other training samples, that is, loop through steps S101 to S105 until the update of the skill encoder parameters and skill representation vector library is completed when certain conditions are met, thereby obtaining the trained skill encoder parameters and skill representation vector library (that is, the constructed skill representation vector library).

[0061] In addition, as an exemplary embodiment, the skill representation vector library can be adaptively expanded according to the new skill requirements of new tasks.

[0062] The exemplary embodiments of this disclosure propose a goal-oriented skill abstraction method (i.e., a goal-oriented ensemble extraction method) to efficiently extract a set of skills from an offline, mixed-task dataset for reuse across different tasks. Furthermore, a skill discretization method based on vector quantization is proposed to facilitate the construction of a discrete skill library and improve skill generalization performance.

[0063] Figure 3 A flowchart of an offline training method for an agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown.

[0064] Reference Figure 3 In step S301, by executing Figure 1 The offline skill extraction method shown in the figure obtains a trained skill encoder and a skill representation vector library.

[0065] In step S302, a training sample is obtained.

[0066] In step S303, the state changes of each sub-trajectory in the training sample are input into the trained skill encoder to obtain the skill encoding vector of each sub-trajectory.

[0067] In step S304, the target skill representation vector corresponding to each sub-trajectory is determined, and the index of the target skill representation vector is used as the skill label of the sub-trajectory.

[0068] The target skill representation vector corresponding to each sub-trajectory is: the skill representation vector in the trained skill representation vector library that is most similar to the skill encoding vector of the sub-trajectory, and the index of the skill representation vector in the skill representation vector library is the index of the target skill representation vector.

[0069] In step S305, for each sub-trajectory, the states and target skill representation vectors corresponding to the sub-trajectories preceding the sub-trajectory, as well as the initial state of the sub-trajectory, are input into the skill strategy network to obtain the predicted index of the skill adopted by the sub-trajectory.

[0070] As an exemplary embodiment, refer to Figure 4 , with sub-trajectory (s t ,a t ,…,s t+H-1 ,a t+H-1 ) as an example, the states (s ＜t ), the target skill representation vector corresponding to each sub-trajectory before this sub-trajectory And the initial state of the sub-trajectory (s t ), input skill strategy network Get the index of the skill that is predicted to be taken by this sub-trajectory

[0071] In step S306, the loss of the skill strategy network is calculated based on the skill index prediction results and skill labels corresponding to each sub-trajectory.

[0072] As an exemplary embodiment, step S306 may include: for each sub-trajectory, based on the probability distribution of the index of the skill predicted to be taken by the sub-trajectory, determining the probability value of the skill label predicted for the sub-trajectory, and calculating the loss of the skill strategy network based on the probability value.

[0073] In step S307, the parameters of the skill strategy network are updated based on the loss of the skill strategy network.

[0074] It should be understood that after executing step S307, you can return to execute step S302 to obtain other training samples, that is, loop through steps S302 to S307 until the update of the parameters of the skill strategy network is completed when certain conditions are met, thereby obtaining a trained skill strategy network.

[0075] In addition, the offline training method for an agent performing multiple tasks according to an exemplary embodiment of the present disclosure may further include: Figure 5 The training method of the action policy network shown.

[0076] Figure 5 A flowchart illustrating an offline training method of an action policy network according to an exemplary embodiment of the present disclosure is shown.

[0077] Reference Figure 5 , in step S501, a training sample is obtained.

[0078] In step S502, for each sub-trajectory in the training sample, based on the sub-trajectory and its corresponding target skill representation vector, the action taken at each step in the sub-trajectory is predicted through the action strategy network.

[0079] As an exemplary embodiment, step S502 may include: for each step in each sub-trajectory, inputting the target skill representation vector corresponding to the sub-trajectory, the state of the step and before the step in the sub-trajectory, and the action actually taken before the step in the sub-trajectory into the action strategy network to obtain the action prediction result of the step.

[0080] As an exemplary embodiment, refer to Figure 6 , with sub-trajectory (s t ,a t ,…,st+H-1 ,a t+H-1 ) as an example, for any step in the sub-trajectory (that is, at any time t ′ ), the target skill representation vector corresponding to the sub-trajectory The state of this step and the state before this step in this sub-trace (s ≤t′ =s t ,s t+1 ,…,s t′ ), the actual action taken before the step in the sub-trace (a <t′ =a t ,a t+1 ,…,a t′-1 ), input the action strategy network P, and get the action prediction result of this step

[0081]

[0082] In step S503 , the loss of the action strategy network is calculated based on the action prediction results corresponding to each step in each sub-trajectory and the actual action taken.

[0083] As an exemplary embodiment, the mean square error loss can be used as the loss L of the action policy network MSE :

[0084]

[0085] Among them, a t′ Indicates t ′ The actual action taken.

[0086] In step S504 , the parameters of the action policy network are updated based on the loss of the action policy network.

[0087] It should be understood that after executing step S504, you can return to execute step S501 to obtain other training samples, that is, loop through steps S501 to S504 until the update of the parameters of the action strategy network is completed when certain conditions are met, thereby obtaining a trained action strategy network.

[0088] This paper proposes an offline multi-task reinforcement learning algorithm based on skill abstraction, which can solve the following problems existing in existing technologies: (1) Insufficient abstraction: Current methods are mostly action-level, lacking effective recognition and reuse of abstract skills, which limits the efficiency of knowledge transfer; (2) Poor generalization ability: Existing methods have difficulty effectively utilizing suboptimal trajectory data and cannot extract transferable skills from imperfect data. This paper improves the effectiveness of offline multi-task reinforcement learning through a more efficient skill abstraction and generalization method.

[0089] Figure 7A flowchart of a skill-guided decision-making method for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown.

[0090] The agent is executed by Figure 3 and Figure 5 The training is performed using the offline training method shown in FIG.

[0091] Reference Figure 7 In step S701, after completing H steps in the previous skill window, the current skill window is entered, and the skill representation vectors of the current state, historical state, and skills taken in the historical skill window are input into the trained skill strategy network to decide the index of the skill taken in the current skill window.

[0092] In step S702 , a skill representation vector that matches the index of the skill adopted in the decision (ie, the skill representation vector of the skill adopted in the current skill window) is obtained from a trained skill representation vector library.

[0093] In step S703, based on the acquired skill representation vector, the state after entering the current skill window and the action taken, the trained action strategy network is used to sequentially decide the actions to be taken in the H steps within the current skill window.

[0094] As an exemplary embodiment, to complete the H steps (a t ,…,a t+H-1 ) and then enter the current skill window. In step S701, the current state and the historical state (s ≤t+H ), the skill representation vector of the skill taken in the historical skill window Input the trained skill strategy network to determine the index of the skill to be taken in the current skill window In step S702, the index of the skill to be taken in the decision is obtained from the trained skill representation vector library. Matching skill representation vector In step S703, the action to be taken in the first step (i.e., step t+H) within the current skill window is first decided. Specifically, the skill representation vector The state and actions taken after entering the current skill window (s t+H ) Input the action strategy network and get the action a taken in this step t+H , after executing action a t+H After that, the action to be taken in the second step (i.e., step t+H+1) within the current skill window is decided. Specifically, the skill representation vector The state and actions taken after entering the current skill window (s t+H 、s t+H+1 、a t+H) Input the action strategy network and get the action a taken in this step t+H+1 , after executing action a t+H+1 After that, the action to be taken in the third step (i.e., step t+H+2) in the current skill window is decided, and so on, until the H steps in the current skill window are executed and the next skill window is entered.

[0095] According to an exemplary embodiment of the present disclosure, during deployment, every H steps, the high-level skill strategy selects a skill index (i.e., taking H steps as the skill decision unit), and the low-level action strategy predicts the specific actions of each step and the interaction with the environment within the skill window.

[0096] According to the exemplary embodiments of the present disclosure, cross-task abstraction and reuse of skills are effectively achieved, significantly improving the generalization ability of the strategy; through goal-oriented skill representation, suboptimal trajectory data can be fully utilized, improving data utilization efficiency; hierarchical strategy learning significantly shortens the decision sequence, improving the efficiency and stability of strategy learning.

[0097] Figure 8 A structural block diagram of an offline skill extraction apparatus for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown.

[0098] Reference Figure 8 According to an exemplary embodiment of the present disclosure, the skill offline extraction device 800 for an intelligent agent performing multiple tasks includes: a sample acquisition unit 801, a skill encoding unit 802, a characterization determination unit 803, a skill extraction loss calculation unit 804, and a skill extraction update unit 805.

[0099] Specifically, the sample acquisition unit 801 is configured to acquire training samples, wherein the training samples include: sub-trajectories obtained by segmenting a complete trajectory according to a preset length H steps, and the complete trajectory is composed of the states of each step to complete a task and the actual actions taken.

[0100] The skill encoding unit 802 is configured to input the state changes of each sub-trajectory in the training sample into the skill encoder to obtain a skill encoding vector for each sub-trajectory, wherein the state change of each sub-trajectory is the change of the end state of the sub-trajectory relative to the starting state of the sub-trajectory.

[0101] The representation determination unit 803 is configured to determine, for each skill encoding vector, a skill representation vector that is most similar to the skill encoding vector from among the M skill representation vectors in the skill representation vector library.

[0102] The skill extraction loss calculation unit 804 is configured to calculate the skill extraction loss based on the difference between each skill encoding vector and the skill representation vector most similar to it.

[0103] The skill extraction and updating unit 805 is configured to update the parameters of the skill encoder and the skill representation vectors in the skill representation vector library based on the skill extraction loss.

[0104] Figure 9 A block diagram of an offline training system for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown. The intelligent agent includes: a skill representation vector library, a skill policy network for deciding which skills to adopt, and an action policy network for deciding which actions to adopt based on the adopted skills.

[0105] Reference Figure 9 According to an exemplary embodiment of the present disclosure, an offline training system 900 for an intelligent agent performing multiple tasks includes: a skill offline extraction device 800, a skill strategy network training device 901, and an action strategy network training device 902.

[0106] Specifically, the skill strategy network training device 901 includes: a sample acquisition unit 9011, a skill encoding unit 9012, a skill label determination unit 9013, a skill prediction unit 9014, a skill loss calculation unit 9015, and a skill strategy network update unit 9016.

[0107] The sample acquisition unit 9011 is configured to acquire training samples.

[0108] The skill encoding unit 9012 is configured to input the state changes of each sub-trajectory in the training sample into the trained skill encoder to obtain the skill encoding vector of each sub-trajectory.

[0109] The skill label determination unit 9013 is configured to determine the target skill representation vector corresponding to each sub-trajectory, and use the index of the target skill representation vector as the skill label of the sub-trajectory, wherein the target skill representation vector is the skill representation vector in the trained skill representation vector library that is most similar to the skill encoding vector of the sub-trajectory.

[0110] The skill prediction unit 9014 is configured to input the states and target skill representation vectors corresponding to the sub-trajectories before the sub-trajectory, as well as the initial state of the sub-trajectory, into the skill strategy network for each sub-trajectory, and obtain the index of the predicted skill adopted by the sub-trajectory.

[0111] The skill loss calculation unit 9015 is configured to calculate the loss of the skill strategy network based on the skill index prediction results and skill labels corresponding to each sub-trajectory.

[0112] The skill strategy network updating unit 9016 is configured to update the parameters of the skill strategy network based on the loss of the skill strategy network.

[0113] The action strategy network training device includes: an action prediction unit 9021, an action loss calculation unit 9022, and an action strategy network updating unit 9023.

[0114] The action prediction unit 9021 is configured to predict, for each sub-trajectory in the training sample, the action taken at each step in the sub-trajectory based on the sub-trajectory and its corresponding target skill representation vector through the action strategy network.

[0115] The action loss calculation unit 9022 is configured to calculate the loss of the action strategy network based on the action prediction results corresponding to each step in each sub-trajectory and the actual action taken.

[0116] The action policy network updating unit 9023 is configured to update the parameters of the action policy network based on the loss of the action policy network.

[0117] Figure 10 A structural block diagram of a skill guidance decision system for an intelligent agent performing multiple tasks according to an exemplary embodiment of the present disclosure is shown. Figure 3 and Figure 5 The training is performed using the offline training method shown in FIG.

[0118] Reference Figure 10 According to an exemplary embodiment of the present disclosure, a skill guidance decision system 1000 for an intelligent agent performing multiple tasks includes: a skill decision unit 1001, a skill representation vector acquisition unit 1002, and an action decision unit 1003.

[0119] Specifically, the skill decision unit 1001 is configured to enter the current skill window after completing H steps in the previous skill window, input the skill representation vectors of the current state and historical state, and the skills taken in the historical skill window into the trained skill strategy network, and decide the index of the skill taken in the current skill window.

[0120] The skill representation vector acquisition unit 1002 is configured to acquire a skill representation vector that matches the index of the skill to be taken in the decision from a trained skill representation vector library.

[0121] The action decision unit 1003 is configured to decide the actions to be taken in the H steps within the current skill window in sequence through the trained action strategy network based on the acquired skill representation vector, the state after entering the current skill window and the actions taken.

[0122] It should be understood that the specific processing performed by the above-mentioned devices and systems has been referred to Figures 1 to 3 The details are described in detail and will not be repeated here.

[0123] It should be understood that each unit in the above-mentioned apparatus and system may be implemented as hardware components and / or software components.

[0124] Figure 11 A structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0125] Reference Figure 11 The electronic device 1100 includes: at least one memory 1101 and at least one processor 1102, wherein the at least one memory 1101 stores a set of computer-executable instructions. When the computer-executable instruction set is executed by the at least one processor 1102, at least one of the following items is executed: the offline skill extraction method as described in the above exemplary embodiment, the offline training method as described in the above exemplary embodiment, and the skill guidance decision method as described in the above exemplary embodiment.

[0126] As an example, the electronic device can be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device is not necessarily a single electronic device, but can also be any collection of devices or circuits that can execute the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0127] In an electronic device, the processor 1102 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor 1102 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0128] The processor 1102 may execute instructions or codes stored in the memory 1101, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0129] The memory 1101 may be integrated with the processor 1102, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 1101 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 1101 and the processor 1102 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 1102 can access files stored in the memory.

[0130] In addition, the electronic device may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0131] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute at least one of the following items: the offline skill extraction method as described in the above exemplary embodiment, the offline training method as described in the above exemplary embodiment, and the skill guidance decision method as described in the above exemplary embodiment. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store the computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0132] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, the instructions in which may be executed by at least one processor to complete at least one of the following items: an offline skill extraction method as described in the above exemplary embodiment, an offline training method as described in the above exemplary embodiment, and a skill guidance decision method as described in the above exemplary embodiment.

[0133] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0134] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for extracting skills offline from an intelligent agent for performing multiple tasks, characterized in that: The offline skill extraction method includes: Obtaining training samples, wherein the training samples include: sub-trajectories obtained by segmenting a complete trajectory according to a preset length H steps, wherein the complete trajectory is composed of the states and actual actions taken at each step to complete a task; Inputting the state change of each sub-trajectory in the training sample into a skill encoder to obtain a skill encoding vector for each sub-trajectory, wherein the state change of each sub-trajectory is the change of the end state of the sub-trajectory relative to the starting state of the sub-trajectory; For each skill encoding vector, determine the skill representation vector that is most similar to the skill encoding vector from the M skill representation vectors in the skill representation vector library; Compute the skill extraction loss based on the difference between each skill encoding vector and its most similar skill representation vector; Based on the skill extraction loss, the parameters of the skill encoder and the skill representation vectors in the skill representation vector library are updated.

2. An offline training method for an intelligent agent to perform multiple tasks, characterized in that: The intelligent agent includes: a skill representation vector library, a skill strategy network for deciding the skills to be adopted, and an action strategy network for deciding the actions to be taken based on the adopted skills, wherein the offline training method includes: By executing the offline skill extraction method according to claim 1, a trained skill encoder and a skill representation vector library are obtained; Get training samples; Inputting the state changes of each sub-trajectory in the training sample into the trained skill encoder to obtain the skill encoding vector of each sub-trajectory; Determine the target skill representation vector corresponding to each sub-trajectory, and use the index of the target skill representation vector as the skill label of the sub-trajectory, where the target skill representation vector is the skill representation vector in the trained skill representation vector library that is most similar to the skill encoding vector of the sub-trajectory; For each sub-trajectory, the state and target skill representation vectors corresponding to the sub-trajectories preceding it, as well as the initial state of the sub-trajectory, are input into the skill strategy network to obtain the index of the skill to be adopted in the predicted sub-trajectory. Calculating the loss of the skill strategy network based on the skill index prediction results and skill labels corresponding to each sub-trajectory; Based on the loss of the skill policy network, parameters of the skill policy network are updated.

3. The offline training method according to claim 2, characterized in that: Also includes: For each sub-trajectory in the training sample, based on the sub-trajectory and its corresponding target skill representation vector, the action strategy network predicts the action to be taken at each step in the sub-trajectory; Calculate the loss of the action policy network based on the action prediction results and the actual actions taken for each step in each sub-trajectory; Based on the loss of the action policy network, parameters of the action policy network are updated.

4. The offline training method according to claim 3, characterized in that: For each sub-trajectory in the training sample, based on the sub-trajectory and its corresponding target skill representation vector, predicting the action to be taken at each step in the sub-trajectory by the action strategy network includes: For each step in each sub-trajectory, the target skill representation vector corresponding to the sub-trajectory, the state of the step and the state before the step in the sub-trajectory, and the actual action taken before the step in the sub-trajectory are input into the action strategy network to obtain the action prediction result of the step.

5. A skill-guided decision-making method for an intelligent agent performing multiple tasks, characterized in that: The skill-guided decision-making method includes: After completing H steps in the previous skill window, the current skill window is entered. The skill representation vectors of the current state, historical state, and skills taken in the historical skill window are input into the trained skill strategy network to determine the index of the skill taken in the current skill window. Obtain a skill representation vector that matches the index of the skill to be taken in the decision from the trained skill representation vector library; Based on the acquired skill representation vector, the state after entering the current skill window, and the action taken, the trained action strategy network is used to sequentially decide the actions to be taken in the H steps within the current skill window. The intelligent agent is trained by executing the offline training method according to any one of claims 2 to 4.

6. A device for extracting skills offline from an intelligent agent for performing multiple tasks, characterized in that: The offline skill extraction device includes: a sample acquisition unit configured to acquire training samples, wherein the training samples include: sub-trajectories obtained by segmenting a complete trajectory according to a preset length H steps, wherein the complete trajectory is composed of the states and actual actions taken at each step of completing a task; a skill encoding unit configured to input the state change of each sub-trajectory in the training sample into a skill encoder to obtain a skill encoding vector for each sub-trajectory, wherein the state change of each sub-trajectory is the change of the end state of the sub-trajectory relative to the starting state of the sub-trajectory; a representation determination unit configured to determine, for each skill encoding vector, a skill representation vector that is most similar to the skill encoding vector from among the M skill representation vectors in the skill representation vector library; a skill extraction loss calculation unit, configured to calculate a skill extraction loss based on a difference between each skill encoding vector and a skill representation vector most similar to the skill encoding vector; A skill extraction and updating unit is configured to update the parameters of the skill encoder and the skill representation vectors in the skill representation vector library based on the skill extraction loss.

7. An offline training system for an intelligent agent performing multiple tasks, characterized in that The intelligent agent includes: a skill representation vector library, a skill strategy network for deciding the skills to be taken, and an action strategy network for deciding the actions to be taken based on the skills to be taken, wherein the offline training system includes: the skill offline extraction device according to claim 6, the skill strategy network training device, and the action strategy network training device; Wherein, the skill strategy network training device includes: A sample acquisition unit is configured to acquire training samples; a skill encoding unit configured to input the state changes of each sub-trajectory in the training sample into a trained skill encoder to obtain a skill encoding vector for each sub-trajectory; a skill label determination unit configured to determine a target skill representation vector corresponding to each sub-trajectory and use the index of the target skill representation vector as the skill label of the sub-trajectory, wherein the target skill representation vector is the skill representation vector in the trained skill representation vector library that is most similar to the skill encoding vector of the sub-trajectory; A skill prediction unit is configured to, for each sub-trajectory, input the states and target skill representation vectors corresponding to the sub-trajectories preceding the sub-trajectory, as well as the initial state of the sub-trajectory, into the skill strategy network to obtain the index of the skill predicted to be adopted in the sub-trajectory; a skill loss calculation unit configured to calculate the loss of the skill strategy network based on the skill index prediction results and skill labels corresponding to each sub-trajectory; a skill strategy network updating unit, configured to update parameters of the skill strategy network based on the loss of the skill strategy network; Wherein, the action strategy network training device includes: an action prediction unit configured to predict, for each sub-trajectory in the training sample, an action to be taken at each step in the sub-trajectory based on the sub-trajectory and its corresponding target skill representation vector using the action strategy network; an action loss calculation unit, configured to calculate the loss of the action strategy network based on the action prediction results and the actual actions taken corresponding to each step in each sub-trajectory; The action policy network updating unit is configured to update the parameters of the action policy network based on the loss of the action policy network.

8. A skill-guided decision-making system for an intelligent agent performing multiple tasks, characterized in that: The skill guidance decision system includes: The skill decision unit is configured to enter the current skill window after completing H steps in the previous skill window, input the skill representation vectors of the current state and the skills adopted in the historical state and historical skill windows into the trained skill strategy network, and decide the index of the skill adopted in the current skill window; A skill representation vector acquisition unit is configured to acquire a skill representation vector that matches an index of a skill to be taken in a decision from a trained skill representation vector library; The action decision unit is configured to sequentially decide the H-step actions to be taken within the current skill window through the trained action strategy network based on the acquired skill representation vector, the state after entering the current skill window, and the actions taken; The intelligent agent is trained by executing the offline training method according to any one of claims 2 to 4.

9. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 4 and / or the method according to claim 5.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; at least one memory storing computer-executable instructions, When the computer-executable instructions are executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform the method according to any one of claims 1 to 4 and / or the method according to claim 5.

11. A computer program product comprising computer executable instructions, characterized in that: When the computer-executable instructions are executed by at least one processor, the method according to any one of claims 1 to 4 and / or the method according to claim 5 are implemented.

Images