Skill redundancy-removal hierarchical reinforcement learning method, device, medium and product

Abstract

The application discloses a skill redundancy-eliminating hierarchical reinforcement learning method and device, medium and product, relates to the technical field of autonomous decision-making of unmanned vehicles, and realizes the combination of offline learning and online learning. A preliminary skill strategy set is obtained through preliminary skill strategy learning of offline trajectory data, then the preliminary skill strategy set is interacted with an environment to learn a high-level strategy, and the skill strategy is relearned to obtain a relatively convergent skill strategy related to a task. The skill redundancy defect of the unmanned vehicle in the autonomous decision-making process can be avoided, the efficiency of the unmanned vehicle in processing the task and making decisions is improved, the complexity of the autonomous decision-making of the unmanned vehicle is reduced, and finally the unmanned vehicle is guided to complete the entire complex dynamic task.

Description

Technical Field

[0001] This invention relates to the field of autonomous decision-making technology for unmanned vehicles, and in particular to a skill redundancy removal hierarchical reinforcement learning method, device, medium, and product based on offline datasets. Background Technology

[0002] Hierarchical reinforcement learning is one of the most important subfields of reinforcement learning and an effective means to solve the binary sparse reward problem. Based on semi-Markov decision processes, temporal abstraction methods, and deep learning methods, hierarchical reinforcement learning is theoretically extended to achieve stronger feature extraction and policy learning capabilities. According to the differences in solution approaches, hierarchical reinforcement learning methods can be divided into skill-based deep hierarchical reinforcement learning methods and sub-objective-based deep hierarchical reinforcement learning methods.

[0003] Skill-based deep hierarchical reinforcement learning methods define a set of skills, each consisting of triples. <I,π s ,β s > is used to represent this. The meaning of this triple is: I represents the initial state set of skill s, π s β represents the internal policy of skill s. s Let represent the termination function of skill s. First, the upper-level controller selects a skill over a longer time span, while the lower-level controller selects and executes an action based on the skill over a shorter time span. Although defining skills increases the complexity of the Markov decision process, the skill-based framework has the advantages of ease of implementation and hierarchical extensibility.

[0004] Sub-goal-based deep hierarchical reinforcement learning methods do not directly define skills, but instead design specific states as a set of sub-goals. The process of the agent reaching a sub-goal from the current state is considered a skill. The general framework involves using state s as the input to the upper-level controller, which outputs a sub-goal; and using the state-sub-goal as the augmented state and as the input to the lower-level controller, allowing the lower-level controller to learn the lower-level policy π under the augmented state. l Let S×G→A, where S is the state space, G is the sub-target space, and A is the action space. The upper-level controller is trained based on external rewards, while the lower-level policy is trained based on internal rewards.

[0005] In summary, current skill-based deep hierarchical reinforcement learning methods mainly address the challenges of policy forgetting and reward coefficients in complex, long-sequence tasks. By decomposing tasks or forming skill sets, they improve learning efficiency, help agents achieve rapid coverage of a wider range of state spaces, and enhance exploration capabilities.

[0006] However, in some current skill-based deep hierarchical reinforcement learning algorithms, the training processes for skills and upper-level policies are separated, resulting in many skills trained in the lower-level networks being task-irrelevant and exhibiting similarities, which greatly reduces the diversity of skills.

[0007] Therefore, in practical applications of autonomous vehicle decision-making, although skill-based deep hierarchical reinforcement learning methods can simplify complex problems and thus solve complex tasks more effectively, autonomous vehicles often suffer from skill redundancy due to their low utilization of sample data and the imbalance between exploration and utilization in dynamic environments. This means that skills exhibit extremely high similarity, significantly reducing skill diversity. Skill redundancy in autonomous vehicles leads to a substantial decrease in task efficiency and decision-making efficiency, increasing the complexity of decision-making. Furthermore, because autonomous vehicles typically select only a subset of skills during training, some skills may not converge, resulting in skills that are useless or even harmful to the autonomous decision-making task. Summary of the Invention

[0008] The purpose of this invention is to provide a skill redundancy removal hierarchical reinforcement learning method, device, medium, and product, which can avoid the defects of skill redundancy in autonomous vehicle autonomous decision-making, improve the efficiency of autonomous vehicle task processing and decision-making, and reduce the complexity of autonomous vehicle autonomous decision-making.

[0009] To achieve the above objectives, the present invention provides the following solution:

[0010] In a first aspect, the present invention provides a skill redundancy removal hierarchical reinforcement learning method, comprising:

[0011] The current state of the unmanned vehicle when it interacts with the test environment in real time while performing autonomous driving tasks is obtained, wherein the autonomous driving tasks include one or more of navigation, traffic rule compliance, obstacle avoidance and path planning;

[0012] The current state is used as input, and a high-level policy network is used to output the probability distribution of different skills selected in the current state. The skills are a preset number of learned skills in the initial skill policy set. The initial skill policy set is obtained by pre-training the offline trajectory dataset of the autonomous vehicle based on the first skill value network and the first skill policy network. The initial skill policy set includes the preset number of mapping relationships. Each mapping relationship is: state space - encoding of the corresponding skill → action space. The state space includes multiple states, and the action space includes multiple actions.

[0013] The current state and the code corresponding to the first target skill are used as inputs, and the first value of the first target skill in the current state is output using a high-level value network, wherein the first target skill is the skill with the highest probability.

[0014] The current state and the encoding corresponding to the first target skill are used as inputs, and the action corresponding to the first target skill is output using the second skill policy network, wherein the second skill policy network is the trained first skill policy network;

[0015] Using the current state as input, the second value of the first target skill in the current state is output using the second skill value network;

[0016] Based on the probability distribution of selecting different skills under the current state, environmental rewards, and the first value, the high-level value network and the high-level policy network are calculated, and the first loss function and the third loss function are obtained accordingly.

[0017] Based on the environmental reward and the second value, the loss functions of the second skill value network and the second skill policy network are calculated, and the second loss function and the fourth loss function are obtained accordingly.

[0018] Determine whether the first training completion condition has been met and obtain the first determination result. The first training completion condition includes whether the current environmental reward fluctuation amplitude is less than a first threshold, or whether the number of training sessions has reached a second threshold.

[0019] If the first judgment result is negative, then update the parameters of the high-level value network, the second skill value network, the high-level policy network and the second skill policy network according to the first loss function, the second loss function, the third loss function and the fourth loss function respectively, and return to the step "obtain the current state of the unmanned vehicle when it performs autonomous driving tasks and interacts with the test environment in real time".

[0020] If the first judgment result is yes, then the training of the high-level value network, the second skill value network, the high-level policy network, and the second skill policy network is completed.

[0021] Optionally, the process of pre-training the first skill value network and the first skill policy network on the offline trajectory dataset of the autonomous vehicle includes:

[0022] The offline trajectory dataset of an autonomous vehicle is obtained when it completes a specific task from its starting position to its target position. The offline trajectory dataset includes at least one motion trajectory, and each motion trajectory includes a trajectory dictionary obtained based on the current state of the autonomous vehicle, the action taken in the current state, the reward obtained after performing the current action, whether the specific task has been completed in the current state, and the state at the next moment.

[0023] Hyperparameters are determined based on the characteristics of the specific task, wherein the hyperparameters are the number of skills to be learned in the initial skill strategy set;

[0024] Based on the hyperparameters, one-hot encoding is performed on each skill to be learned to obtain each encoded skill;

[0025] For each trajectory dictionary, the feature vector of the current state in the trajectory dictionary is used as input, and the third value of each encoded skill in the current state is output using the first skill value network;

[0026] The third highest value encoded skill is taken as the second target skill, and the second target skill is concatenated with the feature vector corresponding to the current state to obtain the skill representation vector;

[0027] Using the skill representation vector as input, the first skill policy network is used to output the corresponding action under the second target skill.

[0028] The skill representation vector is used as input, and the discriminator model is used to output the information entropy between each second target skill and its corresponding state, wherein the corresponding state refers to all states that can be concatenated with the second target skill.

[0029] Based on the information entropy, the loss functions of the first skill value network, the first skill strategy network, and the discriminator model are calculated respectively, resulting in the fifth loss function, the sixth loss function, and the seventh loss function.

[0030] Determine whether the second training completion condition has been met and obtain the second determination result. The second training completion condition includes whether the average value of all the information entropy of the preset number of training sessions meets the third threshold, or whether the number of training sessions has reached the fourth threshold.

[0031] If the second judgment result is negative, then the parameters of the first skill value network, the first skill policy network, and the discriminator model are updated according to the fifth loss function, the sixth loss function, and the seventh loss function, respectively.

[0032] If the second judgment result is yes, then the training of the first skill value network, the first skill policy network, and the discriminator model is complete.

[0033] Optionally, the process of obtaining the feature vector of the current state in the trajectory dictionary includes:

[0034] The current state in the trajectory dictionary includes image state data and radar sensing data of the current environment;

[0035] The image state data is used as input, and the first neural network model is used to extract the image feature vector.

[0036] The radar sensing data is used as input, and the radar feature vector is extracted using the second neural network model.

[0037] Based on the image feature vector and the radar feature vector, the feature vector of the current state in the trajectory dictionary is obtained.

[0038] Optionally, the expression for the first loss function is:

[0039]

[0040] Wherein, J(θ) h ) represents the loss value of the first loss function; For state s t Next, select skill z t The first value; R h The total reward value is calculated based on the task-related skill distribution and the environmental rewards; γ h The learning rate of the high-level value network; For state s t+1 Next, select skill z t The value of.

[0041] Optionally, the expression for the second loss function is:

[0042]

[0043] Wherein, J(θ) l ) represents the loss value of the second loss function; For skill z t Arrival in state s t Second value; γ l The learning rate of the second skill value network; For skill z t Arrival in state s t+1 value; r t This is a reward for the environment.

[0044] Optionally, the expression for the fifth loss function is:

[0045]

[0046] in, The loss value of the fifth loss function; Here are the parameters of the first skill value network; E is the expectation. r represents the value of the skill at the current time t. t ' is the reward value, which is calculated based on the information entropy; α v The update rate of the first skill value network; The value of the skill at the next time step t+1; dones indicates whether the skill is in a terminated state at the current time.

[0047] Optionally, the expression for the seventh loss function is:

[0048]

[0049] in, The loss value of the seventh loss function; are the parameters of the discriminator model; n_skills are the hyperparameters; p(z) is the probability of the skill; The information entropy is denoted as .

[0050] In a second aspect, the present invention provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of a skill redundancy-removing hierarchical reinforcement learning method as described in any of the preceding claims.

[0051] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps of the skill redundancy removal hierarchical reinforcement learning method described in any of the preceding claims.

[0052] Fourthly, the present invention provides a computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of a skill redundancy removal hierarchical reinforcement learning method as described in any of the preceding claims.

[0053] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0054] This invention provides a skill redundancy removal hierarchical reinforcement learning method, device, medium, and product. The method combines offline and online learning. First, it performs preliminary skill policy learning using offline trajectory data to obtain a preliminary set of skill policies. Then, it interacts with the environment to learn higher-level policies. Simultaneously, it relearns the skill policies to obtain relatively convergent and task-related skill policies. This avoids the skill redundancy defects in autonomous vehicle decision-making, improves the efficiency of autonomous vehicle task processing and decision-making, reduces the complexity of autonomous vehicle decision-making, and ultimately guides the autonomous vehicle to complete the entire complex dynamic task. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a skill redundancy removal hierarchical reinforcement learning method provided in Embodiment 1 of the present invention;

[0057] Figure 2 (a) is a schematic diagram of the Unity 3D experimental environment in Embodiment 1 of the present invention;

[0058] Figure 2 (b) is a schematic diagram of the task in Embodiment 1 of the present invention;

[0059] Figure 3 This is a diagram of the internal structure of a computer device. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The purpose of this invention is to provide a skill redundancy removal hierarchical reinforcement learning method, device, medium, and product, which can avoid the defects of skill redundancy in autonomous vehicle autonomous decision-making, improve the efficiency of autonomous vehicle task processing and decision-making, and reduce the complexity of autonomous vehicle autonomous decision-making.

[0062] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] Example 1

[0064] One skill redundancy removal hierarchical reinforcement learning method in this embodiment includes:

[0065] Sa: Obtain the current state of the unmanned vehicle when it interacts with the test environment in real time while performing autonomous driving tasks, wherein the autonomous driving tasks include one or more of navigation, traffic rule compliance, obstacle avoidance and path planning.

[0066] Sb: Taking the current state as input, the high-level policy network outputs the probability distribution of different skills selected in the current state. The skills are a preset number of learned skills in the initial skill policy set. The initial skill policy set is obtained by pre-training the offline trajectory dataset of the autonomous vehicle based on the first skill value network and the first skill policy network. The initial skill policy set includes the preset number of mapping relationships. Each mapping relationship is: state space - encoding of the corresponding skill → action space. The state space includes multiple states, and the action space includes multiple actions.

[0067] The process of training an autonomous vehicle on an offline trajectory dataset using a first-skill value network and a first-skill policy network includes:

[0068] (1) Obtain the offline trajectory dataset of the unmanned vehicle when it completes a specific task from the starting position to the target position, wherein the offline trajectory dataset includes at least one motion trajectory, and each motion trajectory includes a trajectory dictionary obtained based on the current state of the unmanned vehicle, the action taken in the current state, the reward obtained after performing the current action, whether the specific task has been completed in the current state, and the state at the next moment.

[0069] (2) Determine hyperparameters based on the characteristics of the specific task, wherein the hyperparameters are the number of skills to be learned in the initial skill strategy set.

[0070] (3) Based on the hyperparameters, one-hot encoding is performed on each skill to be learned to obtain each encoded skill.

[0071] (4) For each trajectory dictionary, the feature vector of the current state in the trajectory dictionary is used as input, and the third value of each encoded skill in the current state is output using the first skill value network.

[0072] The process of obtaining the feature vector of the current state in the trajectory dictionary includes:

[0073] 1) The current state in the trajectory dictionary includes the image state data and radar sensing data of the current environment.

[0074] 2) Using the image state data as input, extract the image feature vector using the first neural network model, preferably the first neural network is a CNN network.

[0075] 3) Using the radar sensing data as input, extract the radar feature vector using the second neural network model. Preferably, the second neural network model is an MLP network.

[0076] 4) Based on the image feature vector and the radar feature vector, obtain the feature vector of the current state in the trajectory dictionary.

[0077] (5) Take the third highest value encoded skill as the second target skill, and concatenate the second target skill with the feature vector of the corresponding current state to obtain the skill representation vector.

[0078] (6) Using the skill representation vector as input, the first skill policy network is used to output the action corresponding to the second target skill.

[0079] (7) Using the skill representation vector as input, the discriminator model outputs the information entropy between each second target skill and its corresponding state, wherein the corresponding state refers to all states that can be concatenated with the second target skill.

[0080] (8) Calculate the loss functions of the first skill value network, the first skill strategy network and the discriminator model respectively based on the information entropy, and obtain the fifth loss function, the sixth loss function and the seventh loss function respectively.

[0081] (9) Determine whether the second training completion condition has been met and obtain the second judgment result, wherein the second training completion condition includes whether the average value of all the information entropy of the preset number of training sessions meets the third threshold, or whether the number of training sessions has reached the fourth threshold.

[0082] (10) If the second judgment result is negative, then the parameters of the first skill value network, the first skill policy network and the discriminator model are updated according to the fifth loss function, the sixth loss function and the seventh loss function, respectively.

[0083] (11) If the second judgment result is yes, then the training of the first skill value network, the first skill strategy network and the discriminator model is completed.

[0084] Sc: Taking the current state and the encoding corresponding to the first target skill as input, the first value of the first target skill in the current state is output using a high-level value network, wherein the first target skill is the skill with the highest probability.

[0085] Sd: The current state and the encoding corresponding to the first target skill are taken as input, and the action corresponding to the first target skill is output using the second skill policy network, wherein the second skill policy network is the trained first skill policy network.

[0086] Se: Taking the current state as input, the second value of the first target skill in the current state is output using the second skill value network.

[0087] Sf: Based on the probability distribution of selecting different skills in the current state, environmental rewards, and the first value, calculate the high-level value network and the high-level policy network, and obtain the first loss function and the third loss function accordingly.

[0088] Sg: Based on the environmental reward and the second value, calculate the loss functions of the second skill value network and the second skill policy network to obtain the second loss function and the fourth loss function.

[0089] Sh: Determine whether the first training completion condition has been met and obtain the first judgment result, wherein the first training completion condition includes whether the current environmental reward fluctuation amplitude is less than a first threshold, or whether the number of training sessions has reached a second threshold.

[0090] Si: If the first judgment result is negative, then update the parameters of the high-level value network, the second skill value network, the high-level strategy network and the second skill strategy network according to the first loss function, the second loss function, the third loss function and the fourth loss function respectively, and return to step S1;

[0091] Sj: If the first judgment result is yes, then the training of the high-level value network, the second skill value network, the high-level policy network and the second skill policy network is completed.

[0092] Compared with other methods such as skill redundancy removal in hierarchical reinforcement learning, the method described in this embodiment performs preliminary learning of skill policies using offline datasets, uses information entropy to initially remove redundant skills, and relearns relatively non-converged skills through online learning. Then, the higher-level policy calls upon the skills to form a skill policy useful for the task, ultimately completing the task. This solves the problems of skill redundancy and skills being useless for the task in skill-based hierarchical reinforcement learning. When applied to the field of autonomous vehicle decision-making, it can avoid the defects of skill redundancy in autonomous vehicle decision-making, improve the efficiency of autonomous vehicle task processing and decision-making, and reduce the complexity of autonomous vehicle decision-making.

[0093] To enable those skilled in the art to more clearly understand how the above-described skill redundancy removal hierarchical reinforcement learning method in this embodiment combines offline and online learning processes, the following detailed explanation is provided.

[0094] like Figure 1 As shown, the skill redundancy removal hierarchical reinforcement learning method in this embodiment includes the offline learning process S1-S7 and the online learning process S8-S10, specifically:

[0095] S1: Obtain the offline trajectory dataset of the autonomous vehicle completing the entire task, and perform data preprocessing to obtain the data format required by the algorithm. The specific process is as follows:

[0096] This embodiment verifies the algorithm in a Unity3D-based simulation environment, such as... Figure 2 As shown. After obtaining the offline trajectory dataset, its data format is the environment-uniform format Demo = {behavior_spec, info_action_pairs, total_expected}, where each represents the action space, action-state information, and total expectation, respectively. The required trajectory data is all in info_action_pairs. The required trajectory data is organized according to the same trajectory, based on the current state, the action taken in the current state, the reward obtained after performing the current action, whether the task is completed in the current state, and the next state, forming a trajectory dictionary: Dict = {states, actions, rewards, dones, nest_states}.

[0097] S2: Define the hyperparameter of the number of skills to be learned and encode it using one-hot encoding. The specific process is as follows:

[0098] (2-1) Based on the characteristics of the task, define the hyperparameter n_skills, which represents the number of skills in the skill set formed during the execution of the current task. In the test environment of this embodiment, the number of skills is defined as n_skills = 10, that is, in the subsequent network model, the output variables related to skills are all n_skills = 10.

[0099] (2-2) Then, one-hot encoding is performed on the skills. Based on the hyperparameter n_skills, the skills in the skill set are encoded as follows: skill i =[0,…,1 i [,…,0], where skill i Let 1_i represent the i-th skill, where 1_i means the value at the i-th index is 1, and the rest are 0.

[0100] S3: Extract features from the current state in the trajectory dictionary, input them into the first skill value network, select high-value skills, and then concatenate the state features with the one-hot encoding of the skills to obtain the skill representation vector.

[0101] In the dictionary Dict obtained in S1, states contains image state data and radar sensing data. Features need to be extracted from the image state data and radar sensing data separately, and then concatenated. The state feature extraction network includes CNN networks and MLP networks.

[0102] The specific process of S3 is as follows:

[0103] (3-1) Input layer of CNN network: Image state data ImagesData is used as input. In this embodiment, 84*84*3 dimensional image state data is selected.

[0104] (3-2) Convolutional Layers of the CNN Network: Two-dimensional convolution operations are performed on the input image state data. Feature extraction is then performed using three convolutional layers to obtain the image feature vector. Specifically, the first convolutional layer has in_channels=3, out_channels=32, kernel_size=(1,1), and stride=(1,1); the second convolutional layer has in_channels=32, out_channels=32, kernel_size=(1,1), and stride=(1,1); the third convolutional layer has in_channels=32, out_channels=1, kernel_size=1, and stride=1. The nonlinear transformation between layers uses the ReLU function, with the following formula:

[0105]

[0106] (3-3) Output layer of the CNN network: The vector obtained from the convolution in (3-2) is passed through a fully connected layer again to obtain the final image feature vector, which is used as the output. express.

[0107] (3-4) Input layer of MLP network: Radar sensing data RaysData is used as input. In this embodiment, 802-dimensional radar sensing data is selected.

[0108] (3-5) Fully Connected Layers of the MLP Network: Three fully connected layers are used to extract features from the radar sensing data, resulting in radar feature vectors. Specifically, the first fully connected layer has in_features = 802 and out_features = 512; the second fully connected layer has in_features = 512 and out_features = 256; and the third fully connected layer has in_features = 256 and out_features = 64. The final nonlinear transformation uses the Tanh function, with the following formula:

[0109]

[0110] (3-6) Output layer of the MLP network: The output of (3-5) is the final radar data feature vector, which is then used... express.

[0111] (3-7) Finally and Concatenate the feature vectors to obtain the feature vector of the current state, and use s i This is represented as follows. Inputting this into the first skill value network yields the value of each skill in the current state t, denoted as... The output dimension is the hyperparameter n_skills defined above. The skill with the highest value is selected. i Encode it with one-hot encoding [0,…,1] i ,…,0] and s i Concatenate the vectors to obtain the skill representation vector, using z... i express.

[0112] S4: Use the representation vector of this skill as the input of the first skill policy network to obtain the probability distribution of the corresponding action under the current skill. For example, there is an 80% probability of moving forward, a 10% probability of turning left, and a 10% probability of turning right. The autonomous vehicle will execute the action with the higher probability, which means we have obtained the corresponding action under the current skill.

[0113] S5: Input the representation vector of this skill into the discriminator network model to obtain the information entropy between the skill and the state. The specific process is as follows:

[0114] (5-1) Input layer of the discriminator network: The skill representation vector z obtained in step (3-7) is input into the discriminator network. i As input.

[0115] (5-2) Fully connected layer of the discriminator network: Two fully connected layers are used. Specifically, the first fully connected layer has in_features = z iThe first layer has 256 out_features; the second fully connected layer has 256 in_features and 256 out_features; the nonlinear transformation between layers uses the ReLU function.

[0116] (5-3) Output layer of the discriminator network: Taking the output of the fully connected layer in (5-2) as input, and passing it through a fully connected network with in_features = 256 and out_features = n_skills, the information entropy between skills and corresponding states is obtained, expressed as:

[0117] S6: Based on the information entropy obtained in S5, the reward value is used to judge the quality of the current action, and the loss functions of the discriminator model, the first skill value network, and the first skill policy network are calculated accordingly. The specific process is as follows:

[0118] (6-1) Based on the state information entropy obtained in (5-3) The reward function is calculated using the following formula:

[0119]

[0120] (6-2) Based on (6-1), the loss function of the first skill value network is obtained, and the calculation formula is:

[0121]

[0122] Where, α v For the update rate of the first skill value network, The value of the skill is given at the next time step t+1, and dones indicates whether the skill is in a terminated state at that time.

[0123] The loss function of the discriminator network can also be calculated, and the formula is:

[0124]

[0125] Where p(z) is the probability of the skill, typically...

[0126] For the first-skill policy network, its loss function is as follows:

[0127]

[0128] Where, α θ For the reward discount parameters of the first skill strategy network, For state s t+1 The value obtained based on the first skill value network, π θ (a t |(st ,z t ) is in state s t Next execution skill z t Take action a at that time t The probability of.

[0129] S7: If the average entropy every 50 rounds stabilizes or the maximum number of training iterations N is reached in this round, the skill learning phase is complete; otherwise, update the network parameters in S3 and S4 according to the loss function and continue iterative training. The specific process is as follows:

[0130] (7-1) Store the skill information entropy obtained in each round into a list, and calculate the average value every 50 rounds.

[0131] (7-2) The maximum number of training iterations is set to N1. Online training continues when the maximum number of training iterations reaches N1 or the average entropy value calculated in this round stabilizes. Otherwise, the parameters of the first skill value network in S3-S5 are updated. The parameters θ of the first skill policy network and the parameters of the discriminator network. And continue training skills. The parameter update formulas for the first skill value network and the discriminator network are:

[0132]

[0133]

[0134] The parameters of the first skill policy network are updated using the SGD method, with the following formula:

[0135]

[0136] Wherein, λ1, λ2, and λ3 are the learning rates of the first skill value network, the discriminator network, and the first skill policy network, respectively.

[0137] Through the initial training of the above skill strategies, skills distinguished by state are obtained, forming a preliminary set of skill strategies. The action that each known skill needs to perform in each known state is represented by the symbol: S×Z'→A, where S is the state space, Z' is the skill code, and A is the action space. The state space includes multiple states, and the action space includes multiple actions.

[0138] S8: Next, it interacts with the environment, learning skill strategies online through the high-level policy network, high-level value network, second skill value network, and second skill policy network, and obtaining rewards related to skill distribution and corresponding environmental rewards for skill strategies. The two rewards are then weighted and summed to obtain the total reward. The process is as follows:

[0139] (8-1) Obtain the current state of interaction with the environment and conduct online learning and training.

[0140] (8-2) Using the current state as input, utilize the high-level policy network π h Output the probability distribution of different skills selected in the current state, where the skills are a preset number of learned skills in the preliminary skill strategy set mentioned above.

[0141] The current state and the encoding corresponding to the first target skill are used as inputs. The first value of the first target skill in the current state is output by the high-level value network. The first target skill is the skill with the highest probability. The first value is used to evaluate the quality of the current skill selection, and is used to update the parameters of the high-level policy network.

[0142] The current state and the encoding corresponding to the first target skill are used as inputs, and the action corresponding to the first target skill is output using the second skill policy network, where the second skill policy network is the trained first skill policy network.

[0143] Using the current state as input, the second value of the first target skill in the current state is output using the second skill value network.

[0144] The output of the high-level value network guides the update of the high-level strategy network, and the output of the second-level skill value network guides the update of the second-level skill strategy network.

[0145] During online learning, based on the task-related skill distribution p'(z), i.e., the probability distribution of different skills selected in the current state, and the environmental feedback reward r... t The total reward is calculated using the following formula:

[0146]

[0147] Where C is the time step of the current skill execution, and μ is the expected value of the skill distribution.

[0148] S9. Calculate the loss functions for the high-level value network, the second-skill value network, the high-level policy network, and the second-skill policy network. The specific process is as follows:

[0149] (9-1) Based on the environmental reward r obtained from (8-2) t and total reward R h The loss function of the high-level value network is calculated using the following formula:

[0150]

[0151] in, For state s t Next, select skill z tThe primary value is used to evaluate the quality of the current high-level strategic network; γ h The learning rate of the high-level value network.

[0152] (9-2) Simultaneously, calculate the loss function of the second skill value network, as shown in the following formula.

[0153]

[0154] in For skill z t Arrival in state s t Second value; γ l The learning rate of the second skill value network.

[0155] (9-3) The loss function of the high-level policy network is as follows:

[0156]

[0157] in, For the reward discount parameters of the high-level policy network, For high-level value networks in state s t+1 and skill z t The first value obtained is For state s t Next, take skills z t The probability of.

[0158] (9-4) The loss function of the second skill policy network is as follows:

[0159]

[0160] in, For the reward discount parameter of the second skill strategy network, For the second skill value network in state s t+1 The second value obtained below For state s t The following uses skill z t When executing a t The probability of.

[0161] S10. If the environmental reward value in this round of testing tends to stabilize or reaches the iteration limit, the training of the skill-based redundancy-removing hierarchical reinforcement learning method is complete; otherwise, the parameters of the high-level value network, the second skill value network, the high-level policy network, and the second skill policy network are updated using the policy gradient descent method (SGD method), and the process moves to S8 to continue iterative training. The specific process is as follows:

[0162] (10-1) If the reward value of the environmental feedback obtained in each round tends to be stable, that is, the fluctuation range of the reward value compared with the previous K rounds is less than a certain threshold η, then the algorithm is considered to have converged.

[0163] (10-2) The maximum number of training iterations is set to N². When the maximum number of training iterations is reached (N²) or the fluctuation range of the reward value in this round compared to the reward value in the previous K rounds is less than a certain threshold η, the training of the skill redundancy removal hierarchical reinforcement learning method is complete. Otherwise, update the parameters θ of the high-level value network in S8. h The parameter θ of the second skill value network l The parameters θ of the high-level policy network h The parameters θ of the second skill strategy network l And continue training, with the parameter update formula as follows:

[0164]

[0165]

[0166]

[0167]

[0168] Experimental Description and Results: This embodiment verifies the algorithm in a Unity3D-based simulation environment. First, offline trajectory data is collected for the environment. The task in this environment is: the autonomous vehicle agent needs to press a button to generate a pyramid, then navigate to the pyramid, knock it down, and move to the green brick that fell from the top.

[0169] Table 1 shows the reward comparison results of different training methods in this environment. Among them, the Option-Critic method is a representative skill-based hierarchical reinforcement learning method; HAL is one of the latest hierarchical reinforcement learning methods in 2023; and HPLSE is the method used in this embodiment.

[0170] Table 1 Test results of different training methods

[0171]

[0172] As can be seen, this embodiment achieves an environment reward that is 0.71 and 0.19 higher than Option-Critic and HAL, respectively. After ablation comparison of skills after redundancy removal, the environment rewards for HPLSE-2 skills and HPLSE-4 skills are slightly lower than those for HPLSE-3 skills.

[0173] This embodiment includes: (1) acquiring the trajectory dataset for completing the entire task and performing data preprocessing to obtain the data format required by the algorithm; (2) defining the hyperparameter of the number of skills to be learned and encoding the skills using one-hot encoding; (3) extracting features from the state of the trajectory dataset, inputting them into the first skill value network, selecting high-value skills, and then concatenating the state feature vector with the one-hot encoding of the skills to obtain the representation vector of the skills; (4) inputting the representation vector of the skills into the first skill policy network and outputting the action corresponding to the current skill; (5) inputting the representation vector of the skills into the discriminator model to obtain the information entropy between the skills and the state; (6) using the information entropy obtained in step (5) as the reward value to calculate the loss function of the discriminator network and the first skill value network; (7) if the entropy is stable or reaches the iteration limit, the skill learning stage is completed; otherwise, jump to step (3) to continue execution; (8) then interacting with the environment, learning the skill policy online through the high-level policy network, high-level value network, second skill value network and second skill policy network to obtain the reward for the skill distribution and the environmental reward for the corresponding skill policy. (9) Calculate the weighted sum of the two rewards to obtain the total reward; (8) Based on the environmental reward and total reward obtained in step (8), calculate the loss functions of the high-level value network, the second skill value network, the high-level policy network and the second skill policy network, and update the network using the policy gradient method; (10) If the environmental reward value of this round of training tends to be stable or reaches the iteration limit, the training of the hierarchical reinforcement learning method is completed; otherwise, jump to step (8) to continue training.

[0174] In step (3), the first skill value network extracts features from the state of the trajectory dataset and selects skills with high value. This selection mechanism helps ensure that the learned skills are relevant to the task. Only skills that are useful for the task will be selected, while irrelevant skills will be excluded.

[0175] In step (5), the discriminator network is used to measure the information entropy between skills and states, which helps to evaluate the effectiveness and applicability of skills. If the information entropy between skills and states is low, it means that the skills perform relatively consistently in different states, thereby increasing the diversity and applicability of skills.

[0176] In step (6), the information entropy is used as the reward value to calculate the loss function between the discriminator network and the first skill value network. By optimizing this loss function, the selection of skills can be made more task-related, and the diversity among skills can be increased.

[0177] In step (8), the high-level policy network selects the skill to be used in the current state when interacting with the environment. This process ensures that the skill used is task-related and that the total reward is obtained through a weighted sum of the skill distribution rewards and the environmental rewards, thereby better promoting the learning and use of skills.

[0178] By using this update method in step (9), the network parameters can be gradually optimized, making the learned skills more relevant to the task and increasing the diversity among skills.

[0179] In summary, this embodiment learns the basic policy of a skill through unsupervised skill discovery based on offline data, then learns the global policy through interaction with the environment, and relearns the skill policy. This addresses the challenge of skill redundancy caused by the imbalance between skill policy exploration and utilization in hierarchical reinforcement learning, forming a more stable hierarchical reinforcement learning method for decision-making. It also solves the problems of skill irrelevance to the task, high similarity between skills, and reduced skill diversity caused by the separation of the training process of skills and upper-level policies.

[0180] The basic features of this embodiment are as follows: First, the skill base is evaluated using two standards, state information entropy and skill distribution, with the aim of obtaining a diverse and useful set of skills required to complete complex tasks; second, when learning skill combination strategies on a larger time scale, adaptive weight parameters are used to update the skill strategy simultaneously to prevent the addition of skills that are irrelevant to the entire complex task.

[0181] Compared with other methods such as skill redundancy removal in hierarchical reinforcement learning, the method described in this embodiment combines offline and online learning. It performs preliminary skill learning using offline trajectory data, uses state information entropy as a reward, and then interacts with the environment to obtain a reward related to the skill distribution as an intrinsic driver. This encourages asynchronous training between the upper and lower layers, enabling relatively convergent skills to learn task-related policies and ultimately complete the entire complex dynamic task. This solves the problems of skill redundancy and skill uselessness in skill-based hierarchical reinforcement learning.

[0182] Example 2

[0183] A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to implement the steps of a skill redundancy-reduction hierarchical reinforcement learning method of Embodiment 1.

[0184] Example 3

[0185] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a skill redundancy-removing hierarchical reinforcement learning method of Embodiment 1.

[0186] Example 4

[0187] A computer program product includes a computer program that, when executed by a processor, implements the steps of a skill redundancy-removing hierarchical reinforcement learning method as described in Embodiment 1.

[0188] Example 5

[0189] A computer device, which may be a database, may have an internal structure diagram as shown below. Figure 3 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores pending transactions. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a skill redundancy-reduction hierarchical reinforcement learning method as described in Embodiment 1.

[0190] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0191] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0193] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A skill redundancy removal hierarchical reinforcement learning method, characterized in that, The method includes: The current state of the unmanned vehicle when it interacts with the test environment in real time while performing autonomous driving tasks is obtained, wherein the autonomous driving tasks include one or more of navigation, traffic rule compliance, obstacle avoidance and path planning; The current state is used as input, and a high-level policy network is used to output the probability distribution of different skills selected in the current state. The skills are a preset number of learned skills in the initial skill policy set. The initial skill policy set is obtained by pre-training the offline trajectory dataset of the autonomous vehicle based on the first skill value network and the first skill policy network. The initial skill policy set includes the preset number of mapping relationships. Each mapping relationship is: state space - encoding of the corresponding skill → action space. The state space includes multiple states, and the action space includes multiple actions. The current state and the code corresponding to the first target skill are used as inputs, and the first value of the first target skill in the current state is output using a high-level value network, wherein the first target skill is the skill with the highest probability. The current state and the encoding corresponding to the first target skill are used as inputs, and the action corresponding to the first target skill is output using the second skill policy network, wherein the second skill policy network is the trained first skill policy network; Using the current state as input, the second value of the first target skill in the current state is output using the second skill value network; Based on the probability distribution of selecting different skills under the current state, environmental rewards, and the first value, the high-level value network and the high-level policy network are calculated, and the first loss function and the third loss function are obtained accordingly. Based on the environmental reward and the second value, the loss functions of the second skill value network and the second skill policy network are calculated, and the second loss function and the fourth loss function are obtained accordingly. Determine whether the first training completion condition has been met and obtain the first determination result. The first training completion condition includes whether the current environmental reward fluctuation amplitude is less than a first threshold, or whether the number of training sessions has reached a second threshold. If the first judgment result is negative, then update the parameters of the high-level value network, the second skill value network, the high-level policy network and the second skill policy network according to the first loss function, the second loss function, the third loss function and the fourth loss function respectively, and return to the step "obtain the current state of the unmanned vehicle when it performs autonomous driving tasks and interacts with the test environment in real time". If the first judgment result is yes, then the training of the high-level value network, the second skill value network, the high-level policy network, and the second skill policy network is completed.

2. The skill redundancy removal hierarchical reinforcement learning method according to claim 1, characterized in that, The process of pre-training the first-skill value network and the first-skill policy network on the offline trajectory dataset of the autonomous vehicle includes: The offline trajectory dataset of an autonomous vehicle is obtained when it completes a specific task from its starting position to its target position. The offline trajectory dataset includes at least one motion trajectory, and each motion trajectory includes a trajectory dictionary obtained based on the current state of the autonomous vehicle, the action taken in the current state, the reward obtained after performing the current action, whether the specific task has been completed in the current state, and the state at the next moment. Hyperparameters are determined based on the characteristics of the specific task, wherein the hyperparameters are the number of skills to be learned in the initial skill strategy set; Based on the hyperparameters, one-hot encoding is performed on each skill to be learned to obtain each encoded skill; For each trajectory dictionary, the feature vector of the current state in the trajectory dictionary is used as input, and the third value of each encoded skill in the current state is output using the first skill value network; The third highest value encoded skill is taken as the second target skill, and the second target skill is concatenated with the feature vector corresponding to the current state to obtain the skill representation vector; Using the skill representation vector as input, the first skill policy network is used to output the corresponding action under the second target skill. The skill representation vector is used as input, and the discriminator model is used to output the information entropy between each second target skill and its corresponding state, wherein the corresponding state refers to all states that can be concatenated with the second target skill. Based on the information entropy, the loss functions of the first skill value network, the first skill strategy network, and the discriminator model are calculated respectively, resulting in the fifth loss function, the sixth loss function, and the seventh loss function. Determine whether the second training completion condition has been met and obtain the second determination result. The second training completion condition includes whether the average value of all the information entropy of the preset number of training sessions meets the third threshold, or whether the number of training sessions has reached the fourth threshold. If the second judgment result is negative, then the parameters of the first skill value network, the first skill policy network, and the discriminator model are updated according to the fifth loss function, the sixth loss function, and the seventh loss function, respectively. If the second judgment result is yes, then the training of the first skill value network, the first skill policy network, and the discriminator model is complete.

3. The skill redundancy removal hierarchical reinforcement learning method according to claim 2, characterized in that, The process of obtaining the feature vector of the current state in the trajectory dictionary includes: The current state in the trajectory dictionary includes image state data and radar sensing data of the current environment; The image state data is used as input, and the first neural network model is used to extract the image feature vector. The radar sensing data is used as input, and the radar feature vector is extracted using the second neural network model. Based on the image feature vector and the radar feature vector, the feature vector of the current state in the trajectory dictionary is obtained.

4. The skill redundancy removal hierarchical reinforcement learning method according to claim 1, characterized in that, The expression for the first loss function is: Where J(θ) h ) represents the loss value of the first loss function; For state s t Next, select skill z t The first value; R h The total reward value is calculated based on the skill distribution related to the task and the environmental rewards; γ h The learning rate of the high-level value network; For state s t+1 Next, select skill z t The value of.

5. The skill redundancy removal hierarchical reinforcement learning method according to claim 1, characterized in that, The expression for the second loss function is: Where J(θ) l ) represents the loss value of the second loss function; For skill z t Arrival in state s t Second value; γ l The learning rate of the second skill value network; For skill z t Arrival in state s t+1 value; r t This is a reward for the environment.

6. The skill redundancy removal hierarchical reinforcement learning method according to claim 2, characterized in that, The expression for the fifth loss function is: in, The loss value of the fifth loss function; Here are the parameters of the first skill value network; E is the expectation. r represents the value of the skill at the current time t. t ' is the reward value, which is calculated based on the information entropy; α v The update rate of the first skill value network; The value of the skill at the next time step t+1; dones indicates whether the skill is in a terminated state at the current time.

7. The skill redundancy removal hierarchical reinforcement learning method according to claim 2, characterized in that, The expression for the seventh loss function is: in, The loss value of the seventh loss function; are the parameters of the discriminator model; n_skills are the hyperparameters; p(z) is the probability of the skill; The information entropy is denoted as .

8. A computer device comprising: The memory and processor contain a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the skill redundancy-removing hierarchical reinforcement learning method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of a skill redundancy-removing hierarchical reinforcement learning method as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of a skill redundancy-removing hierarchical reinforcement learning method as described in any one of claims 1-7.

Images