Decision control strategy generation method and device for expectation type constraint

Abstract

The embodiment of the invention relates to the technical field of control and decision, and provides a decision control strategy generation method and device for an expected constraint, and the method comprises the steps: constructing a constraint Markov model which comprises an action value function and an action constraint value function according to the expected constraint of a target agent; modeling according to the action value function and the action constraint value function to obtain an action value normal distribution model and an action constraint value normal distribution model of each state action pair, and determining a mean value and a variance of each state action pair according to the action value normal distribution model and the action constraint value normal distribution model; the sampling probability of the intelligent agent in each state action pair is calculated; and performing iterative sampling on the action in the agent action space according to the sampling probability to obtain an optimal decision control strategy of the target agent. According to the embodiment of the invention, the probability of gradually obtaining the maximum and optimal strategy can be improved, and the sampling efficiency is integrally improved.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of control and decision-making, and in particular, to a method and device for generating a decision control strategy for expected constraints. Background Art

[0002] In an intelligent vehicle system, as an intelligent agent, the vehicle needs to make corresponding decisions based on the current environmental state (such as road conditions, the positions and speeds of other vehicles, etc.), and these decisions will affect the future state of the vehicle. Additionally, during the decision-making process of the intelligent vehicle system, there are often various inequality constraints, such as safety distance constraints, speed constraints, acceleration constraints, etc. The intelligent vehicle needs to plan future actions based on the current state and inequality constraints. Therefore, the decision control optimization problem of the intelligent vehicle system can be modeled as a constrained Markov decision problem, enabling the intelligent vehicle to make optimal decisions while satisfying the constraint conditions.

[0003] In the prior art, there are significant challenges in addressing the control and decision-making problems of the intelligent vehicle system facing expected constraints. Expected constraints are usually characterized in forms such as safety probability, and such constraints are difficult to accurately calculate through simple mathematical or probability models and need to be characterized based on a large number of simulation samples. However, the intelligent vehicle system is a deeply complex system with characteristics such as a complex carrier model and high randomness of the working environment, which makes it extremely difficult to predict the safety and performance of policy optimization. Existing methods perform poorly in improving the sampling efficiency, resulting in slow evaluation and optimization speeds. Therefore, there is an urgent need for a method that can use simulation samples more efficiently, improve the sampling efficiency, and thus accelerate the optimization process of the optimal policy. Summary of the Invention

[0004] Aiming at the above problems of the prior art, the purpose of the embodiments of this specification is to provide a method and device for generating a decision control strategy for expected constraints, so as to improve the sampling efficiency and thus gradually obtain the probability of the optimal policy while satisfying the expected constraints.

[0005] To solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:

[0006] On the one hand, the embodiments of this specification provide a method for generating a decision control strategy for expected constraints, and the method includes:

[0007] Regarding the target intelligent vehicle system as an intelligent agent, obtain the expected constraints of the intelligent agent;

[0008] Construct a constrained Markov model of the intelligent agent according to the expected constraints, where the constrained Markov model includes an action value function and an action constraint value function;

[0009] Based on the action value function and the action constraint value function, an action value normal distribution model and an action constraint value normal distribution model for each state-action pair are modeled;

[0010] Based on the action value normal distribution model and the action constraint value normal distribution model, the mean and variance of each state-action pair are determined, and the sampling weight of the agent for each state-action pair is calculated according to the mean and variance;

[0011] Based on the sampling weight of the agent for each state-action pair, the sampling probability of each state-action pair is calculated;

[0012] Based on the sampling probability of each state-action pair, actions in the action space of the agent are iteratively sampled until a preset iteration termination condition is reached, and the optimal decision control strategy of the target intelligent vehicle system is obtained.

[0013] Further, the construction of the constraint Markov model of the agent according to the expectation type constraint includes:

[0014] The constraint Markov model of the agent is constructed using the following formula:

[0015]

[0016] where π represents the sampling strategy, s t represents the state at time step t, a t represents the action at time step t, γ t represents the discount factor, r(s t , a t ) represents the immediate reward function value of action a t in state s t , d0(s) represents the initial state distribution, h(s t , a t ) represents the constraint function, and η represents the expectation type constraint target value.

[0017] Further, the modeling of the action value normal distribution model and the action constraint value normal distribution model for each state-action pair according to the action value function and the action constraint value function includes:

[0018] Each state-action pair is input into a number of pre-trained neural network models to obtain a number of action value prediction values and action constraint value prediction values for each state-action pair;

[0019] The corresponding action value mean and action value variance are calculated based on the predicted values of the action values of each state-action pair, and the corresponding action constraint value mean and action constraint value variance are calculated based on the predicted values of the action constraint values of each state-action pair;

[0020] An action value normal distribution model for each state-action pair is modeled based on the action value mean and action value variance of each state-action pair;

[0021] An action constraint value normal distribution model for each state-action pair is modeled based on the action constraint value mean and action constraint value variance of each state-action pair.

[0022] Further, the action value function and the action constraint value function are determined using the following formulas:

[0023]

[0024] where Q π (s,a) represents the action value function of action a in state s, γ represents the discount factor, r(s,a) represents the immediate reward function value of action a in state s, represents the action constraint value function of action a in state s, P(s'|s,π(s)) represents the state transition probability from state s to s’, h(s,a) represents the constraint function, Q π (s',π(s')) represents the action value function under policy π, represents the action constraint value function under policy π.

[0025] Further, the sampling weights of the agent for each state-action pair calculated based on the mean and variance include:

[0026] Based on the action value normal distribution model, the action value mean and action value variance of each state-action pair are obtained, and the action with the largest action value mean in each state is determined as the optimal action in that state;

[0027] The difference in the action value means between the remaining actions and the optimal action in each state is calculated;

[0028] Based on the action constraint value normal distribution model, the action constraint value mean and action constraint value variance are obtained, and the difference in the action constraint value means between the action constraint value mean of each state-action pair and the expected type constraint target value is calculated;

[0029] Based on the difference in the action value means, the difference in the action constraint value means, the action value variance, and the action constraint value variance of each state-action pair, the sampling weights of the agent for each state-action pair are calculated.

[0030] Further, calculating the sampling weight of the agent for each state-action pair based on the mean difference in action value, the mean difference in action constraint value, the action value variance, and the action constraint value variance of each state-action pair includes:

[0031] Calculating the sampling weight of the agent for each state-action pair using the following formula:

[0032]

[0033] where β(s,a i ) represents the sampling weight of action a in state s i , σ 2 (s,a i ) represents the action value variance of action a in state s i , δ(s,a i ) represents the action value difference of action a in state s i , represents the action constraint value variance of action a in state s i , δ H (s,a i ) represents the action constraint value difference of action a in state s i , represents the action constraint value of action a in state s i , and η represents the expected type constraint target value.

[0034] Further, calculating the sampling probability of each state-action pair based on the sampling weight of the agent for each state-action pair includes:

[0035] Calculating the sampling probability of each state-action pair using the following formula:

[0036]

[0037] where p(a i |s) represents the sampling probability of action a in state s i , β(s,a i ) represents the sampling weight of action a in state s i , and ∑ j β(s,a j ) represents the sum of the sampling weights of all actions in state s.

[0038] On the other hand, an embodiment of the present specification provides a decision control strategy generation device for an expected type constraint, and the device includes:

[0039] A constraint acquisition module, configured to take the target intelligent vehicle system as an agent and acquire the expected constraints of the agent;

[0040] A constraint Markov model construction module, configured to construct a constraint Markov model of the agent according to the expected constraints, where the constraint Markov model includes an action value function and an action constraint value function;

[0041] A normal distribution model modeling module, configured to model an action value normal distribution model and an action constraint value normal distribution model for each state-action pair according to the action value function and the action constraint value function;

[0042] A sampling weight calculation module, configured to determine the mean and variance of each state-action pair according to the action value normal distribution model and the action constraint value normal distribution model, and calculate the sampling weight of the agent in each state-action pair according to the mean and variance;

[0043] A sampling probability calculation module, configured to calculate the sampling probability of each state-action pair according to the sampling weight of the agent in each state-action pair;

[0044] An optimal policy output module, configured to iteratively sample actions in the action space of the agent according to the sampling probability of each state-action pair until a preset iteration termination condition is reached, and obtain the optimal decision control policy of the target intelligent vehicle system.

[0045] In another aspect, an embodiment of the present specification further provides a computer device, including a memory, a processor, and a computer program stored on the memory. When the computer program is run by the processor, it executes the instructions of any one of the above methods.

[0046] In another aspect, an embodiment of the present specification further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor of a computer device, it executes the instructions of any one of the above methods.

[0047] In another aspect, an embodiment of the present specification further provides a computer program product. When the computer program product is run by a processor of a computer device, it executes the instructions of any one of the above methods.

[0048] With the above technical solution, the method for generating a decision control strategy for an expectation-based constraint provided by the embodiments of this specification models the decision control problem of the intelligent vehicle system into a constrained Markov decision model, thereby being able to comprehensively reflect the value and constraint conditions of the agent during the action process. The value function of the constrained Markov decision model is modeled as an action value normal distribution model and an action constraint value normal distribution model. The mean of the normal distribution model reflects the prediction performance of the sampling strategy, and its variance can reflect the strategy uncertainty information. Then, according to the mean and variance provided by the normal distribution model, the sampling probability of each state-action pair is determined, and the actions in the action space of the agent are iteratively sampled according to the sampling probability, enabling the agent to learn valuable information faster and updating the action value function and the action constraint value function with this information. Finally, the optimal decision control strategy of the target agent is obtained, which improves the overall sampling efficiency while taking into account both the strategy performance and the expected constraints, so as to find the optimal decision control strategy at a faster speed.

[0049] The above description is only an overview of the technical solutions of some embodiments of this specification. In order to be able to more clearly understand the technical means of some embodiments of this specification, it can be implemented according to the content of the specification. And to make the above and other purposes, features, and advantages of the embodiments of this specification more obvious and understandable, the following specifically enumerates preferred embodiments and, in conjunction with the accompanying drawings, makes the following detailed description. Brief Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 Shows a schematic diagram of the steps of a method for generating a decision control strategy for an expectation-based constraint in some embodiments of this specification;

[0052] Figure 2 Shows a schematic diagram of the steps of constructing an action value normal distribution model and an action constraint value normal distribution model for each state-action pair in some embodiments of this specification;

[0053] Figure 3 Shows a schematic diagram of the steps of calculating the sampling weight in some embodiments of this specification;

[0054] Figure 4 Shows an iterative sampling flowchart of a method for generating a decision control strategy for an expectation-based constraint in some embodiments of this specification;

[0055] Figure 5 It shows a schematic diagram of a method for calculating sampling probability in some embodiments of this specification;

[0056] Figure 6 It shows a schematic diagram of the structure of a decision control strategy generation device for an expectation - type constraint in some embodiments of this specification;

[0057] Figure 7 It shows a schematic diagram of the structure of a computer device in this specification.

[0058] Description of the accompanying drawing symbols:

[0059] 601, Constraint acquisition module;

[0060] 602, Constraint Markov model construction module;

[0061] 603, Normal distribution model modeling module;

[0062] 604, Sampling weight calculation module;

[0063] 605, Sampling probability calculation module;

[0064] 606, Optimal strategy output module;

[0065] 702, Computer device;

[0066] 704, Processor;

[0067] 706, Memory;

[0068] 708, Driving mechanism;

[0069] 710, Input / output module;

[0070] 712, Input device;

[0071] 714, Output device;

[0072] 716, Presentation device;

[0073] 718, Graphical user interface;

[0074] 720, Network interface;

[0075] 722, Communication link;

[0076] 724, Communication bus. Detailed implementation manners

[0077] The following will clearly and completely describe the technical solutions in the embodiments of this specification with reference to the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0078] It should be noted that the terms "first", "second", etc. in this specification, the claims and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of this specification described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or equipment that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or equipment.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. And the acquisition, storage, use, processing, etc. of data in the technical solutions described in the embodiments of this application all comply with relevant regulations.

[0080] To solve the above problems, the embodiments of this specification provide a method for generating a decision control strategy for expected constraints. Figure 1 It is a schematic diagram of the steps of a method for generating a decision control strategy for expected constraints provided by the embodiments of this specification. This specification provides the method operation steps as described in the embodiments or flowcharts, but based on routine or non-creative labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is only one way among the execution orders of numerous steps and does not represent the only execution order. When the actual system or device product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or executed in parallel. Specifically, as Figure 1 shown, the method may include:

[0081] S101: Regarding the target intelligent vehicle system as an agent, obtain the expected constraints of the agent.

[0082] The intelligent vehicle system is a complex system that integrates deep computing, communication, and control, with characteristics such as a complex carrier model and a highly random working environment. Generally speaking, this system needs to process the state information of the physical world sensed by sensors, analyze and process it through a computing unit, and then the controller regulates physical entities according to instructions. It is very difficult to characterize the dynamic characteristics of this system through precise mathematical - physical formulas, such as differential equation systems, etc., and it is also very difficult to describe the randomness of its working environment using simple probability models. Another important characteristic of such systems is that they have extremely high requirements for safety, but the above characteristics mean that it is extremely difficult to predict the safety of a decision - making and control strategy. Among many ways of characterizing safety, the expected - type constraint is one of the more difficult ones to predict. This kind of constraint is generally characterized in the form of safety probability, etc. (i.e., in the form of mathematical expectation), and it needs to be accurately characterized through a large number of actual measurement cases. However, considering the physical safety of the system, it is generally carried out in a simulation way. In some embodiments, the expected - type constraints of the intelligent vehicle system may include safety constraints, such as avoiding collisions and keeping in the lane, performance constraints, such as speed limits and acceleration limits, and energy - efficiency constraints, such as battery usage efficiency and energy consumption, etc. These constraints are described in the form of probability and are generated based on a large number of simulation samples.

[0083] S102: Construct a constrained Markov model of the intelligent agent according to the expected - type constraint, where the constrained Markov model includes an action - value function and an action - constrained value function.

[0084] In the intelligent vehicle system, the vehicle, as an intelligent agent, needs to make corresponding decisions based on the current environmental state (such as road conditions, the positions and speeds of other vehicles, etc.), and these decisions will affect the future state of the vehicle. Moreover, in the decision - making process of the intelligent vehicle, there are also expected - type constraints in the form of inequalities. The intelligent vehicle needs to plan future actions based on the current state and inequality constraints. Therefore, the sequential decision - making problem with inequality constraints in the intelligent vehicle system can be modeled as a constrained Markov decision problem (CMDP), enabling the intelligent vehicle to make optimal decisions under the premise of meeting the constraint conditions. In the embodiments of this specification, the constrained Markov model of the intelligent agent can be represented by the following formula:

[0085]

[0086] where, π represents the sampling strategy, s t represents the state at time step t, a t represents the action at time step t, γ t represents the discount factor, r(s t , a t ) represents the reward in state s t for action a tThe immediate reward function value, d0(s) represents the initial state distribution, h(s t ,a t ) represents the constraint function, and when its value is 1, it means that the state-action pair does not meet the expected constraints. η represents the expected constraint target value, and η ∈ [0, 1]. At each time step t, the agent will, according to the policy π, based on the observed state s t select the action a for the next time step t , where the policy can be: That is, for each state s, select the action a that maximizes π(s,a).

[0087] In the embodiments of this specification, the constrained Markov model includes two Q-value functions, namely the action value function and the action constraint value function. The action value function is used to reflect the cumulative reward that can be obtained by taking a specific action in a given state, and the action constraint value function is used to reflect whether taking a specific action in a given state meets the constraint conditions. Preferably, the action value function and the action constraint value function can be expressed by the following formulas:

[0088]

[0089] Among them, Q π (s,a) represents the action value function of action a in state s, γ represents the discount factor, r(s,a) represents the immediate reward function value of action a in state s, represents the action constraint value function of action a in state s, P(s'|s,π(s)) represents the state transition probability from state s to s’, h(s,a) represents the constraint function, Q π (s',π(s')) represents the action value function under the policy π, represents the action constraint value function under the policy π. In this way, when sampling, the agent can consider both value and constraints at the same time, thereby avoiding sampling state-action pairs that do not meet the expected constraints and improving the effectiveness of sampling.

[0090] S103: According to the action value function and the action constraint value function, model the action value normal distribution model and the action constraint value normal distribution model for each state-action pair.

[0091] Based on the two Q-value functions of the proposed constrained Markov model, for each state-action pair, a corresponding normal distribution model can be generated by training a large number of data samples. The implementation methods include, but are not limited to, using Gaussian distribution models, Gaussian processes, aggregation of multiple neural networks, Bayesian neural networks, etc. By modeling the action value and action constraint value of the state-action pair as a normal distribution model, the action value prediction and sampling weight of each state-action pair can be obtained according to the provided mean and variance, so as to calculate its sampling probability based on the above two parameters subsequently, thereby guiding the sampling process of policy optimization.

[0092] In the embodiments of this specification, with reference to Figure 2 , constructing the action value normal distribution model and action constraint value normal distribution model for each state-action pair may include the following steps:

[0093] S201: Input each state-action pair into a number of pre-trained neural network models to obtain a number of action value prediction values and action constraint value prediction values for each state-action pair;

[0094] S202: Calculate the corresponding action value mean and action value variance according to the number of action value prediction values of each state-action pair, and calculate the corresponding action constraint value mean and action constraint value variance according to the number of action constraint value prediction values of each state-action pair;

[0095] S203: Model the action value normal distribution model for each state-action pair according to the action value mean and action value variance of each state-action pair;

[0096] S204: Model the action constraint value normal distribution model for each state-action pair according to the action constraint value mean and action constraint value variance of each state-action pair.

[0097] It can be understood that in order to obtain the action value normal distribution model and action constraint value normal distribution model for each state-action pair, the intelligent agent first needs to take various actions in different states and record the rewards obtained and constraint satisfaction for each action. Then, based on these data, multiple different neural network models are trained. Each neural network model can receive the state-action pair as input and output the corresponding action value prediction value and action constraint value prediction value. Since different neural network models have different architectures, parameters, and training strategies, there will be differences in the predicted values they output. However, it should be noted that although the output values of different neural networks are different, the prediction difference values for the same state-action pair will not be too large. Therefore, by comprehensively calculating the mean and variance based on the predicted values of several different neural network models, the robustness and accuracy of the prediction results can be increased. Finally, the corresponding normal distribution model can be constructed according to the mean and variance.

[0098] In the embodiments of this specification, through the above modeling method, the action value normal distribution model and the action constraint value normal distribution model of each state-action pair are obtained and represented by the following formulas:

[0099]

[0100] Among them, Q(s,a) represents the action value function of action a in state s, represents the mean of the action value of action a in state s, and σ 2 (s,a) represents the variance of the action value of action a in state s, and Q H (s,a) represents the action constraint value function of action a in state s, represents the mean of the action constraint value of action a in state s, and σ H 2 (s,a) represents the variance of the action constraint value of action a in state s.

[0101] S104: Determine the mean and variance of each state-action pair according to the action value normal distribution model and the action constraint value normal distribution model, and calculate the sampling weight of the agent in each state-action pair according to the mean and variance.

[0102] It can be understood that the sampling weight reflects the uncertainty of the agent regarding the return that can be obtained from a certain action and whether the constraint is satisfied when performing the action. This uncertainty may stem from various factors, such as environmental noise, sensor errors, model inaccuracies, etc. When the sampling weight is large, it means that the agent does not have sufficient understanding of the current state-action pair, or there are significant potential risks in the current state-action pair (such as possible high returns or serious violations of constraints). In this case, the agent has a stronger incentive to further explore this state-action pair to obtain more accurate information.

[0103] In the embodiments of this specification, referring to Figure 3 , the calculation of the sampling weight of the agent in each state-action pair according to the mean and variance may include the following steps:

[0104] S301: Obtain the action value mean and action value variance of each state-action pair according to the action value normal distribution model, and determine the optimal action in each state as the action with the largest action value mean in that state;

[0105] S302: Calculate the difference in the action value means between the remaining actions and the optimal action in each state;

[0106] S303: Obtain the mean value of the action constraint value and the variance of the action constraint value according to the action constraint value normal distribution model, and calculate the difference between the mean value of the action constraint value of each state-action pair and the mean value of the action constraint value of the expected type constraint target value;

[0107] S304: Calculate the sampling weight of the agent for each state-action pair according to the difference in the mean value of the action value, the difference in the mean value of the action constraint value, the variance of the action value, and the variance of the action constraint value of each state-action pair.

[0108] Specifically, in step S301, according to the action value normal distribution model, the mean value of the action value and the variance of the action value of each state-action pair can be obtained, and these mean values reflect the expected returns that can be obtained by performing different actions in different states. The action with the largest mean value of the action value in each state is determined as the optimal action a b , and the optimal action represents the action that the agent believes is most likely to obtain a high return in the current state. In step S302, it is necessary to calculate the difference in the mean value of the action value between the remaining actions and the optimal action in each state. Through this difference, the difference in returns between other actions and the optimal action can be reflected. The larger the difference, the greater the return gap between the action and the optimal action. In step S303, similarly, according to the action value normal distribution model, the mean value of the action constraint value and the variance of the action constraint value of each state-action pair can be obtained, and then the difference between the mean value of the action constraint value of each state-action pair and the mean value of the action constraint value of the expected type constraint target value is calculated. This difference reflects the gap between the action constraint value and the target constraint. If this value is greater than 0, it means that the current state-action pair does not meet the constraint requirements in the estimation. In step S304, the following formula is used to calculate the sampling weight of the agent for each state-action pair according to the difference in the mean value of the action value, the difference in the mean value of the action constraint value, the variance of the action value, and the variance of the action constraint value:

[0109]

[0110] where, β(s,a i ) represents the sampling weight of action a i in state s, σ 2 (s,a i ) represents the variance of the action value of action a i in state s, δ(s,a i ) represents the difference in the mean value of the action value of action a i in state s, represents the variance of the action constraint value of action a i in state s, δ H (s,a i ) represents the difference in the mean value of the action constraint value of action ai The mean difference in the action constraint value, denotes the action a in state s i The mean of the action constraint value of the action a, η represents the expected constraint target value, where η ∈ [0, 1]. In practical applications, the specific value is set manually according to historical experience. It can be seen from Equation (7) that according to the mean of the action constraint value all state-action pairs can be divided into a feasible set and an infeasible set, that is, when , it indicates that this state-action pair does not meet the expected constraint and is divided into the infeasible set. When , it indicates that this state-action pair meets the expected constraint and is divided into the feasible set. For different sets, the calculation method of the sampling weight is different. When , its sampling weight calculation formula is: where in the first calculation formula on the left, the numerator represents the action value variance, and the denominator represents the mean difference in the action value. The larger the action value variance, the greater the uncertainty, and the more sampling should be performed. The smaller the mean difference in the action value, the smaller the gap from the optimal action, and the greater the probability of being the optimal action. Therefore, β(s, a i ) is larger. By calculating the sampling weight, the agent can preferentially sample those state-action pairs with higher uncertainty to quickly reduce uncertainty and improve the accuracy of decision-making, which helps the agent make better decisions in a complex and changing environment.

[0111] S105: Calculate the sampling probability of each state-action pair according to the sampling weight of the agent in each state-action pair.

[0112] Specifically, the sampling probability can intuitively reflect the possibility that the agent selects a certain action during the iterative sampling process. The larger the sampling probability, the greater the possibility that the agent selects this action. In the embodiments of this specification, the sampling probability of each state-action pair is calculated using the following formula:

[0113]

[0114] where p(a i |s) represents the sampling probability of the action a in state s i , β(s, a i ) represents the sampling weight of the action a in state s i , and ∑ j β(s, a j ) represents the sum of the sampling weights of all actions in state s.

[0115] S106: Iteratively sample actions in the action space of the agent according to the sampling probabilities of each state-action pair until a preset iteration termination condition is reached, to obtain the optimal decision control strategy of the target intelligent vehicle system.

[0116] It can be understood that in each iteration, the agent selects and executes the action with the highest probability according to the current state and observes the result. Through continuous iteration, the agent gradually learns the optimal decision control strategy that can obtain the maximum long-term reward under the condition of satisfying the expected type constraints. The iteration process continues until a preset iteration termination condition is reached, such as reaching the maximum number of iterations, policy convergence, etc. Refer to Figure 4 As shown, each iteration process contains multiple time steps. After the agent calculates the sampling probabilities of each action in the state s at time step t t , it samples the action of the current time step according to the sampling probability. Then the agent interacts with the environment to obtain information such as the next environment state and reward, stores the obtained information as a sample in the sample pool, and updates the corresponding action value normal distribution model and action constraint value normal distribution model according to the above information. Then it is judged whether the iteration termination condition is reached. If so, the iteration ends. If not, the above step of sampling the action according to the sampling probability is repeated until the iteration termination condition is reached, and the optimal decision strategy is output.

[0117] To better illustrate how this specification performs sampling based on the means and variances provided by the action value normal distribution model and the action constraint value normal distribution model, and how the optimal strategy probability can be gradually obtained through this sampling form, Figure 5 shows a schematic diagram of the overall sampling logic. As Figure 5 shown, the agent interacts with the environment to obtain the state s of the current time step t , and according to s t , the action space A = {a1, a2,..., a i} in this state can be obtained. Then, according to the action value normal distribution model and the action constraint value normal distribution model, the corresponding action value mean and action constraint value mean are obtained. Using the action a i as the abscissa and the action value mean as the ordinate, a coordinate graph as shown in Figure 5 (a) is established. Using the action a i as the abscissa and the action constraint value mean as the ordinate, a coordinate graph as shown in Figure 5 (b) is established. It can be seen from Figure 5 (a) that: First, the b of the action a is the largest, which is the optimal action in the state s t . Second, the action a2 and the optimal action ab has the smallest mean difference in action value, and action a i-1 has the largest mean difference in action value from the optimal action a b . At the same time, it can be seen from (b) in Figure 5 that for action a i-1 and a i , the value is greater than η, indicating that actions a i-1 and a i do not meet the expected constraints. Therefore, they are classified into the infeasible set, while for actions a1, a2, and a b , the value is less than η, meeting the expected constraints. Therefore, they are classified into the feasible set.

[0118] Next, use equations (7) and (8) to calculate the sampling probability p(a t |s i ) for each action under state s t . Taking the sampling probability p(a i |s t ) as the abscissa and action a i as the ordinate, establish a graph as shown in (c) of Figure 5 . It can be seen from (c) in Figure 5 that the sampling probability of the optimal action a b is the largest, followed by action a i in the infeasible set. This is because the mean difference in action value between this action and the optimal action a b is not large, and the variance of the action value of this action is the largest, that is, the uncertainty is the largest. It can be speculated that at this time step, action a i does not meet the expected constraints, probably affected by environmental noise, sensor errors, etc. In this case, the agent has a stronger motivation to further explore this state-action pair to determine the probability of this action becoming the optimal action. In addition, it can also be seen from (c) in Figure 5 that although the mean difference in action value between action a2 and the optimal action a b is the smallest, its sampling probability is not large. This is because the variance of the action value of this action is the smallest, and the uncertainty is the smallest. Therefore, when the agent performs action sampling, the probability of sampling this action is relatively small. For action a i-1 , the mean difference in action value between this action and the optimal action a b is the largest, indicating that under state s t , the action value of this action is the lowest. Most importantly, the variance of the action value of this action is relatively small, indicating that the uncertainty is not high, and this action does not meet the expected constraints. Therefore, the sampling probability of this action is the smallest, and the possibility of sampling this action in the next iteration is the smallest. Thus, due to the optimal action a bhas the highest sampling probability. When the agent samples an action at each time step, the probability of sampling b is the highest. Through this sampling probability calculation method, the sampling probability of the optimal action at each time step is maximized, so that in iterative sampling, the probability of obtaining the optimal policy is gradually increased, improving the sampling efficiency.

[0119] Adopting the above technical solution, the decision control strategy generation method for the expectation-type constraint provided in this specification embodiment can comprehensively reflect the value and constraint conditions of the agent during the action process by modeling the decision control problem of the intelligent vehicle system into a constrained Markov decision model. The value function of the constrained Markov decision model is modeled as an action value normal distribution model and an action constraint value normal distribution model. The mean of the normal distribution model reflects the prediction performance of the sampling strategy, and its variance can reflect the strategy uncertainty information. Then, according to the mean and variance provided by the normal distribution model, the sampling probability of each state-action pair is determined, and the actions in the agent's action space are iteratively sampled according to the sampling probability, enabling the agent to learn valuable information faster, and updating the action value function and the action constraint value function with this information. Finally, the optimal decision control strategy of the target agent is obtained, which improves the overall sampling efficiency while taking into account the strategy performance and the expectation constraint, so as to find the optimal decision control strategy at a faster speed.

[0120] Based on the above-mentioned decision control strategy generation method for the expectation-type constraint, this specification embodiment also correspondingly provides a decision control strategy generation device for the expectation-type constraint. The described device may include a system (including a distributed system), software (application), module, component, server, client, etc. that use the method described in this specification embodiment and are combined with the necessary implementation hardware. Based on the same innovative concept, the devices in one or more embodiments provided in this specification embodiment are as described in the following embodiments. Since the implementation solutions for the device to solve problems are similar to those of the method, the implementation of the specific device in this specification embodiment can refer to the implementation of the foregoing method, and the repeated parts will not be elaborated. Hereinafter, the term "unit" or "module" may be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0121] Specifically, Figure 6 is a schematic diagram of the module structure of an embodiment of a decision control strategy generation device for the expectation-type constraint provided in this specification embodiment. Referring to Figure 6 shown, a decision control strategy generation device for the expectation-type constraint provided in this specification embodiment includes:

[0122] A constraint acquisition module 601, configured to use the target intelligent vehicle system as an agent and acquire the desired type of constraints of the agent;

[0123] A constraint Markov model construction module 602, configured to construct a constraint Markov model of the agent according to the desired type of constraints, where the constraint Markov model includes an action value function and an action constraint value function;

[0124] A normal distribution model modeling module 603, configured to model an action value normal distribution model and an action constraint value normal distribution model for each state-action pair according to the action value function and the action constraint value function;

[0125] A sampling weight calculation module 604, configured to determine the mean and variance of each state-action pair according to the action value normal distribution model and the action constraint value normal distribution model, and calculate the sampling weight of the agent in each state-action pair according to the mean and variance;

[0126] A sampling probability calculation module 605, configured to calculate the sampling probability of each state-action pair according to the sampling weight of the agent in each state-action pair;

[0127] An optimal policy output module 606, configured to iteratively sample actions in the action space of the agent according to the sampling probability of each state-action pair until a preset iteration termination condition is reached, and obtain the optimal decision control policy of the target intelligent vehicle system.

[0128] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method, and will not be elaborated here.

[0129] Refer to Figure 7As shown, based on the method for generating a decision control strategy for expected constraints described above, an embodiment of this specification further provides a computer device 702, on which the above method runs. The computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 702 may also include any memory 706 for storing any kind of information such as code, settings, data, etc. Non-limiting examples include any combination of any type of RAM, any type of ROM, flash devices, hard disks, optical discs, etc. More generally, any memory may use any technology to store information. Further, any memory may provide volatile or non-volatile retention of information. Further, any memory may represent a fixed or removable component of the computer device 702. In one case, when the processor 704 executes the associated instructions stored in any memory or combination of memories, the computer device 702 may perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as a hard disk drive mechanism, an optical disc drive mechanism, etc.

[0130] The computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via an input device 712) and for providing various outputs (via an output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), the input device 712, and the output device 714 may not be included, and it may only be a computer device in a network. The computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0131] The communication link 722 may be implemented in any way, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0132] Corresponding to the method as Figures 1 to 3 shown, an embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, it executes the steps of the above method.

[0133] The embodiments of this specification also provide a computer-readable instruction. When the processor executes the instruction, the program therein causes the processor to execute as Figures 1 to 3 shown in the method.

[0134] The embodiments of this specification also provide a computer program product, including at least one instruction or at least one segment of program. The at least one instruction or the at least one segment of program is loaded and executed by the processor to implement as Figures 1 to 3 shown in the method.

[0135] It should be understood that in the various embodiments of this specification, the magnitudes of the serial numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of this specification.

[0136] It should also be understood that in the embodiments of this specification, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this specification generally represents an "or" relationship between the associated objects before and after.

[0137] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this specification can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this specification.

[0138] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again.

[0139] In several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed couplings or direct couplings or communication connections between each other can be indirect couplings or communication connections through some interfaces, devices, or units, and can also be electrical, mechanical, or other forms of connection.

[0140] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiments of this specification.

[0141] In addition, each functional unit in the various embodiments of this specification can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0142] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, in essence, or the part that contributes to the prior art, or all or part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this specification. The foregoing storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs, and other media that can store program codes.

[0143] Specific embodiments are used in this specification to elaborate on the principles and implementation manners of this specification. The descriptions of the above embodiments are only used to help understand the methods and their core ideas of this specification; at the same time, for those of ordinary skill in the art, according to the ideas of this specification, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this specification.

Claims

1. A decision control strategy generation method for expected constraints, characterized in that: The method comprises: Taking the target intelligent vehicle system as an intelligent agent, obtaining the desired constraints of the intelligent agent; Constructing a constrained Markov model of the agent according to the expected constraint, wherein the constrained Markov model includes an action value function and an action constraint value function; According to the action value function and the action constraint value function, a normal distribution model of the action value and a normal distribution model of the action constraint value of each state-action pair are obtained by modeling; Determine the mean and variance of each state-action pair according to the action value normal distribution model and the action constraint value normal distribution model, and calculate the sampling weight of the agent in each state-action pair according to the mean and variance; Calculate the sampling probability of each state-action pair according to the sampling weight of the agent in each state-action pair; The actions in the action space of the agent are iteratively sampled according to the sampling probability of each state-action pair until a preset iteration termination condition is reached, thereby obtaining an optimal decision-making control strategy for the target intelligent vehicle system.

2. The method according to claim 1, characterized in that The step of constructing the constrained Markov model of the agent according to the expected constraint comprises: The constrained Markov model of the agent is constructed using the following formula: Among them, π represents the sampling strategy, s t represents the state at time step t, a t represents the action at time step t, γ t represents the discount factor, r(s t ,a t ) indicates state s t Next action a t The instant reward function value, d0(s) represents the initial state distribution, h(s t ,a t ) represents the constraint function, and η represents the expected constraint target value.

3. The method according to claim 1, characterized in that The method of modeling the action value normal distribution model and the action constraint value normal distribution model of each state-action pair according to the action value function and the action constraint value function includes: Input each state-action pair into a number of pre-trained neural network models to obtain a number of action value prediction values ​​and action constraint value prediction values ​​for each state-action pair; Calculating the corresponding action value mean and action value variance according to the several action value prediction values ​​of each state-action pair, and calculating the corresponding action constraint value mean and action constraint value variance according to the several action constraint value prediction values ​​of each state-action pair; The action value normal distribution model of each state-action pair is obtained by modeling the action value mean and action value variance of each state-action pair; The normal distribution model of the action constraint value of each state-action pair is obtained by modeling the mean and variance of the action constraint value of each state-action pair.

4. The method according to claim 1, characterized in that The action value function and the action constraint value function are determined using the following formula: Among them, Q π (s,a) represents the action value function of action a in state s, γ represents the discount factor, r(s,a) represents the immediate reward function value of action a in state s, represents the action constraint value function of action a under state s, P(s'|s,π(s)) represents the state transition probability of state s to s', h(s,a) represents the constraint function, Q π (s',π(s')) represents the action value function under strategy π, represents the action constraint value function under policy π.

5. The method according to claim 2, characterized in that: The step of calculating the sampling weight of each state-action pair of the agent according to the mean and variance includes: The action value mean and action value variance of each state-action pair are obtained according to the action value normal distribution model, and the action with the largest action value mean in each state is determined as the optimal action in that state; Calculate the mean difference between the action values ​​of the remaining actions and the optimal action in each state; Obtaining the action constraint value mean and action constraint value variance according to the action constraint value normal distribution model, and calculating the difference between the action constraint value mean of each state-action pair and the action constraint value mean of the expected constraint target value; According to the action value mean difference, action constraint value mean difference, action value variance and action constraint value variance of each state-action pair, the sampling weight of the agent in each state-action pair is calculated.

6. The method according to claim 5, characterized in that The sampling weight of the agent in each state-action pair is calculated based on the action value mean difference, action constraint value mean difference, action value variance and action constraint value variance of each state-action pair, including: The sampling weight of the agent in each state-action pair is calculated using the following formula: Among them, β(s,a i ) represents action a in state s i The sampling weight, σ 2 (s,a i ) represents action a in state s i The action value variance, δ(s,a i ) represents action a in state s i The action value difference, Indicates action a in state s i The variance of the action constraint value, δ H (s,a i ) represents action a in state s i The action constraint value difference, Indicates action a in state s i The action constraint value, η represents the expected constraint target value.

7. The method according to claim 1, characterized in that The step of calculating the sampling probability of each state-action pair according to the sampling weight of each state-action pair of the agent includes: The sampling probability of each state-action pair is calculated using the following formula: Among them, p(a i |s) indicates action a in state s i The sampling probability of β(s,a i ) represents action a in state s i The sampling weight, ∑ j β(s,a j ) represents the sum of the sampling weights of all actions in state s.

8. A decision control strategy generation device for expected constraints, characterized in that: The device comprises: A constraint acquisition module, used to take the target intelligent vehicle system as an intelligent agent and acquire the desired type constraints of the intelligent agent; A constrained Markov model construction module, used to construct a constrained Markov model of the agent according to the expected type constraint, wherein the constrained Markov model includes an action value function and an action constraint value function; A normal distribution model modeling module, used to model the action value normal distribution model and the action constraint value normal distribution model of each state-action pair according to the action value function and the action constraint value function; A sampling weight calculation module, used to determine the mean and variance of each state-action pair according to the action value normal distribution model and the action constraint value normal distribution model, and calculate the sampling weight of the agent in each state-action pair according to the mean and variance; A sampling probability calculation module, used to calculate the sampling probability of each state-action pair according to the sampling weight of the agent in each state-action pair; The optimal strategy output module is used to iteratively sample the actions in the action space of the intelligent agent according to the sampling probability of each state-action pair until a preset iteration termination condition is reached to obtain the optimal decision-making control strategy of the target intelligent vehicle system.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 7.

Images