A DRL-based end-to-end vehicle motion control method for unsignalized intersections

Abstract

A kind of DRL-based end-to-end vehicle motion control method of unsignalized intersection, to cope with in unsignalized intersection high dynamic random traffic flow, improve the driving safety of vehicle.The present application relates to the field of intelligent driving, including information screening module, DRL module and smoothing processing module.Information screening module obtains state information from environment and extracts as low-dimensional environment information and space-time interaction state, constitutes state space and outputs driving state.Actor network and critic network in DRL module respectively extract spatial interaction features and time features in driving state, and are spliced with low-dimensional environment information features.Actor network will the state features after splicing be sampled by Gaussian distribution reparameterization output vehicle control quantity.Smoothing processing module carries out smoothing processing to vehicle control quantity, and the action is transmitted to the automatic driving vehicle in environment to carry out motion control, while updating the state information of environment.

Description

Technical field:

[0001] The present invention belongs to the field of intelligent driving, and in particular to a DRL-based end-to-end vehicle motion control method for an unsignaled intersection. Background technology:

[0002] With the rapid development of computer science and communications technology, autonomous driving technology has become a cutting-edge research field. It has shown broad application prospects in improving traffic safety, optimizing traffic efficiency, and promoting the development of intelligent transportation, becoming an important technical path to solving traditional transportation problems.

[0003] Currently, autonomous driving has achieved commercial application in motion control for simple autonomous driving scenarios, such as ACC and AEB. However, autonomous driving still poses safety risks in complex traffic scenarios. This is particularly true at unsignalized intersections, a critical urban traffic scenario. Due to the lack of effective traffic lights, traffic flow is highly dynamic and random. In these situations, autonomous vehicles struggle to effectively handle the complex interactions between vehicles and are unable to complete motion control tasks while ensuring safety.

[0004] Reinforcement learning, a data-driven learning method, offers new possibilities for addressing the challenges of autonomous driving in complex traffic scenarios. Reinforcement learning doesn't rely on pre-programmed rules. Instead, it uses a reward mechanism to guide intelligent agents' learning, enabling vehicles to master optimal decisions in changing environments through a process of trial and error.

[0005] Currently, research on autonomous driving technology mainly revolves around modular and end-to-end approaches. The traditional modular approach decomposes autonomous driving tasks into multiple independent modules and transmits information through explicit rules. This approach has good interpretability and causality, which facilitates problem tracing and module replacement. However, due to the accumulation of errors in the information transmission process between modules, the overall performance of the system may be limited, especially when dealing with complex traffic environments, making it difficult to achieve sufficient flexibility and adaptability. The end-to-end approach directly maps sensor data into trajectories or control signals through neural networks, has strong adaptability and global optimization capabilities, and can show unique advantages when dealing with complex tasks in highly dynamic scenarios such as unsignaled intersections.

[0006] In the study of motion control tasks in unsignaled intersection scenarios, a variety of methods have been proposed, each with its own advantages but also limitations. Patent CN118155429A proposes a traffic adaptive control method based on multi-agent reinforcement learning, which introduces a priority experience replay mechanism and multi-agents, enhances the sample utilization efficiency in the learning process, and accelerates the convergence speed of the algorithm. However, this method does not take into account the complex interaction relationship between vehicles in unsignaled intersections, which reduces driving safety. Patent CN108932840A proposes a method for unmanned vehicle traffic at urban intersections based on reinforcement learning, which uses a camera method to collect continuous vehicle operation status information and position information, providing rich data support for decision-making and optimizing traffic strategies. However, the reinforcement learning neural network described in this method uses a simple fully connected layer, which has certain limitations in understanding the complex interaction relationship between vehicles in unsignaled intersection scenarios and is difficult to handle existing traffic conflicts. Although the above method can improve the efficiency of vehicle cooperative traffic in an unsignaled intersection environment, it is difficult to ensure the safety and stability of smart car driving. Summary of the invention:

[0007] In order to solve the problems existing in the above technical background, the present invention provides an end-to-end vehicle motion control method for unsignaled intersections based on DRL. The method adopts an end-to-end reinforcement learning algorithm, fully utilizes the end-to-end ability to directly perform global optimization from input to output, makes decisions in real time and efficiently, and realizes the acquisition of the optimal safe passage strategy in an unsignaled intersection environment. In addition, the method introduces the spatiotemporal interaction state based on graph structure data in the basic state space, and fully considers the complex interaction relationship between vehicles, so that the intelligent car has the ability to infer the behavioral intentions of traffic conflict objects and improves driving safety. At the same time, a control signal smoother is designed to prevent the safety hazards caused by the oscillation of the Actor network control signal.

[0008] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0009] This invention provides an end-to-end vehicle motion control method for unsignalized intersections based on DRL. First, an information screening module acquires state information from the environment, refines it into low-dimensional environmental information and spatiotemporal interaction states, forming a state space. The driving state is then output from the state space and transmitted to the DRL module. Second, the actor network in the DRL module receives the driving state, uses a graph attention layer to extract spatial interaction features from the driving state, and uses a multi-head attention layer to capture temporal features from the driving state. A fully connected layer combines the spatial interaction features, temporal features, and low-dimensional environmental information features, further mapping them to the mean and standard deviation of the action. The vehicle control variable is then output using Gaussian reparameterized sampling. After receiving the driving state, the critic network uses a graph convolutional layer and a long short-term memory layer to extract spatial interaction features and temporal features from the driving state, respectively. The fully connected layer combines the spatial interaction features, temporal features, and low-dimensional environmental information features. Finally, a smoothing module smoothes the vehicle control variable and transmits the smoothed vehicle control variable as an action to the autonomous vehicle in the environment for motion control, thereby updating the environment's state information. The critic network outputs the action state value based on the state characteristics of the driving state and the action output by the smoothing module.

[0010] The method comprises the following steps:

[0011] Step 1: Reinforcement learning model design:

[0012] Step 1.1, state space design:

[0013] The present invention designs a state space that contains spatiotemporal interaction states and low-dimensional environmental information. This enables the intelligent agent to obtain spatiotemporal interaction information between traffic participants and itself. The state space S is as follows:

[0014] S=(S st ,S ld ) (1)

[0015] Where S st is the space-time interaction state, S ld It is low-dimensional environmental information.

[0016] Step 1.2, action space design:

[0017] In this invention, the vehicle control amount u is used as the action space a, and the lateral control amount and longitudinal control amount in the vehicle control amount are limited to a threshold range to prevent the vehicle from performing overly dangerous actions. Therefore, the action space a is defined as formula (2):

[0018] a=u=[α,δ],α0≤α≤α1; δ0≤δ≤δ1 (2)

[0019] Where δ is the lateral control variable, α is the longitudinal control variable, and α0, α1, δ0, and δ1 are the set constraint thresholds.

[0020] Step 1.3, reward function design:

[0021] In this paper, for the intelligent vehicle motion control task at an unsignalized intersection, the present invention designs a reward function that comprehensively considers the safety and comfort of the intelligent vehicle, traffic efficiency, path-keeping ability, and road traffic regulations. Different rewards are given different levels of importance at different stages of the task. The reward function is mainly set as shown in Equations (3) to (5):

[0022]

[0023]

[0024]

[0025] Where r target is the target reward item, r track is the path deviation reward, r rule is the rule reward item, r v is the speed bonus, r safe is the safety reward item. ω1, ω2, ω3, ω4 and ω5 are the reward allocation weights, which have different values ​​in different mission stages. ego ,y ego is the horizontal and vertical coordinates of the vehicle, x end and y end are the horizontal and vertical coordinates of the vehicle's endpoint, b1, b2, b3, c1, c2, c3, and c4 are constants. path is the path deviation, n is the total number of steps, n step is the number of steps taken by the car. ego is the vehicle speed, v E is the expected speed, v0 is the initial speed of the ego vehicle, ttc is the collision time, t1 is the maximum reaction time, s0 is the relative distance between the ego vehicle and the traffic participants, s E is the expected safety distance. A,j and y A,j is the horizontal and vertical coordinates of the jth point on the predicted path, x path,j and y path,j The horizontal and vertical coordinates of the corresponding reference path point, and are the yaw angle of the jth point in the predicted path and the yaw angle of the corresponding reference path point, c Δ is a constant term.

[0026] Step 2: Construction of information screening module:

[0027] The information screening module extracts the spatiotemporal interaction state and low-dimensional environmental information from the state information obtained from the environment, which constitutes the state space, and outputs the driving state from the state space to the DRL module. The spatiotemporal interaction state includes spatial interaction information and time information. Among them, the spatial interaction information is obtained in the form of graph structure data. Specifically, the traffic participants at each moment are modeled as nodes in the graph structure data, which contain four eigenvalues, namely the horizontal coordinate, vertical coordinate, speed and yaw angle of the traffic participants, which are expressed as the node feature matrix X t The interaction between traffic participants is represented by the edges between nodes, which is represented as the adjacency matrix A; the node feature matrix X t , the adjacency matrix A is as shown in formula (6) and formula (7):

[0028]

[0029]

[0030] Where, and is the horizontal and vertical coordinates of the vehicle at time t, is the vehicle speed at time t, is the yaw angle of the vehicle at time t. and is the horizontal and vertical coordinates of the first cycle at time t, is the speed of the first cycle car at time t, is the heading angle at time t of the first circle. and is the horizontal and vertical coordinates of the nth cycle at time t, is the speed of the nth weekly train at time t, is the heading angle of the nth vehicle at time t. The adjacency matrix A is an n×n matrix, where n represents the number of vehicles participating in the interaction. If there is an interaction between nodes i and j, the value of the element in the matrix is ​​1, otherwise it is 0.

[0031] Secondly, the prediction equation is established with reference to the kinematic equation to obtain the time series of the motion state and determine the time information in the space-time interaction state;

[0032] In summary, the spatiotemporal interaction state S is determined based on the spatial interaction information and time information. st , expressed as formula (8):

[0033] S st =((X t-j ;A)…(X t ; A)(X t+k ;A)) (8)

[0034] Where, X t-j、X t and X t+k are the node feature matrices at historical moments j, moment t, and future moments k, respectively. A is the adjacency matrix representing the interaction between the ego vehicle and surrounding vehicles.

[0035] After obtaining the spatiotemporal interaction state, the low-dimensional environment information S is extracted by combining the relative distance between the ego vehicle and the surrounding vehicles and the starting and ending points of the task. ld , as shown in formula (9):

[0036] S ld =(l1…l n , x start ,y start , x end ,y end ) (9)

[0037] Where l1…l n Represents the relative distance between the vehicle and each surrounding vehicle, a total of n. start and y start is the starting position coordinate of the task, x end and y end The coordinates of the task's end point.

[0038] Finally, the spatiotemporal interaction state S st and low-dimensional environment information S ld Integrate into the state space S, as shown in formula (10):

[0039] S=(S st ,S ld ) (10)

[0040] Step 3: Construction of DRL module:

[0041] The DRL module includes an actor network and a critic network. The actor network outputs vehicle control variables based on the driving state. The critic network outputs an action state value based on the driving state characteristics and the action output by the smoothing module. The actor network updates its parameters based on the action state value output by the critic network, while the critic network updates its parameters based on rewards fed back by the environment.

[0042] The Actor network in the DRL module includes a graph attention layer, a multi-head attention layer, and a fully connected layer. The graph attention layer calculates the attention weights between nodes, assigning different weights to adjacent nodes, and then extracting the spatial interaction features of the driving state. The attention weight calculation is as shown in formula (11):

[0043]

[0044] Where aij For the i-th node h i and the jth adjacent node h j The attention weight between them, LeakyReLU is the activation function, a T is the attention vector, ω i and ω j is the linear transformation parameter matrix, and || represents vector concatenation. h' i is the vehicle node feature after attention weight update, and ω is the weight matrix.

[0045] The multi-head attention layer in the actor network receives the spatial interaction features extracted by the graph attention layer, arranges the spatial interaction features in chronological order to form a temporal feature sequence, and performs a linear transformation on the temporal feature sequence to construct a query Q, key K, and value V matrix. Subsequently, the attention weights are calculated in parallel by multiple attention heads, and the attention weights calculated by each attention head are integrated to obtain the final multi-head attention output vector, thereby capturing the temporal characteristics of the driving state. The linear transformation, attention weight calculation, and information integration are shown in Equation (12):

[0046]

[0047] Where Q, K and V are query, key and value matrices respectively, ω Q 、ω K 、ω V and ω o is the weight matrix, x is the time feature sequence, Attention(Q,K,V) is the calculation of a single attention head, softmax is the activation function, QK T represents the dot product of Q and K, d k is the dimension of the key vector K. MultiHead(Q,K,V) is a multi-head attention mechanism, head h For the hth attention head, Concat(head1,head2,...,head h ) represents the concatenation of the outputs of multiple attention heads.

[0048] The fully connected layer in the Actor network concatenates the spatial interaction features output by the graph attention layer, the temporal features output by the multi-head attention layer, and the low-dimensional environmental information features in the driving state to construct state features. The state features are mapped to the mean and standard deviation of the action, and then the vehicle control amount is output through Gaussian distribution reparameterization sampling.

[0049] The Critic network in the DRL module includes a graph convolution layer, a long short-term memory layer, and a fully connected layer. The graph convolution layer is added to the identity matrix I through the adjacency matrix A. N Constructing a self-connected adjacency matrix Secondly, through the graph convolution operation, using the weight matrix W (l) For the node feature matrix H of layer l (l) Perform a linear transformation and combine it with the self-connected adjacency matrix Calculation is performed to realize the propagation and fusion of neighbor node information. Finally, the node feature matrix H of the l-th layer is made (l) Update to the l+1 layer node feature matrix H containing the features of itself and neighboring nodes (l+1) , thereby extracting the spatial interaction features of the driving state. This process is formalized as formula (13):

[0050]

[0051] Where, To add the self-connected adjacency matrix, A is the adjacency matrix, I N is the identity matrix, H (l) W (l) is a linear transformation process, is the process of propagation and fusion of neighbor node information, σ(·) is the process of introducing nonlinearity into the node feature matrix, and H (l+1) and H (l) is the node feature matrix of the l+1th and lth layers, σ is the activation function, W (l) is the weight matrix.

[0052] The long short-term memory layer in the critic network receives the spatial interaction features extracted by the graph convolution layer and arranges the spatial interaction features in chronological order to form a temporal feature sequence. This sequence is combined with the forget gate f at each time step through the gating mechanism of the long short-term memory network. t , input gate i t and output gate o t , jointly control the cell state c at the current time step t and hidden state h t The update of , thus achieving selective retention and integration of information. The hidden state h of each time step is output t It can be used as the time feature of this time step. Finally, in order to simplify the representation and aggregate historical information, the hidden state of the last time step in the sequence is selected as the time feature of the driving state. The state update process in the long short-term memory network is as shown in formula (14):

[0053]

[0054] Where h t-1 is the hidden state of the previous time step, x t is the time feature sequence, σ is the activation function, tanh represents the hyperbolic tangent function, ω f 、ω i 、ω cand ω k is a weight matrix, b f , b i , b c and b o is a bias vector. c t-1 is the cell state of the previous time step, f t is the forget gate, i t is the input gate, is the candidate cell state, c t is the cell state of the current time step, o t is the output gate, h t denotes the hidden state of the current time step.

[0055] The full connection layer in the Critic network splices the spatial interaction features output by the graph convolution layer, the time features output by the long short-term memory layer, and the low-dimensional environment information features in the driving state to determine the state features. After the smoothing processing module transmits the action to the Critic network, the action is evaluated in combination with the state features of the driving state, and the action state value is output.

[0056] Step 4, construction of the smoothing processing module:

[0057] The smoothing processing module uses the moving average method to smooth the vehicle control quantity output by the DRL module, and transmits the smoothed action to the autonomous vehicle in the environment for motion control and updates the state information of the environment. At the same time, it is transmitted to the Critic network in the DRL module, so that it evaluates the action according to the state features of the driving state, and outputs the action state value. The smoothing processing by the moving average method is as formula (15):

[0058]

[0059] In the formula, and are new longitudinal and lateral control quantities, α t + α t-1 + … + α t-n+1 and δ t + δ t-1 + … + δ t-n+1 are the longitudinal and lateral control quantities of the Actor network at the past n time points.

[0060] Step 5, network parameter update in the DRL module:

[0061] Step 5.1, reinforcement learning algorithm selection:

[0062] In order to extract the spatiotemporal interaction features of driving status in a neural network and complete the motion control task of an intelligent vehicle in an unsignaled intersection environment, the present invention uses the Soft Actor-Critic algorithm as a reinforcement learning algorithm for intelligent vehicles.

[0063] Step 5.2, experience sample extraction:

[0064] After a cycle, the Actor and Critic networks in the DRL module need to update their network parameters to optimize the policy. The Actor network is updated based on the action state value output by the Critic network, while the Critic network is updated based on the reward feedback from the environment after the selected action. This update involves randomly sampling a batch of experience samples from the experience pool and distributing them to each network.

[0065] Step 5.3, experience playback:

[0066] The four-tuple [s, a, r, s _ ] is split into state s, action a, reward r and next state s. _ The vector is transmitted to the Actor network and the Critic network for experience replay. The loss function Loss of the Actor network and the Critic network is calculated according to the four-tuple, and the parameters are updated by gradient descent back propagation, so as to learn a better safety control strategy after the update.

[0067] The beneficial effects of the present invention are as follows: the present invention relates to a signal-free end-to-end vehicle motion control method based on DRL, which includes an information screening module, a DRL module, and a smoothing processing module. The information screening module effectively converts the complex interaction relationship between the self-vehicle and surrounding vehicles in the environment into a spatiotemporal interaction state and low-dimensional environmental information expression. The Actor network in the DRL module uses a graph attention layer and a multi-head attention layer, and the Critic network uses a graph convolution layer and a long short-term memory layer to fully extract the feature information of the driving state, extract the spatiotemporal features, and give the network the ability to analyze the spatiotemporal interaction features of the scene in real time. Finally, the smoothing processing module smoothes the decided action and transmits it back to the autonomous driving vehicle in the environment for motion control, and updates the state information of the environment. This method enables the intelligent car to accurately identify the complex traffic conflict relationship between the self-vehicle and surrounding vehicles, and between surrounding vehicles in a signal-free intersection scene with complex traffic flow, and make the current optimal strategy selection to complete the motion control task while ensuring safety, thereby improving driving safety. Description of the drawings:

[0068] Figure 1 It is a flow chart of the method of the present invention.

[0069] Figure 2 is the application scenario of the present application.

[0070] Figure 3 is the information filtering module diagram of the present application. DETAILED DESCRIPTION

[0071] The present application will be described in detail below with reference to the accompanying drawings.

[0072] The present application is a DRL-based end-to-end vehicle motion control method for a signalized intersection. First, an information filtering module obtains state information from the environment, refines it into low-dimensional environment information considering the relative distance between the ego vehicle and surrounding vehicles and the start and end points of the task, and spatiotemporal interaction state considering the spatiotemporal interaction information between the ego vehicle and surrounding vehicles in the form of a graph structure, forms a state space, and outputs the driving state from the state space to the DRL module. Second, the Actor network in the DRL module receives the driving state, extracts the spatial interaction features in the driving state using a graph attention layer, and captures the temporal features in the driving state through a multi-head attention layer. The spatial interaction features, temporal features, and low-dimensional environment information features are concatenated using a fully connected layer, and the concatenated state features are further mapped to the mean and standard deviation of the action. The vehicle control quantity is output by reparameterization sampling of the Gaussian distribution. After receiving the driving state, the Critic network extracts the spatial interaction features and temporal features in the driving state using a graph convolution layer and a long short-term memory layer, respectively, and concatenates the spatial interaction features, temporal features, and low-dimensional environment information features through a fully connected layer. Finally, the smoothing processing module smoothes the vehicle control quantity, which is then transmitted to the autonomous vehicle in the environment as the action for motion control and updates the state information of the environment. The Critic network outputs the action state value according to the state features of the driving state and the action output by the smoothing processing module. Referring to the schematic of Figure 1 , the specific steps include the following:

[0073] Step 1, construction of the environment:

[0074] The environment of the present application selects a two-way four-lane intersection position of a map in CARLA, and the vehicle learns to turn left. As shown in Figure 2 , the solid line in the figure is the reference trajectory of the ego vehicle, and the dashed line is the reference trajectory of other traffic participants. The initial speed of the ego vehicle v0 is 15 km / h, and the initial speed of other traffic participants is 15-18 km / h.

[0075] Step 2, reinforcement learning model design:

[0076] Step 2.1, state space design:

[0077] At unsignaled intersections, the behavioral intentions of traffic participants from different directions are complex and changeable. Smart cars may interact with multiple traffic conflict objects simultaneously. Therefore, it is necessary to extract the state information of traffic conflict objects from the complex traffic environment and comprehensively consider the behavioral intentions of all traffic conflict objects to dynamically adjust the motion control strategy to ensure traffic safety. Therefore, the present invention designs a state space containing spatiotemporal interaction states, which enables the intelligent agent to obtain the spatiotemporal interaction information between traffic participants and itself. In addition, a low-dimensional environmental information is set in it, which considers the Euclidean distance between the vehicle and surrounding vehicles and the starting and ending points of motion control to supplement the environmental state. The state space is as shown in Equation (16):

[0078] S=(S st ,S ld ) (16)

[0079] This state space contains high-dimensional spatiotemporal interaction states S st and low-dimensional environment information S ld .

[0080] Step 2.2, action space design:

[0081] The motion space design of the smart car in this invention refers to the performance parameters of the Tesla Model 3 standard version, and the acceleration range is about -12m / s 2 ~4.5m / s 2 , the front wheel angle range is approximately -0.6 rad to 0.6 rad. The performance parameter range of the Tesla Model 3 standard version is mapped to the vehicle control variable u, and u is used as the action space a. The lateral control variable and the longitudinal control variable in the vehicle control variable are limited to a threshold range to prevent the vehicle from performing overly dangerous actions. Therefore, the action space in reinforcement learning is defined as Equation (17):

[0082] a=u=[α,δ],α0≤α≤α1; δ0≤δ≤δ1 (17)

[0083] Here, α represents the longitudinal control variable, which controls the ratio of accelerator and brake pedal application. δ is the lateral control variable, corresponding to the normalized control ratio of the front wheel steering angle. Both control variables are subject to upper and lower bounds. α0 and α1 are the minimum and maximum values ​​for the longitudinal control, respectively, and range from [-1, 1]. δ0 and δ1 are the minimum and maximum values ​​for the lateral control, also ranging from [-1, 1].

[0084] Step 2.3, reward function design:

[0085] In order to solve the intelligent vehicle motion control task at the unsignalized intersection, the reward function is designed to consider the safety and comfort of the intelligent vehicle, the traffic efficiency, the path keeping ability and the road traffic rules. And in different stages of the task, different rewards are given different attention. Therefore, a variable weight reward function is set to give more weight to the reward that has greater impact on the stage in different stages of the task. According to the environment in step 1, the main setting of the reward function is as formula (18):

[0086] R = ω1r target + ω2r track + ω3r rule + ω4r v + ω5r safe (18)

[0087] Wherein, r target is the target reward item, r track is the path deviation reward item, r rule is the rule reward item, r v is the speed reward item, and r safe is the safety reward item.

[0088] From the perspective of vehicle interaction, the intelligent vehicle motion control task can be divided into three stages: before interaction, during interaction and after interaction. In the pre-interaction stage, the intelligent vehicle does not need to pay too much attention to the behavior of the surrounding vehicles, and its primary task is the motion control of driving to the target position. During this process, the intelligent vehicle should try to keep the vehicle trajectory consistent with the expected path, and the speed of the intelligent vehicle should also be maintained at the expected speed. At this time, the reward function should allocate similar weights to r target , r track , r v , so ω1 = 0.3, ω2 = 0.4, ω3 = 1, ω4 = 0.3, ω5 = 0. In the interaction stage, the intelligent vehicle should prioritize driving safety, so it needs to pay more attention to r v , r safe , so ω1 = 0, ω2 = 0, ω3 = 1, ω4 = 0.5, ω5 = 0.5. In the post-interaction stage, the intelligent vehicle needs to speed up in the shortest time and return to the expected path faster, so the greatest weight is given to r v , and r target , r track are allocated the same weight, so ω1 = 0.3, ω2 = 0.3, ω3 = 1, ω4 = 0.4, ω5 = 0.

[0089] r target The expected position of the intelligent vehicle is constantly approaching the end position, so the deviation of the horizontal and vertical coordinates of the position of the intelligent vehicle and the end position is set as the deviation item of the intelligent vehicle and the end position. r targetThe specific expression of is as follows:

[0090] r target =b1-c1(|x ego -x end |+|y ego -y end |) (19)

[0091] Among them, x ego ,y ego is the vehicle coordinate, x end and y end are the end point coordinates, b1 and c1 are constants.

[0092] r v The purpose is to minimize the deviation between the vehicle speed and the desired speed, which is determined by the scenario risk coefficient ε. The specific expressions are as follows:

[0093] r v =b2-c2|v ego -v E | (20)

[0094]

[0095] Among them, v ego is the vehicle speed, v E is the desired velocity, v0 is the initial velocity of the ego vehicle, b2 and c2 are constant terms, ttc is the collision time, and t1 is the maximum reaction time, which is set to 2.6 seconds.

[0096] r safe is a safety reward item, which is used to guide the smart car to keep a relatively safe distance from the traffic conflict object as much as possible. The specific expression is as follows:

[0097] r safe =b3-c3(s0-s E ) (twenty two)

[0098] Among them, s0 is the relative distance between the vehicle and traffic participants, s E is the desired safety distance. b3 and c4 are constants.

[0099] r track is the path deviation reward item. The purpose of this reward is to minimize the deviation between the future trajectory of the smart car and the reference trajectory. The specific expressions are as follows:

[0100]

[0101] r track =b2-c4(Δ path) (twenty four)

[0102] Among them, Δ path is the path deviation, x A,j and y A,j is the horizontal and vertical coordinates of the jth point on the predicted path, x path,j and y path,j The horizontal and vertical coordinates of the corresponding reference path point, and are the yaw angle of the jth point in the predicted path and the yaw angle of the corresponding reference path point, b2, c4 and c Δ is a constant term. The reference path is a series of coordinate points and the desired yaw angle corresponding to each horizontal and vertical coordinate. The ego vehicle reference path is calculated as follows:

[0103]

[0104] Among them, v x , v y is the component of the current vehicle velocity in the x and y coordinates; a x , a y It is the component of the current vehicle acceleration in the x and y coordinates.

[0105] r rule is a rule reward item. It is triggered when the smart car violates traffic rules such as collision, driving against traffic, or driving out of the lane. The more steps the smart car takes when the event occurs, the greater the reward. The specific expression is as follows:

[0106] r rule =-(nn step ) (26)

[0107] Among them, n is the total number of steps, set to 400, n step is the number of steps taken by the car.

[0108] Step 3: Construction of information screening module:

[0109] The information screening module extracts the spatiotemporal interaction state and low-dimensional environmental information from the state information obtained from the environment to form a state space, and outputs the driving state from the state space to the DRL module. The spatiotemporal interaction state includes spatial interaction information and time information. Among them, the spatial interaction information is obtained in the form of graph structure data. Specifically, the traffic participants at each moment are modeled as nodes in the graph structure data, which contain four eigenvalues, namely the horizontal coordinate, the vertical coordinate, the speed and the yaw angle, which are expressed as the node feature matrix X t The interaction between traffic participants is represented by the edges between nodes, which is represented as the adjacency matrix A. Then the node feature matrix X tAnd the adjacency matrix A is as shown in Equation (27) and Equation (28):

[0110]

[0111]

[0112] in, and is the coordinate of the vehicle at time t, is the vehicle speed at time t, is the yaw angle of the vehicle at time t. and is the coordinate of the first cycle at time t, is the speed of the first cycle car at time t, is the heading angle at time t of the first circle. and is the coordinate of the nth weekly vehicle at time t, is the speed of the nth weekly train at time t, is the heading angle of the nth vehicle at time t. The adjacency matrix A is an n×n matrix, where n represents the number of vehicles participating in the interaction and is set to 5. If there is an interaction relationship between node i and node j, then the element A in the matrix ij is 1 if the value is set, otherwise it is 0.

[0113] Secondly, the prediction equation is established with reference to the kinematic equation to obtain the time series of the motion state and determine the time information in the space-time interaction state;

[0114] In summary, the spatiotemporal interaction state S is determined based on the spatial interaction information and time information. st , expressed as formula (29):

[0115] S st =((X t-j ; A) … (X t ; A) … (X t+k ;A)) (29)

[0116] Where, X t-j 、X t and X t+k are the node feature matrices at historical moments j, moment t, and future moments k respectively; A is the adjacency matrix representing the interaction relationship between the vehicle and surrounding vehicles.

[0117] After obtaining the spatiotemporal interaction state, the low-dimensional environment information S is extracted by combining the relative distance between the ego vehicle and the surrounding vehicles and the starting and ending points of the task. ld , as shown in formula (30):

[0118] S ld =(l1…l n , xstart ,y start , x end ,y end ) (30)

[0119] Where l1…l n Represents the relative distance between the vehicle and each surrounding vehicle, a total of n, n is set to 4. start and y start is the starting position coordinate of the task, x end and y end The coordinates of the task's end point.

[0120] Finally, the spatiotemporal interaction state S st and low-dimensional environment information S ld Integrate into the state space S, as shown in formula (31):

[0121] S=(S st ,S ld ) (31)

[0122] Step 4: Construction of DRL module:

[0123] The DRL module includes an actor network and a critic network. The actor network outputs vehicle control variables based on the driving state. The critic network outputs action state values ​​based on the driving state characteristics and the actions output by the smoothing module. The actor network updates its parameters based on the action state values ​​output by the critic network, while the critic network updates its parameters based on rewards fed back by the environment.

[0124] The Actor network in the DRL module includes a graph attention layer, a multi-head attention layer, and a fully connected layer. The graph attention layer calculates the attention weights between nodes, assigns different weights to adjacent nodes, and then extracts the spatial interaction features of the driving state. The attention weight calculation is as shown in Equation (32):

[0125]

[0126] Among them, a ij For the i-th node h i and the jth adjacent node h j The attention weight between them, LeakyReLU is the activation function, a T is the attention vector, ω i and ω j is the linear transformation parameter matrix, and || represents vector concatenation. h' i is the vehicle node feature after attention weight update, and ω is the weight matrix.

[0127] The multi-head attention layer in the actor network receives the spatial interaction features extracted by the graph attention layer, arranges the spatial interaction features in chronological order to form a temporal feature sequence, and constructs the query (Q), key (K), and value (V) matrices by performing a linear transformation on the temporal feature sequence. Subsequently, the attention weights are calculated in parallel by multiple attention heads, and the attention weights calculated by each attention head are integrated to obtain the final multi-head attention output vector, thereby capturing the temporal characteristics of the driving state. The linear transformation, attention weight calculation, and information integration are shown in Equation (33):

[0128]

[0129] Where Q, K and V are query, key and value matrices respectively, ω Q 、ω K 、ω V and ω o is the weight matrix, x is the time feature sequence, Attention(Q,K,V) is the calculation of a single attention head, softmax is the activation function, QK T represents the dot product of Q and K, d k is the dimension of the key vector K. MultiHead(Q,K,V) is a multi-head attention mechanism, head h For the hth attention head, Concat(head1,head2,...,head h ) represents the concatenation of the outputs of multiple attention heads.

[0130] The fully connected layer in the Actor network concatenates the spatial interaction features output by the graph attention layer, the temporal features output by the multi-head attention layer, and the low-dimensional environmental information features in the driving state to construct state features. The state features are mapped to the mean and standard deviation of the action, and then the vehicle control amount is output through Gaussian distribution reparameterization sampling.

[0131] In the DRL module, the Critic network has relatively lower requirements for spatiotemporal feature extraction capabilities than the Actor network. In order to save computing power and avoid unnecessary waste, the Critic network uses graph convolution layers and long short-term memory layers to extract the spatial interaction features and temporal features of the driving state.

[0132] The Critic network in the DRL module includes a graph convolution layer, a long short-term memory layer, and a fully connected layer. The graph convolution layer is added to the identity matrix I through the adjacency matrix A. N Constructing a self-connected adjacency matrix Secondly, through the graph convolution operation, using the weight matrix W (l) For the node feature matrix H of layer l (l) Perform a linear transformation and combine it with the self-connected adjacency matrix Calculation is performed to realize the propagation and fusion of neighbor node information. Finally, the node feature matrix H of the l-th layer is made (l) Update to the l+1 layer node feature matrix H containing the features of itself and neighboring nodes (l+1) , thereby extracting the spatial interaction features of the driving state. This process is formalized as formula (34):

[0133]

[0134] in, To add the self-connected adjacency matrix, A is the adjacency matrix, I N is the identity matrix, H (l) W (l) is a linear transformation process, is the process of propagation and fusion of neighbor node information, σ(·) is the process of introducing nonlinearity into the node feature matrix, and H (l+1) and H (l) is the node feature matrix of the l+1th and lth layers, σ is the activation function, W (l) is the weight matrix.

[0135] The long short-term memory layer in the critic network receives the spatial interaction features extracted by the graph convolution layer and arranges the spatial interaction features in chronological order to form a temporal feature sequence. This sequence is combined with the forget gate f at each time step through the gating mechanism of the long short-term memory network. t , input gate i t and output gate o t , jointly control the cell state c at the current time step t and hidden state h t The update of , thus achieving selective retention and integration of information. The hidden state h of each time step is output t This can be used as the temporal feature for that time step. Finally, to simplify the representation and aggregate historical information, the hidden state at the last time step in the sequence is selected as the temporal feature of the driving state. The specific state update process of the LSTM network is carried out as follows.

[0136] First, the time feature sequence x at time t is t and the previous hidden state h t-1 Splice them into a sequence and calculate the degree of forgetting of information in the forget gate, as shown in formula (35):

[0137] f t =σ(ω f [h t-1 ,x t ]+b f ) (35)

[0138] Among them, ft is the forget gate, σ is the activation function, ω f is the weight matrix, h t-1 is the hidden state of the previous time step, x t is the time characteristic series, b f is the bias vector.

[0139] Secondly, the information update degree is calculated in the input gate to generate candidate information values, as shown in formula (36):

[0140]

[0141] Among them, i t is the input gate, is the candidate cell state, σ is the activation function, tanh represents the hyperbolic tangent function, h t-1 is the hidden state of the previous time step, x t is the time characteristic series, b i 、b c is the bias vector, ω i 、ω c is the weight matrix.

[0142] Update the cell state according to the degree of information forgetting and the degree of information updating, as shown in formula (37):

[0143]

[0144] Among them, c t is the cell state at the current time step, f t is the forget gate, c t-1 is the cell state at the previous time step, i t is the input gate, is the candidate cell state.

[0145] Finally, the output gate outputs the hidden state according to the cell state, as shown in formula (38):

[0146]

[0147] Among them, t is the output gate, h t Represents the hidden state of the current time step. h t-1 is the hidden state of the previous time step, x t is the time feature sequence, σ is the activation function, tanh represents the hyperbolic tangent function, ω k is the weight matrix, b o is the bias vector, c t is the cell state at the current time step.

[0148] The fully connected layer in the critic network combines the spatial interaction features output by the graph convolutional layer, the temporal features output by the long short-term memory layer, and the low-dimensional environmental information features of the driving state to determine the state characteristics. After the smoothing module passes the action to the critic network, it evaluates the action based on the state characteristics of the driving state and outputs the action state value.

[0149] Step 5: Construction of smoothing processing module:

[0150] The smoothing module uses a sliding average method to smooth the vehicle control quantity output by the DRL module and transmits the smoothed action to the autonomous driving vehicle in the environment for motion control and updates the state information of the environment. At the same time, it is transmitted to the Critic network in the DRL module, which evaluates the action based on the state characteristics of the driving state and outputs the action state value. The sliding average method is shown in Equation (39):

[0151]

[0152] in, and are the new longitudinal and lateral control quantities, α t +α t-1 +…+α t-n+1 and δ t +δ t-1 +…+δ t-n+1 The vertical and horizontal control values ​​of the Actor network in the past n moments, where n is set to 10.

[0153] Step 6: Update network parameters in the DRL module:

[0154] Step 6.1, reinforcement learning algorithm selection:

[0155] To extract spatiotemporal interaction features of driving states within a neural network and complete motion control tasks for intelligent vehicles in unsignaled intersections, this paper employs the Soft Actor-Critic algorithm as a reinforcement learning algorithm for intelligent vehicles. The Soft Actor-Critic algorithm adapts to continuous action spaces and is highly sample-efficient. By maximizing entropy and expected reward, the learned strategies are more robust and random, helping to improve the generalizability of control strategies.

[0156] Step 6.2, experience sample extraction:

[0157] After one cycle, the actor and critic networks in the DRL module update their network parameters to optimize the policy. The actor network updates based on the action state value output by the critic network, while the critic network updates based on the reward feedback from the environment after the selected action. This update requires randomly drawing a batch of experience samples from the experience pool and distributing them to each network.

[0158] Step 6.3, experience playback:

[0159] The four-tuple [s, a, r, s _ ] is split into state s, action a, reward r and next state s. _ The vector is transmitted to the Actor network and the Critic network for experience replay. The loss function Loss of each network is calculated through the quadruple, and the parameters are updated using the gradient descent method backpropagation to learn the updated safety control strategy and better complete the motion control task. The loss function of the Actor network, gradient descent, is as follows:

[0160]

[0161] φ←φ+η▽ φ J π (φ) (41)

[0162] Among them, J π (φ) is the loss function of the Actor network, φ is the network parameter; E is the expected value, and S~D means that the driving state S is sampled from the state distribution D. a~π φ Indicates that action a is based on strategy π φ Sampling is done from the driving state S. α is the entropy weight, log(π φ (a|s)) is the policy π φ The log probability of choosing action a under driving state S. It is the value function estimate of the Critic network, and the minimum value of the two Critic networks is taken as the estimate of the value function. represents the expected reward of taking action a in driving state S. η is the learning rate, ▽ φ is the gradient vector with respect to the parameter φ.

[0163] The loss function of the critic network, gradient descent, is as follows:

[0164]

[0165] θ←θ-α θ ▽θ J Q (θ) (43)

[0166] Among them, J Q (θ) is the loss function of the Critic network, θ is the network parameter; E is the expected value, (s,a)~D means that the driving state S and action a are sampled from the state distribution D. is the expected reward of taking action a in driving state S. r is the immediate reward, and γ is the discount factor used to balance the importance of immediate rewards and future rewards. To select the minimum Q value from the two Critic networks and reduce the estimated variance, θ' j is the parameter of the target Critic network. α is the entropy weight, logπ(a _ |s _ ) is the Actor network in the next driving state s _ Next, select the next action a _ The logarithmic probability of α θ is the learning rate of the Critic network, ▽ θ is the gradient vector with respect to the parameter θ.

[0167] In summary: The present invention proposes an end-to-end vehicle motion control method for unsignalized intersections based on DRL, which can effectively extract the spatiotemporal characteristics of driving status, and has the ability to infer the behavioral intentions of traffic conflict objects from complex traffic environments, process complex interactions between vehicles, and then dynamically adjust the motion control strategy to ensure traffic safety; it improves the generalization and safety of the motion control strategy in an unsignaled intersection environment.

Claims

1. A DRL-based end-to-end vehicle motion control method for an unsignalized intersection, characterized by: First, the information screening module obtains state information from the environment and refines it into low-dimensional environmental information and spatiotemporal interaction states to form a state space, and outputs the driving state from the state space to the DRL module; secondly, the Actor network in the DRL module receives the driving state, uses the graph attention layer to extract the spatial interaction features in the driving state, and uses the multi-head attention layer to capture the temporal features in the driving state. The fully connected layer is used to splice the spatial interaction features, temporal features and low-dimensional environmental information features, which are further mapped to the mean and standard deviation of the action, and the vehicle control amount is output through Gaussian distribution reparameterized sampling; after receiving the driving state, the Critic network uses the graph convolution layer and the long short-term memory layer to extract the spatial interaction features and temporal features in the driving state, and then splices the spatial interaction features, temporal features and low-dimensional environmental information features through the fully connected layer; finally, the smoothing processing module smoothes the vehicle control amount and passes the action to the autonomous driving vehicle in the environment for motion control, while updating the state information of the environment; the Critic network outputs the action state value based on the state characteristics of the driving state and the action output by the smoothing processing module; The spatiotemporal interaction state includes spatial interaction information and time information; the spatial interaction information is obtained based on the form of graph structure data, specifically as follows: the traffic participants at each moment are modeled as nodes in the graph structure data, which contain four eigenvalues. The first three items are the horizontal coordinate, vertical coordinate and speed of the traffic participants, respectively. The fourth item is the yaw angle for the ego vehicle and the heading angle for the surrounding vehicles, which is expressed as the node feature matrix X t The interaction between traffic participants is represented by the edges between nodes, which is represented as the adjacency matrix A; the node feature matrix X t , the adjacency matrix A is as shown in formula (1) and formula (2): in, and is the horizontal and vertical coordinates of the vehicle at time t, is the vehicle speed at time t, is the yaw angle of the vehicle at time t; and is the horizontal and vertical coordinates of the first cycle at time t, is the speed of the first cycle car at time t, is the heading angle at time t of the first circle; and is the horizontal and vertical coordinates of the nth cycle at time t, is the speed of the nth weekly train at time t, is the heading angle of the nth vehicle at time t; the adjacency matrix A is an n×n matrix, where n represents the number of vehicles participating in the interaction; if there is an interaction relationship between node i and node j, the element value in the matrix is ​​1, otherwise it is 0; Secondly, the prediction equation is established with reference to the kinematic equation to obtain the time series of the motion state and determine the time information in the space-time interaction state; In summary, the spatiotemporal interaction state S is determined based on the spatial interaction information and time information. st , expressed as formula (3): S st =((X t-j ;A)…(X t ;A)…(X t+k ;A)) (3) Among them, X t-j 、X t and X t+k are the node feature matrices at historical moments j, moment t, and future moments k, respectively; A is the adjacency matrix representing the interaction between the ego vehicle and surrounding vehicles; After obtaining the spatiotemporal interaction state, the low-dimensional environment information S is extracted by combining the relative distance between the self-vehicle and the surrounding vehicles and the starting and ending points of the task. ld , as shown in formula (4): S ld =(l1…l n ,x start ,and start ,x end ,and end ) (4) Among them, l1…l n Represents the relative distance between the vehicle and each surrounding vehicle, a total of n; x start and y start is the horizontal and vertical coordinates of the starting position of the task, x end and y end is the horizontal and vertical coordinates of the task end point; Finally, the spatiotemporal interaction state and low-dimensional environment information are integrated into the state space S, as shown in formula (5): S=(S st ,S ld ) (5) Among them, S st is the space-time interaction state, S ld It is low-dimensional environmental information.

2. The end-to-end vehicle motion control method at an unsignalized intersection based on a DRL as claimed in claim 1, characterized in that: The DRL module includes an Actor network and a Critic network. The Actor network outputs the vehicle control value based on the driving state. The Actor network updates its parameters based on the action state value output by the Critic network, while the Critic network updates its parameters based on the rewards fed back by the environment. The Actor network in the DRL module includes a graph attention layer, a multi-head attention layer, and a fully connected layer. The graph attention layer calculates the attention weights between nodes, assigns different weights to adjacent nodes, and extracts the spatial interaction features of the driving state. The attention weight is calculated as shown in formula (6): Among them, a ij For the i-th node h i and the jth adjacent node h j The attention weight between them, LeakyReLU is the activation function, a T is the attention vector, ω i and ω j is the linear transformation parameter matrix, || represents vector splicing; h i ' is the vehicle node feature after attention weight update, ω is the weight matrix; The multi-head attention layer in the actor network receives the spatial interaction features extracted by the graph attention layer, arranges the spatial interaction features in chronological order to form a temporal feature sequence, and performs a linear transformation on the temporal feature sequence to construct a query Q, key K, and value V matrix. Subsequently, the attention weights are calculated in parallel by multiple attention heads, and the attention weights calculated by each attention head are integrated to obtain the final multi-head attention output vector, thereby capturing the temporal characteristics of the driving state. The linear transformation, attention weight calculation, and information integration are shown in formula (7): Where Q, K and V are query, key and value matrices respectively, ω Q 、ω K 、ω V and ω o is the weight matrix, x is the time feature sequence, Attention(Q,K,V) is the calculation of a single attention head, softmax is the activation function, QK T represents the dot product of Q and K, d k is the dimension of the key vector K; MultiHead(Q,K,V) is the multi-head attention mechanism, head h For the hth attention head, Concat(head1,head2,…,head h ) represents the concatenation of the outputs of multiple attention heads; The fully connected layer in the actor network combines the spatial interaction features output by the graph attention layer, the temporal features output by the multi-head attention layer, and the low-dimensional environmental information features in the driving state to construct state features. The state features are then mapped to the mean and standard deviation of the action, and the vehicle control variable is then output through Gaussian distribution reparameterization sampling. The Critic network in the DRL module includes a graph convolution layer, a long short-term memory layer, and a fully connected layer; wherein the graph convolution layer is added to the unit matrix I through the adjacency matrix A N Constructing a self-connected adjacency matrix Secondly, through the graph convolution operation, using the weight matrix W (l) For the node feature matrix H of layer l (l) Perform a linear transformation and combine it with the self-connected adjacency matrix Calculation is performed to realize the propagation and fusion of neighbor node information. Finally, the node feature matrix H of the l-th layer is made (l) Update to the l+1 layer node feature matrix H containing the features of itself and neighboring nodes (l+1) , thereby extracting the spatial interaction features of the driving state. This process is formalized as formula (8): in, To add the self-connected adjacency matrix, I N is the identity matrix, H (l) W (l) is a linear transformation process, is the process of propagation and fusion of neighbor node information, σ(·) is the process of introducing nonlinearity into the node feature matrix, and H (l+1) and H (l) is the node feature matrix of the l+1th and lth layers, σ is the activation function, W (l) is the weight matrix; The long short-term memory layer in the critic network receives the spatial interaction features extracted by the graph convolution layer, arranges the spatial interaction features in chronological order, and forms a temporal feature sequence; this sequence passes through the gating mechanism of the long short-term memory network at each time step, combined with the forget gate f t , input gate i t and output gate o t , jointly control the cell state c at the current time step t and hidden state h t Update, thereby achieving selective retention and integration of information; the hidden state h of each time step output t can be used as the time feature of this time step; finally, in order to simplify the representation and aggregate historical information, the hidden state of the last time step in the sequence is selected as the time feature of the driving state; the state update process in the long short-term memory network is as shown in formula (9): Among them, h t-1 is the hidden state of the previous time step, x t is the time feature sequence, σ is the activation function, tanh represents the hyperbolic tangent function, ω f 、ω i 、ω c and ω k is the weight matrix, b f 、b i 、b c and b o is the bias vector; c t-1 is the cell state at the previous time step, f t For the forget gate, i t is the input gate, is the candidate cell state, c t is the cell state at the current time step, o t is the output gate, h t represents the hidden state of the current time step; The fully connected layer in the critic network combines the spatial interaction features output by the graph convolutional layer, the temporal features output by the long short-term memory layer, and the low-dimensional environmental information features of the driving state to determine the state features. After the smoothing module passes the action to the critic network, it evaluates the action based on the state features of the driving state and outputs the action state value. After one cycle, the Actor network updates its network parameters based on the action state value output by the Critic network, while the Critic network updates its network parameters based on the reward feedback from the environment. For the motion control task of the intelligent vehicle at an unsignaled intersection, the reward function is designed by comprehensively considering the safety and comfort, traffic efficiency, path keeping ability, and road traffic rules of the intelligent vehicle, as shown in Equations (10) to (12): Among them, r target is the target reward item, r track is the path deviation reward, r rule is the rule reward item, r v is the speed bonus, r safe is the safety reward item, ω1, ω2, ω3, ω4 and ω5 are the reward allocation weights; x ego ,y ego is the horizontal and vertical coordinates of the vehicle, x end and y end are the horizontal and vertical coordinates of the vehicle's endpoint, b1, b2, b3, c1, c2, c3, and c4 are constants; Δ path is the path deviation, n is the total number of steps, n step is the number of steps taken by the car, v ego is the vehicle speed, v E is the expected speed, v0 is the initial speed of the ego vehicle, ttc is the collision time, t1 is the maximum reaction time, s0 is the relative distance between the ego vehicle and the traffic participants, s E is the expected safety distance; x A,j and y A,j is the horizontal and vertical coordinates of the jth point on the predicted path, x path,j and y path,j The horizontal and vertical coordinates of the corresponding reference path point, and are the yaw angle of the jth point in the predicted path and the yaw angle of the corresponding reference path point, c Δ is a constant term; The smoothing processing module uses the sliding average method to smooth the vehicle control quantity output by the DRL module, and transmits the smoothed action to the autonomous driving vehicle in the environment for motion control and updates the state information of the environment; at the same time, it is transmitted to the Critic network in the DRL module so that it evaluates the action according to the state characteristics of the driving state and outputs the action state value; the sliding average method is used for smoothing, and the action space is as shown in Equations (13) and (14): a=u=[α,δ],α0≤α≤α1; δ0≤δ≤δ1 (14) in, and are the new longitudinal and lateral control quantities, α t +α t-1 +…+α t-n+1 and δ t +δ t-1 +…+δ t-n+1 are the longitudinal and lateral control quantities of the Actor network in the past n moments; a is the action space, u is the vehicle control quantity, where δ is the lateral control quantity; α is the longitudinal control quantity; α0, α1, δ0, δ1 are the set constraint thresholds.

Images