Control method and system of autonomous vehicle, electronic device and storage medium

Abstract

The application discloses a control method and system of an automatic driving vehicle, an electronic device and a storage medium, and belongs to the technical field of automatic driving. The control method comprises the following steps: acquiring an environment image of an area where a target vehicle is located, fusing a feature vector of the environment image with a vehicle state of the target vehicle to obtain a fusion state, judging whether the fusion state matches driving experience data, if yes, determining a first behavior action corresponding to the fusion state by using an experience pool model, and controlling the target vehicle to execute the first behavior action, and if not, determining a second behavior action corresponding to the fusion state by using a deep reinforcement learning model, and controlling the target vehicle to execute the second behavior action. The application can improve the reliability of automatic driving control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a control method and system of an automatic driving vehicle, an electronic device and a storage medium. BACKGROUND

[0002] An automatic driving vehicle integrates environment perception, behavior decision, planning and control technology, and is widely considered as an important way to effectively alleviate traffic congestion, reduce traffic accidents and improve fuel efficiency. In recent years, driven by the development of artificial intelligence, automatic driving technology has developed rapidly, and more and more automobile enterprises have begun to invest in the development of automatic driving vehicles. Although most of the current automatic driving vehicles have reached L3 level automatic driving, or even some have reached L3.5, there are still many challenges in realizing efficient automatic driving in complex traffic environments.

[0003] In an automatic driving vehicle, a decision module as a hub connecting environment perception and planning control can provide a series of driving behaviors for an automatic driving vehicle, so as to complete the driving task of the vehicle. In recent years, deep reinforcement learning has been gradually applied to the research of automatic driving. However, the existing automatic driving behavior decision method based on deep reinforcement learning can only be used in a set environment or similar conditions to the training environment.

[0004] Therefore, how to improve the reliability of automatic driving control is a technical problem to be solved by those skilled in the art at present. SUMMARY

[0005] The purpose of the present application is to provide a control method and system of an automatic driving vehicle, an electronic device and a storage medium, which can improve the reliability of automatic driving control.

[0006] To solve the above technical problems, the present application provides a control method of an automatic driving vehicle, which comprises:

[0007] An environment image of an area where a target vehicle is located is obtained, and a feature vector of the environment image is fused with a vehicle state of the target vehicle to obtain a fusion state;

[0008] It is judged whether the fusion state matches driving experience data;

[0009] If yes, a first behavior action corresponding to the fusion state is determined by using an experience pool model, and the target vehicle is controlled to execute the first behavior action;

[0010] If no, a second behavior action corresponding to the fusion state is determined by using a deep reinforcement learning model, and the target vehicle is controlled to execute the second behavior action.

[0011] The deep reinforcement learning model is obtained by learning interaction experience in a deep reinforcement learning training process; the experience pool model and the optimal reward function are used to assist the deep reinforcement learning model in deep reinforcement learning training; the experience pool model and the optimal reward function are obtained by inverse reinforcement learning on the driving experience data and reward function back-propagation.

[0012] Optionally, before the feature vector of the environment image is fused with the vehicle self-state of the target vehicle, the method further comprises:

[0013] locating a target in the environment image by using a perception model;

[0014] generating a two-dimensional grid map corresponding to the environment image according to the location result; wherein the value of a cell corresponding to the target in the two-dimensional grid map is 1, and the value of other cells is 0;

[0015] extracting features of the two-dimensional grid map by using a convolutional neural network to obtain a feature vector of the environment image.

[0016] Optionally, before the feature vector of the environment image is fused with the vehicle self-state of the target vehicle, the method further comprises:

[0017] obtaining the vehicle self-state by using a vehicle chassis sensor of the target vehicle, and normalizing the vehicle self-state; wherein the vehicle self-state comprises any one or a combination of a plurality of items selected from the group consisting of driving speed, steering wheel angle, accelerator opening degree, and brake opening degree.

[0018] Optionally, before the experience pool model is used to determine the first behavior action corresponding to the fused state, the method further comprises:

[0019] determining a generator and a discriminator, and setting a function corresponding to the discriminator as a reward function; the function corresponding to the discriminator is θ represents a discriminator parameter, s represents a fused state, a represents a behavior action, and f θ represents an auxiliary network parameterized by θ, π(a|s) represents a generator policy, and r seg represents a semantic reward;

[0020] performing inverse reinforcement learning and reward function back-propagation on the generator and the discriminator based on driving experience data, so as to correct the generator and the discriminator;

[0021] setting the corrected generator as the experience pool model;

[0022] setting a function corresponding to the corrected discriminator as an optimal reward function.

[0023] Optionally, the semantic reward includes any one or a combination of a collision semantic reward, a lane invasion semantic reward, and a lane change semantic reward.

[0024] Optionally, before determining the second behavior action corresponding to the fusion state by using the deep reinforcement learning model, the method further comprises:

[0025] building the deep reinforcement learning model; wherein the deep reinforcement learning model comprises an Actior network, a Critic network, and an experience pool;

[0026] training the deep reinforcement learning model from scratch by using the experience pool model and the optimal reward function;

[0027] distributing the combination data learned by the deep reinforcement learning model in a manner of quantile regression, and optimizing N combination data with the lowest scores in the deep reinforcement learning model according to an optimization formula; wherein the combination data is a corresponding relationship between a fusion state and a behavior action;

[0028] wherein the optimization formula is Q(s,a)=r(s,a)+γinf(Q(s′,a)); Q(s,a) represents the optimized combination data, r(s,a) represents a reward, γ represents a discount rate, inf represents a lower bound, and Q(s′,a) represents the combination data before optimization.

[0029] Optionally, training the deep reinforcement learning model from scratch by using the experience pool model and the optimal reward function comprises:

[0030] training the deep reinforcement learning model from scratch according to a behavior selection strategy by using the experience pool model and the optimal reward function;

[0031] wherein the behavior selection strategy is used to describe probabilities of selecting an experience pool model action, a random exploration action, and a deep reinforcement learning model action in the process of training the deep reinforcement learning model; the probability of selecting the experience pool model action is p1=α(1-t / T t ), α represents an initial probability of p1, t represents a current training frequency, T t represents an experience pool model training period; the probability of selecting the random exploration action is p2=β(1-p1), β represents a greed coefficient; and the probability of selecting the deep reinforcement learning model action is p3=(1-β)(1-p1).

[0032] The application also provides a control system of an autonomous vehicle, which comprises:

[0033] The state determination module is configured to acquire an environment image of an area where the target vehicle is located, and fuse a feature vector of the environment image with a vehicle state of the target vehicle to obtain a fusion state.

[0034] The experience matching module is configured to determine whether the fusion state matches driving experience data.

[0035] The first control module is configured to, if the fusion state matches the driving experience data, determine a first behavior action corresponding to the fusion state by using an experience pool model, and control the target vehicle to perform the first behavior action.

[0036] The second control module is configured to, if the fusion state does not match the driving experience data, determine a second behavior action corresponding to the fusion state by using a deep reinforcement learning model, and control the target vehicle to perform the second behavior action.

[0037] The deep reinforcement learning model is obtained by learning interaction experience in a deep reinforcement learning training process; the experience pool model and the optimal reward function are used to assist the deep reinforcement learning model in deep reinforcement learning training; and the experience pool model and the optimal reward function are obtained by inverse reinforcement learning on the driving experience data and reward function back-propagation.

[0038] The application further provides a storage medium having a computer program stored thereon, and the computer program implements the steps of the control method of the automatic driving vehicle when executed.

[0039] The application further provides an electronic device including a memory and a processor, and the memory has a computer program stored therein, and the processor implements the steps of the control method of the automatic driving vehicle when calling the computer program in the memory.

[0040] The application provides a control method of an autonomous vehicle, comprising the following steps: acquiring an environment image of a region where a target vehicle is located, and fusing a feature vector of the environment image with a vehicle self-state of the target vehicle to obtain a fusion state; judging whether the fusion state matches driving experience data; if yes, determining a first behavior action corresponding to the fusion state by using an experience pool model, and controlling the target vehicle to execute the first behavior action; if no, determining a second behavior action corresponding to the fusion state by using a deep reinforcement learning model, and controlling the target vehicle to execute the second behavior action; wherein the deep reinforcement learning model is obtained by learning interaction experience in a deep reinforcement learning training process; the experience pool model and an optimal reward function are used for assisting the deep reinforcement learning model to perform deep reinforcement learning training; and the experience pool model and the optimal reward function are obtained by performing inverse reinforcement learning on the driving experience data and back-propagating a reward function.

[0041] The application fuses the feature vector of the environment image with the vehicle self-state to obtain the fusion state, if the fusion state matches the driving experience data, the experience pool model is used to determine the first behavior action to be executed, if the fusion state does not match the driving experience data, the deep reinforcement learning model is used to determine the second behavior action to be executed. The experience pool model is obtained by inverse reinforcement learning and back-propagating the reward function, the experience pool model can assist environment exploration in the deep reinforcement learning training, and the back-propagated reward function enables the deep reinforcement learning model to learn how to output a high-return action. The experience pool model and the optimal reward function can assist the deep reinforcement learning model to perform deep reinforcement learning training, so that the deep reinforcement learning model learns interaction experience in the deep reinforcement learning training process. Through the above method, reasonable behavior actions can be executed according to the environment image and the vehicle self-state of the target vehicle, so that the application can improve the reliability of autonomous driving control. The application also provides a control system of an autonomous vehicle, a storage medium and an electronic device, which have the above beneficial effects, and details are not repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 A flow chart of a control method of an autonomous vehicle provided by the embodiments of the application;

[0044] Figure 2A perception information fusion processing flowchart provided by an embodiment of the present application;

[0045] Figure 3 A structural schematic diagram of a deep learning model provided by an embodiment of the present application;

[0046] Figure 4 A probability change schematic diagram of a deep reinforcement learning model training behavior selection strategy provided by an embodiment of the present application;

[0047] Figure 5 A reward curve schematic diagram in a training process provided by an embodiment of the present application;

[0048] Figure 6 A process diagram of an automatic driving behavior decision provided by an embodiment of the present application;

[0049] Figure 7 A deep reinforcement learning model training flowchart provided by an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present application.

[0051] Please see the following Figure 1 , Figure 1 A flowchart of a control method of an autonomous vehicle provided by an embodiment of the present application.

[0052] The specific steps can include:

[0053] S101: Obtain an environment image of a region where a target vehicle is located, and fuse a feature vector of the environment image with a vehicle self-state of the target vehicle to obtain a fusion state;

[0054] The present embodiment can be applied to a target vehicle with an autonomous driving function, and the present embodiment can use a vehicle-mounted camera of the target vehicle, a vehicle-mounted camera of another vehicle and a roadside camera to obtain the above-mentioned environment image. After obtaining the environment image, a feature extraction operation can be performed on the environment image to obtain the feature vector.

[0055] The vehicle state of the target vehicle can be obtained by using the vehicle chassis sensor of the target vehicle, and the vehicle state is fused with the feature vector of the environment image to obtain a fused state. The vehicle state includes any one or a combination of the driving speed, steering wheel angle, accelerator opening degree, and brake opening degree.

[0056] As a feasible implementation, before the feature vector of the environment image is fused with the vehicle state of the target vehicle, the vehicle state of the target vehicle can be obtained by using the vehicle chassis sensor of the target vehicle, and the vehicle state is normalized to be fused with the feature vector of the environment image to obtain a fused state.

[0057] S102: Determine whether the fused state matches the driving experience data; if yes, go to S103; if no, go to S104;

[0058] Before this step, a human driving experience database can be obtained, and the fused state is matched with the driving experience data in the human driving experience database. If there is driving experience data matching the fused state, the experience pool model is used to determine the behavior action to be performed; if there is no driving experience data matching the fused state, the deep learning model is used to determine the behavior action to be performed.

[0059] S103: Use the experience pool model to determine the first behavior action corresponding to the fused state, and control the target vehicle to perform the first behavior action;

[0060] Before this step, there is an operation of performing inverse reinforcement learning on the driving experience data and back-propagating the reward function to obtain the experience pool model. If there is driving experience data matching the fused state, the fused state is input into the experience pool model to obtain the first behavior action, and the target vehicle is controlled to perform the first behavior action. By performing the first behavior action, any one or a combination of the steering wheel angle, accelerator opening degree, and brake opening degree of the target vehicle can be changed.

[0061] S104: Use the deep reinforcement learning model to determine the second behavior action corresponding to the fused state, and control the target vehicle to perform the second behavior action;

[0062] Wherein, before the present step, there can also be an operation of assisting the deep reinforcement learning model in deep reinforcement learning training by using the experience pool model and the optimal reward function. The deep reinforcement learning model can learn from the interactive experience in the deep reinforcement learning training process. If there is no driving experience data matching the fusion state, the deep reinforcement learning model is used to determine a second behavior action corresponding to the fusion state, and the target vehicle is controlled to perform the second behavior action. By performing the second behavior action, any one or a combination of the steering wheel angle, the accelerator opening degree and the brake opening degree of the target vehicle can be changed.

[0063] The embodiment fuses the feature vector of the environment image and the vehicle state to obtain a fusion state. If the fusion state matches the driving experience data, the experience pool model is used to determine a first behavior action to be performed. If the fusion state does not match the driving experience data, a second behavior action to be performed is determined using a deep reinforcement learning model. The experience pool model is obtained by inverse reinforcement learning and reward function backpropagation. The experience pool model can assist in environmental exploration in deep reinforcement learning training. The backpropagated reward function enables the deep reinforcement learning model to learn how to output high-return actions. The experience pool model and the optimal reward function can assist the deep reinforcement learning model in deep reinforcement learning training, so that the deep reinforcement learning model learns from the interactive experience in the deep reinforcement learning training process. In this way, a reasonable behavior action can be performed according to the environment image and the vehicle state of the target vehicle, so that the reliability of automatic driving control can be improved.

[0064] As for Figure 1 For further introduction of the corresponding embodiment, before fusing the feature vector of the environment image and the vehicle state of the target vehicle, the feature vector of the environment image can be extracted and the fusion state can be obtained in the following way:

[0065] Step 1: Use a perception model to locate the target in the environment image.

[0066] Wherein, the target can include vehicles, pedestrians and other obstacles affecting automatic driving.

[0067] Step 2: Generate a two-dimensional grid map corresponding to the environment image according to the positioning result; wherein the value of the cell corresponding to the target in the two-dimensional grid map is 1, and the value of other cells is 0.

[0068] Step 3: Use a convolutional neural network to extract features from the two-dimensional grid map to obtain the feature vector of the environment image.

[0069] Step 4: Fuse the feature vector of the environment image and the vehicle state to obtain a fusion feature.

[0070] Please refer to Figure 2 , Figure 2 The perception information fusion processing flowchart provided by the embodiment of the application converts the environment image into a two-dimensional grid image, and then uses a convolutional neural network to extract a feature vector, and fuses the feature vector with the driving speed, steering wheel angle, throttle opening and brake opening. Specifically, the target detection and positioning result information obtained by the perception model can be used to reflect the road environment state by using a two-dimensional grid, and the road environment state can be displayed in the form of a bird's eye view. The cells with targets are set to "1", and the cells without targets are set to "0". The convolutional neural network is used to extract features from the two-dimensional grid of the environment state, the vehicle state information (driving speed, steering wheel angle, throttle opening and brake opening) is normalized, and the vector information obtained after feature extraction is spliced and fused to obtain the final decision model current state input, i.e. the fusion state S t .

[0071] To fully ensure the accuracy and safety of the behavior decision of the autonomous vehicle, sufficient environmental perception information needs to be obtained. In a traffic jam and a driving environment with occlusion, the information obtained by relying only on the vehicle-mounted camera for environmental perception is often limited. In the embodiment, the image information captured by other vehicles and roadside cameras is obtained through the vehicle-road cooperation technology, together with the image information obtained by the vehicle-mounted camera of the vehicle itself, and the YOLOv5 algorithm is used to detect and locate the targets such as vehicles and pedestrians in the environment. By synthesizing all the perception results, the panoramic perception within a certain range is finally obtained, so as to fully ensure the integrity of the state input information of the autonomous vehicle decision.

[0072] As a feasible implementation manner, before the experience pool model is used to determine the first behavior action corresponding to the fusion state, the experience pool model and the optimal reward function can be set in the following manner:

[0073] Step B1: determining the generator and the discriminator, and setting the function corresponding to the discriminator as the reward function;

[0074] The expression of the function D θ corresponding to the above-mentioned discriminator is θ represents the discriminator parameter, s represents the fusion state, a represents the behavior action, and f θ represents the auxiliary network parameterized by θ, π(a|s) represents the generator strategy, and r seg represents the semantic reward. The semantic reward includes any one or a combination of a plurality of items selected from the group consisting of collision semantic reward, lane invasion semantic reward and lane changing semantic reward. The semantic reward r seg ={r c ,r i ,r r} indicates, involving the collision of specific semantics (r c = -30), lane invasion (r i = -10), frequent lane change (r r = -2), which helps to improve learning efficiency.

[0075] Step B2: inverse reinforcement learning and reward function back propagation based on driving experience data are performed on the generator and the discriminator, so as to correct the generator and the discriminator;

[0076] Step B3: the corrected generator is set as the experience pool model;

[0077] Step B4: the function corresponding to the corrected discriminator is set as the optimal reward function.

[0078] Imitation learning includes behavior cloning and inverse reinforcement learning. Behavior cloning aims to completely copy the behavior of experts without any evaluation index, so the upper limit of learning depends on the level of experts, and the learning ability of the method is limited, and sometimes the wrong behavior may be selected for copying. In the task of autonomous driving, the behavior cloning method sometimes cannot make correct behavior decisions when encountering some dangerous situations.

[0079] When using inverse reinforcement learning, driving experience data is needed, but through learning of the driving experience data and back propagation of the reward function, correction is continuously made in the training process. The embodiment learns by using an adversarial inverse reinforcement learning method, taking the generator as the inverse reinforcement learning model, and the discriminator function representing the reward function.

[0080] In the above process, the discriminator is used to minimize the cross-entropy loss L(D) between the driving experience data (π) and the generated sample (G) as follows:

[0081] L(D) = E (s，a)～π [-log D θ (s, a)] + E (s，a)～G [-log(1-D θ (s, a))]

[0082] To reduce the probability of the generator being classified as a generated sample, define the generator loss L(G):

[0083] L(G) = E (s，a)～G [log(1-D θ (s, a))-log D θ (s, a)]

[0084] E (s，a)～π is for the sample taken from the real driving experience data distribution, E (s，a)～GData generated after the generator. The above embodiment can collect sufficient driving experience data, process information, construct state input-behavior experience pool, and continuously train the generator and the discriminator, so that the discriminator is more and more in line with the expert level, and finally the optimal reward function is obtained to guide the training of the deep reinforcement learning model.

[0085] As a feasible implementation, before determining the second behavior action corresponding to the fusion state by using the deep reinforcement learning model, the deep reinforcement learning model can also be trained in the following manner, and the specific process is as follows:

[0086] Step C1: building the deep reinforcement learning model;

[0087] The deep reinforcement learning model comprises an Actor network, a Critic network, and an experience pool.

[0088] Step C2: training the deep reinforcement learning model from scratch by using the experience pool model and the optimal reward function;

[0089] Step C3: distributing the combination data learned by the deep reinforcement learning model by using the quantile regression method, and optimizing the N combination data with the lowest scores in the deep reinforcement learning model according to the optimization formula; wherein the combination data is the corresponding relationship between the fusion state and the behavior action.

[0090] The optimization formula is Q(s,a)=r(s,a)+γinf(Q(s′,a)); Q(s,a) represents the optimized combination data, r(s,a) represents the reward, γ represents the discount rate, inf represents the lower bound, and Q(s′,a) represents the combination data before optimization.

[0091] After inverse reinforcement learning training, the optimal reward function and a generator close to the expert level can be obtained. A new deep reinforcement learning model is built, trained from scratch by using the obtained optimal reward function, and the generator is used as an experience pool model to assist the training of the new model. The new deep reinforcement learning model can use the DDPG network. The model comprises a group of Actor networks, a group of Critic networks, and an experience pool. The experience pool only saves the interaction experience in the deep reinforcement learning training process.

[0092] See Figure 3 , Figure 3 The structure diagram of the deep learning model provided by the embodiment of the application is shown in FIG. 1. The optimizer of the Actor network of the deep learning model updates the parameter θ μ to the target Actor network Soft update parameter θ μThe test Actor network can transmit policy gradient update parameters to the optimizer The optimizer of the Critic network soft updates the parameters θ of the test Critic network Q The test Critic network soft updates the parameters θ of the target Critic network Q The target Critic network returns the target Q value y i to the test Critic network, and the test Critic transmits policy gradient update parameters to the optimizer The test Actor network can transmit a i = μ(s i ) to the test Critic network, and the test Critic network can transmit Q(s i , a i | θ Q ) to the test Actor network. The target Actor network can transmit μ'(s i+1 ) to the target Critic network. θ represents the critic network parameters, and ▽θ represents the gradient of θ. μ represents the actor network parameters, and y i represents the target Q value. The experience replay pool is sampled to obtain {s i , a i , r i , s i+1}. s i represents the current fusion state, a i represents the behavior action, r i represents the reward, and s i+1 represents the next fusion state.

[0093] For the problem of uncertainty, a conservative rule is usually considered for decision-making, that is, it is assumed that there is a hidden target in the occluded part or outside the sensor perception range, to ensure the safety of the vehicle driving. However, the above method will cause the model to be too conservative and produce a huge amount of calculation. Therefore, the present embodiment uses the method of quantile regression to distribute the state-action possibilities of reinforcement learning, and optimizes the worst case (i.e. the lowest score case) according to the following formula:

[0094] Q(s, a) = r(s, a) + γinf(Q(s', a))

[0095] s represents the state, a represents the action, Q(s, a) represents the state-action value function, r(s, a) represents the reward, γ represents the discount rate, and inf represents the lower bound. For DDPG, the Critic network is trained to estimate the value of following the policy, and the worst-case reward is maximized instead of the average reward by using distribution RL.

[0096] As a feasible implementation, the deep reinforcement learning model can be trained by using the experience pool model and the optimal reward function to train the deep reinforcement learning model from scratch according to a behavior selection strategy, wherein the behavior selection strategy is used to describe probabilities of selecting experience pool model actions, random exploration actions and deep reinforcement learning model actions during training of the deep reinforcement learning model.

[0097] Please refer to Figure 4 , Figure 4 A probability change diagram of a behavior selection strategy for training a deep reinforcement learning model provided by an embodiment of the present application. Since the deep reinforcement learning model is trained from scratch after initialization, in order to improve the learning efficiency of the model, a behavior selection strategy is set during the model training process: using experience pool model, random exploration, deep reinforcement learning model to explore, and setting corresponding probabilities. As the model training proceeds, the probabilities of various ways are also adjusted.

[0098] The probability p1 of selecting the experience pool model action is α(1-t / T t ), α represents the initial probability of p1, t represents the current training frequency, T t represents the training period of the experience pool model, when the training frequency reaches T t , it means that the experience pool model is no longer used for exploration. The probability p2 of selecting the random exploration action is β(1-p1), β represents the greed coefficient, which is initially 1, and decreases to 0.1 when the training frequency reaches the exploration period Tm. The probability p3 of selecting the deep reinforcement learning model action is (1-β)(1-p1). In the early stage of training, the behavior actions are mainly output by the experience pool model and random exploration, encouraging the deep reinforcement learning model to explore in the environment. As the model training proceeds, the probability of selecting the deep reinforcement learning model gradually increases until the probability is 0.9. The random exploration still retains a probability of 0.1, so that the deep reinforcement learning model still has exploration in the later stage of training.

[0099] After using inverse reinforcement learning (IRL) to define the reward function, increasing the behavior selection policy (CP), and considering the worst result (QR), the accuracy and convergence speed of the deep reinforcement learning (DRL) model are improved, and a higher reward can be obtained, such as Figure 5 A reward curve diagram during training provided by an embodiment of the present application. Figure 5 In the diagram, Reward represents the reward, and Episode represents the training data amount. After the deep reinforcement learning model converges, the offline data is used for testing, and the task completion rate, driving score and collision situation are counted, as shown in Table 1. The performance using the inverse reinforcement learning method is obviously better than that using the behavior cloning method.

[0100] Table 1 training effect comparison table

[0101]

[0102] The above-described flow is illustrated by the following examples in actual applications.

[0103] The problems of insufficient generalization ability caused by behavior cloning in expert experience learning, uncertainty caused by occlusion or limited field of view, and low learning efficiency in deep reinforcement learning training are solved. Based on the inverse reinforcement learning method, the human driving experience is learned, and the experience pool model truly learns the driving behavior by inversely deducing the reward function. When there is uncertainty in perception, the worst case of the result is considered by accessing the reward distribution, and the safety of the model is guaranteed. The obtained reward function is used to train the deep reinforcement learning model, and the learning efficiency of the model is effectively improved by adjusting the priority level of the experience pool model suggestion, random exploration and deep reinforcement learning model output.

[0104] Please refer to Figure 6 , Figure 6 A kind of automatic driving behavior decision-making process chart provided by the embodiment of the application. The embodiment obtains surrounding environment image by vehicle-mounted camera, other vehicle camera and road side camera etc., obtains target detection and positioning information in environment by perception model. The perception result is fully reflected road condition by constructing two-dimensional grid, then two-dimensional grid is input into CNN, and the obtained feature vector and the vehicle itself state information (vehicle speed, steering wheel angle, throttle opening, brake opening) obtained by vehicle chassis sensor are information processed, fused, to obtain state input S t . State input is matched with data in experience pool, and the behavior action (steering wheel angle, throttle opening, brake opening) matched with state input is output by experience pool model, if there is no matching experience, the behavior action (steering wheel angle, throttle opening, brake opening) is output by trained deep reinforcement learning model.

[0105] Please refer to Figure 7 , Figure 7The data in the experience pool is from human driver driving data. The embodiment is based on the human driving experience data of inverse reinforcement learning, the reward function is backstepped, and the experience pool model is obtained. Then, the obtained reward function is used to train the deep reinforcement learning model, and the final behavior action output is derived from the experience pool model, random exploration and the deep reinforcement learning model, and is selected and output according to the probabilities P1, P2 and P3 of each source. The probability of each output source is adjusted with the model training. In addition, in the deep reinforcement learning process, the uncertainty of the driving scene is considered to optimize the model. The above process involves three main parts of state input acquisition, experience pool model training and deep reinforcement learning model training. The accuracy and generalization ability of the behavior decision output are ensured from three aspects of perception information source and processing, reward function acquisition and deep reinforcement learning model learning efficiency.

[0106] The embodiment proposes an adversarial inverse reinforcement learning experience pool model, which is used in actual interaction and assists in environmental exploration in deep reinforcement learning training; the backstepped reward function enables the deep reinforcement learning model to truly learn how to output high-return actions; semantic rewards are added in the adversarial inverse reinforcement learning framework to improve the performance of the model. In the training of the deep reinforcement learning model, based on the uncertainty of perception, the application proposes a method considering the worst result, maximizing the lower limit of random results, and ensuring the safety of the model; based on environmental exploration, the embodiment proposes a behavior action selection strategy, which selects from the inverse reinforcement learning experience pool model, random exploration and the deep reinforcement learning model according to the probability, so that the model fully explores the environment, thereby improving the learning efficiency of the model and the application effect of the deep reinforcement learning method in automatic driving.

[0107] The embodiment of the application also provides a control system of an automatic driving vehicle, which can include:

[0108] The state determination module is configured to acquire an environment image of a region where the target vehicle is located, and fuse a feature vector of the environment image with a vehicle state of the target vehicle to obtain a fused state;

[0109] The experience matching module is configured to determine whether the fused state matches the driving experience data;

[0110] The first control module is configured to, if the fused state matches the driving experience data, determine a first behavior action corresponding to the fused state by using the experience pool model, and control the target vehicle to perform the first behavior action;

[0111] a second control module configured to determine a second behavior action corresponding to the fusion state by using a deep reinforcement learning model if the fusion state does not match the driving experience data, and control the target vehicle to perform the second behavior action;

[0112] The deep reinforcement learning model is obtained by learning interaction experience in a deep reinforcement learning training process. The experience pool model and the optimal reward function are used to assist the deep reinforcement learning model in deep reinforcement learning training. The experience pool model and the optimal reward function are obtained by inverse reinforcement learning on the driving experience data and reward function backpropagation.

[0113] In this embodiment, the feature vector of the environment image is fused with the vehicle state to obtain a fusion state. If the fusion state matches the driving experience data, a first behavior action to be performed is determined by using an experience pool model. If the fusion state does not match the driving experience data, a second behavior action to be performed is determined by using a deep reinforcement learning model. The experience pool model is obtained by inverse reinforcement learning and reward function backpropagation. The experience pool model can assist in environmental exploration in deep reinforcement learning training. The backpropagated reward function enables the deep reinforcement learning model to learn how to output high-return actions. The experience pool model and the optimal reward function can assist the deep reinforcement learning model in deep reinforcement learning training, so that the deep reinforcement learning model learns from interaction experience in a deep reinforcement learning training process. In this way, a reasonable behavior action can be performed according to the environment image of the target vehicle and the vehicle state, so that the reliability of automatic driving control can be improved.

[0114] Optionally, the method further comprises:

[0115] The feature extraction module is configured to locate a target in the environment image by using a perception model before fusing the feature vector of the environment image with the vehicle state of the target vehicle. The feature extraction module is further configured to generate a two-dimensional grid map corresponding to the environment image according to the positioning result. In the two-dimensional grid map, the value of a cell corresponding to the target is 1, and the values of other cells are 0. The feature extraction module is further configured to extract features of the two-dimensional grid map by using a convolutional neural network to obtain the feature vector of the environment image.

[0116] Further, the method further comprises:

[0117] The normalization module is configured to obtain the vehicle state by using a vehicle chassis sensor of the target vehicle and perform normalization processing on the vehicle state before fusing the feature vector of the environment image with the vehicle state of the target vehicle. The vehicle state includes any one or a combination of a plurality of items selected from the group consisting of a driving speed, a steering wheel angle, an accelerator opening degree, and a brake opening degree.

[0118] Furthermore, it also includes:

[0119] The experience pool model training module is used to determine the generator and the discriminator before using the experience pool model to determine the first behavior action corresponding to the fusion state, and set the function corresponding to the discriminator as the reward function; the function corresponding to the discriminator θ represents the discriminator parameters, s represents the fusion state, a represents the behavior action, and f θ represents the auxiliary network parameterized by θ, π(a|s) represents the generator strategy, and r seg Representing a semantic reward; also used to perform inverse reinforcement learning and reward function inversion on the generator and the discriminator based on driving experience data, so as to correct the generator and the discriminator; also used to set the corrected generator as the experience pool model; and also used to set the function corresponding to the corrected discriminator as the optimal reward function.

[0120] Furthermore, the semantic reward includes any one or a combination of any two of a collision semantic reward, a lane intrusion semantic reward, and a lane change semantic reward.

[0121] Furthermore, it also includes:

[0122] A deep reinforcement learning model training module, configured to build the deep reinforcement learning model before using the deep reinforcement learning model to determine the second behavioral action corresponding to the fusion state; wherein the deep reinforcement learning model includes an actor network, a critic network, and an experience replay pool; further configured to train the deep reinforcement learning model from scratch using the experience pool model and the optimal reward function; further configured to distribute the combination data learned by the deep reinforcement learning model by means of quantile regression, and optimize the N lowest-scoring combination data in the deep reinforcement learning model according to an optimization formula; wherein the combination data is the correspondence between the fusion state and the behavioral action;

[0123] The optimization formula is Q(s,a)=r(s,a)+γinf(Q(s′,a)); Q(s,a) represents the optimized combination data, r(s,a) represents the reward, γ represents the discount rate, inf represents the lower bound, and Q(s′,a) represents the combination data before optimization.

[0124] Furthermore, the process of the deep reinforcement learning model training module using the experience pool model and the optimal reward function to train the deep reinforcement learning model from scratch includes: using the experience pool model and the optimal reward function to train the deep reinforcement learning model from scratch according to the behavior selection strategy;

[0125] The behavior selection strategy is used to describe probabilities of selecting experience pool model actions, random exploration actions and deep reinforcement learning model actions in the process of training the deep reinforcement learning model; the probability of selecting the experience pool model action is p1=α(1-t / T t ), α represents an initial probability of p1, t represents a current training frequency, T t represents an experience pool model training period; the probability of selecting the random exploration action is p2=β(1-p1), β represents a greed coefficient; and the probability of selecting the deep reinforcement learning model action is p3=(1-β)(1-p1).

[0126] Since the embodiments of the system part correspond to the embodiments of the method part, the embodiments of the system part are described in the description of the embodiments of the method part, and are not described here.

[0127] The application further provides a storage medium having a computer program stored thereon, and the computer program can implement the steps provided by the above embodiments when executed. The storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0128] The application further provides an electronic device, which can include a memory and a processor, the memory has a computer program stored therein, and the processor can implement the steps provided by the above embodiments when calling the computer program in the memory. Of course, the electronic device can also include various network interfaces, power supplies and other components.

[0129] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principles of the application, the application can be improved and modified, and these improvements and modifications also fall within the protection scope of the claims of the application.

[0130] It should also be noted that, in this specification, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

Claims

1. A control method of an autonomous vehicle, characterized by, The method comprises the following steps: acquiring an environment image of an area where a target vehicle is located, and fusing a feature vector of the environment image with a vehicle state of the target vehicle to obtain a fused state; determining whether the fused state matches driving experience data; if yes, determining a first behavior action corresponding to the fused state by using an experience pool model, and controlling the target vehicle to perform the first behavior action; if no, determining a second behavior action corresponding to the fused state by using a deep reinforcement learning model, and controlling the target vehicle to perform the second behavior action; wherein the deep reinforcement learning model is obtained by learning interaction experience in a deep reinforcement learning training process; the experience pool model and an optimal reward function are used to assist the deep reinforcement learning model in deep reinforcement learning training; the experience pool model and the optimal reward function are obtained by inverse reinforcement learning on the driving experience data and reward function backpropagation; before determining the first behavior action corresponding to the fused state by using the experience pool model, the method further comprises the following steps: determining a generator and a discriminator, and setting a function corresponding to the discriminator as a reward function; the function corresponding to the discriminator , denotes a discriminator parameter, s denotes a fusion state, denotes a behavior action, denotes a reward function parameterized by an auxiliary network, denotes a generator policy, denotes a semantic reward; performing inverse reinforcement learning and reward function backpropagation on the generator and the discriminator based on the driving experience data, so as to correct the generator and the discriminator; setting the corrected generator as the experience pool model; setting a function corresponding to the corrected discriminator as the optimal reward function.

2. The control method of the automated vehicle according to claim 1, wherein before fusing the feature vector of the environment image with the vehicle state of the target vehicle, the method further comprises the following steps: positioning a target in the environment image by using a perception model; generating a two-dimensional grid map corresponding to the environment image according to the positioning result; wherein the value of a cell corresponding to the target in the two-dimensional grid map is 1, and the value of other cells is 0; extracting features of the two-dimensional grid map by using a convolutional neural network to obtain the feature vector of the environment image.

3. The control method of an automated vehicle according to claim 1, wherein before fusing the feature vector of the environment image with the vehicle state of the target vehicle, the method further comprises the following steps: acquiring the vehicle state by using a vehicle chassis sensor of the target vehicle, and performing normalization processing on the vehicle state; wherein the vehicle state comprises any one or a combination of driving speed, steering wheel angle, throttle opening degree and brake opening degree.

4. The control method of an automated vehicle according to claim 1, wherein The semantic reward comprises any one or a combination of collision semantic reward, lane invasion semantic reward and lane change semantic reward.

5. The control method of an automated vehicle according to claim 1, wherein before determining the second behavior action corresponding to the fused state by using the deep reinforcement learning model, the method further comprises the following steps: building the deep reinforcement learning model; wherein the deep reinforcement learning model comprises an Actior network, a Critic network and an experience replay pool; training the deep reinforcement learning model from scratch by using the experience pool model and the optimal reward function; distributing combination data learned by the deep reinforcement learning model by using quantile regression, and optimizing N combination data with the lowest scores in the deep reinforcement learning model according to an optimization formula; wherein the combination data is a corresponding relationship between a fused state and a behavior action. The optimization formula is ; The combined data after optimization is represented by r(s,a), and the reward is represented by The discount rate is represented by The lower bound is represented by The combined data before optimization is represented by 6. The control method of an automated vehicle according to claim 5, wherein training the deep reinforcement learning model from scratch using the experience pool model and the optimal reward function, comprising: training the deep reinforcement learning model from scratch using the experience pool model and the optimal reward function according to a behavior selection policy; The behavior selection strategy is used to describe probabilities of selecting experience pool model actions, random exploration actions and deep reinforcement learning model actions in the process of training the deep reinforcement learning model. , represents an initial probability, t represents a current training number, represents an experience pool model training period; the probability of selecting the random exploration action , represents a greed coefficient; the probability of selecting the deep reinforcement learning model action .

7. A control system of an autonomous vehicle, characterized by, comprising: The state determination module is used to acquire the environment image of the region where the target vehicle is located, and fuse the feature vector of the environment image with the vehicle state of the target vehicle to obtain a fusion state. The experience matching module is used to judge whether the fusion state matches the driving experience data. The first control module is used to determine a first behavior action corresponding to the fusion state by using an experience pool model if the fusion state matches the driving experience data, and control the target vehicle to perform the first behavior action. The second control module is used to determine a second behavior action corresponding to the fusion state by using a deep reinforcement learning model if the fusion state does not match the driving experience data, and control the target vehicle to perform the second behavior action. The deep reinforcement learning model is obtained by learning the interaction experience in the deep reinforcement learning training process; the experience pool model and the optimal reward function are used to assist the deep reinforcement learning model to perform deep reinforcement learning training; the experience pool model and the optimal reward function are obtained by inverse reinforcement learning on the driving experience data and reward function back-propagation; The experience pool model training module is configured to determine the generator and the discriminator before determining the first behavior action corresponding to the fusion state by using the experience pool model, and set a function corresponding to the discriminator as a reward function; the function corresponding to the discriminator , represents the discriminator parameter, s represents the fusion state, represents the behavior action, represents an auxiliary network parameterized by , represents the generator strategy, represents the semantic reward; the experience pool model training module is further configured to perform inverse reinforcement learning and reward function backstepping on the generator and the discriminator based on driving experience data, so as to correct the generator and the discriminator; the experience pool model training module is further configured to set the corrected generator as the experience pool model; and the experience pool model training module is further configured to set a function corresponding to the corrected discriminator as an optimal reward function.

8. An electronic device, comprising: The memory and the processor, the memory has the computer program, the processor calls the computer program in the memory, and the steps of the control method of the automatic driving vehicle in any one of claims 1 to 6 are realized.

9. A storage medium, characterized by The storage medium stores computer executable instructions, and the computer executable instructions are loaded and executed by the processor to realize the steps of the control method of the automatic driving vehicle in any one of claims 1 to 6.

Images