Automatic driving decision method and device, vehicle and storage medium

Abstract

The application discloses an automatic driving decision method and device, a vehicle and a storage medium. The method comprises the following steps: collecting driving environment data of a target vehicle, wherein the driving environment data comprises vehicle data and obstacle data of the target vehicle; analyzing the driving environment data by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, and the reward function is used for training the deep neural network model according to driving speeds and driving displacements of the target vehicle at at least two time points; and controlling the target vehicle to execute the decision action. The application solves the technical problem that the design of the reward function in the decision model is unreasonable, and thus the decision result output by the model is not reasonable enough.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving decision method and device, a vehicle and a storage medium. BACKGROUND

[0002] The automobile industry is in an era of change, and the development and application of automatic driving related technologies are in full swing. The current automatic driving technology scheme is to process sensor data through perception, output unmanned vehicle instructions through the decision module, plan specific driving trajectories through the planning module, and output them to the control module, which controls the steering wheel and acceleration and deceleration of the unmanned vehicle.

[0003] The current relatively excellent automatic driving decision model based on deep reinforcement learning combines the "perception ability" of deep learning and the "decision ability" of reinforcement learning. The agent in deep reinforcement learning does not need human expert experience or artificial coding under the framework of reinforcement learning. The agent relies entirely on its own learning and environmental interaction signals, and the agent can be realized through deep reinforcement learning. However, these decision models are not comprehensive when designed, which leads to an unreasonable reward function design in the model, and further leads to an unreasonable decision result. SUMMARY

[0004] The embodiments of the present application provide an automatic driving decision method, device, vehicle and storage medium to at least solve the technical problem of unreasonable reward function design in the decision model, which further leads to an unreasonable decision result output by the model.

[0005] According to an aspect of an embodiment of the present application, an automatic driving decision method is provided, comprising:

[0006] Collecting driving environment data of a target vehicle, the driving environment data including vehicle data and obstacle data of the target vehicle; analyzing the driving environment data by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, the reward function is used to train the deep neural network model according to the driving speed and driving displacement of the target vehicle at least two time points; and controlling the target vehicle to perform the decision action.

[0007] Optionally, the construction and training of the deep neural network model based on the reward function and the deep reinforcement learning algorithm comprises: calculating an accumulated reward value according to the value of the reward function; constructing an action value function according to the accumulated reward value and the deep reinforcement learning algorithm; constructing a loss function according to the action value function; constructing an initial deep neural network model according to the loss function and the action value function; and training the initial deep neural network model according to a preset sample and the loss function to obtain the deep neural network model.

[0008] Optionally, constructing the loss function according to the action value function comprises: setting a deep network with a preset weight as a function approximator of the action value function; and constructing the loss function according to the action value function and the function approximator.

[0009] Optionally, training the initial deep neural network model according to the preset sample and the loss function to obtain the deep neural network model comprises: training the initial deep neural network model according to the preset sample to obtain a first output result; placing the first output result into a preset buffer queue; selecting the first output result from the preset buffer queue as a training sample by using a uniform random sampling method; and training the initial deep neural network model by using the training sample to obtain the deep neural network model.

[0010] Optionally, constructing the initial deep neural network model according to the loss function and the action value function comprises: constructing an input layer according to the driving environment data, the input layer comprising a first state space and a second state space, the first state space being configured to input vehicle data of a target vehicle, and the second state space being configured to input obstacle data; constructing a first data extraction layer and a second data extraction layer according to the driving environment data, wherein the first data extraction layer is connected to the first state space, the second data extraction layer is connected to the second state space, the first data extraction layer is configured to extract the vehicle data of the target vehicle to obtain first extraction data, and the second data extraction layer is configured to extract the obstacle data to obtain second extraction data; constructing a first fusion layer and a second fusion layer according to the driving environment data, wherein the first fusion layer is connected to the first data extraction layer and the second data extraction layer, and is configured to fuse the first extraction data and the second extraction data to obtain first fusion data, the second fusion layer is connected to the first data extraction layer, the second data extraction layer and the first fusion layer, and is configured to fuse the first fusion data, the first extraction data and the second extraction data to obtain second fusion data; and constructing an output layer based on the loss function and the action value function, wherein the output layer is connected to the second data fusion layer, and the output layer is configured to make a decision according to the second fusion data and output a decision action.

[0011] Optionally, the vehicle data of the target vehicle at least comprises a speed of the target vehicle and a distance between a geometric center of the target vehicle and a center line of a current lane, and the obstacle data at least comprises a distance between the obstacle and the target vehicle and a speed of the obstacle.

[0012] Optionally, the decision action comprises at least one of the following: an acceleration action, a deceleration action, a speed maintaining action, a left lane changing action and a right lane changing action.

[0013] According to one of the embodiments of the present application, an automatic driving decision device is also provided, comprising:

[0014] The collection module is configured to collect driving environment data of the target vehicle, the driving environment data comprising vehicle data and obstacle data of the target vehicle; the decision module is configured to analyze the driving environment data by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, and the reward function is configured to train the deep neural network model according to driving speeds and driving displacements of the target vehicle at at least two time points; and the control module is configured to control the target vehicle to perform the decision action.

[0015] Optionally, the decision module is further configured to construct and train the deep neural network model based on the reward function and the deep reinforcement learning algorithm, including: calculating an accumulated reward value according to a value of the reward function; constructing an action value function according to the accumulated reward value and the deep reinforcement learning algorithm; constructing a loss function according to the action value function; constructing an initial deep neural network model according to the loss function and the action value function; and training the initial deep neural network model according to a preset sample and the loss function to obtain the deep neural network model.

[0016] Optionally, the decision module is further configured to construct the loss function according to the action value function, including: setting a deep network with a preset weight as a function approximator of the action value function; and constructing the loss function according to the action value function and the function approximator.

[0017] Optionally, the decision module is further configured to train the initial deep neural network model according to the preset sample and the loss function to obtain the deep neural network model, including: training the initial deep neural network model according to the preset sample to obtain a first output result; placing the first output result into a preset buffer queue; selecting the first output result from the preset buffer queue as a training sample by using a uniform random sampling method; and training the initial deep neural network model by using the training sample to obtain the deep neural network model.

[0018] Optionally, the decision module is further configured to construct the initial deep neural network model according to the loss function and the action value function, including: constructing an input layer according to the driving environment data, the input layer including a first state space and a second state space, the first state space being configured to input vehicle data of the target vehicle, and the second state space being configured to input obstacle data; constructing a first data extraction layer and a second data extraction layer according to the driving environment data, wherein the first data extraction layer is connected to the first state space, the second data extraction layer is connected to the second state space, the first data extraction layer is configured to extract the vehicle data of the target vehicle to obtain first extraction data, and the second data extraction layer is configured to extract the obstacle data to obtain second extraction data; constructing a first fusion layer and a second fusion layer according to the driving environment data, wherein the first fusion layer is connected to the first data extraction layer and the second data extraction layer, and is configured to fuse the first extraction data and the second extraction data to obtain first fusion data, and the second fusion layer is connected to the first data extraction layer, the second data extraction layer and the first fusion layer, and is configured to fuse the first fusion data, the first extraction data and the second extraction data to obtain second fusion data; and constructing an output layer based on the loss function and the action value function, wherein the output layer is connected to the second data fusion layer, and the output layer is configured to make a decision according to the second fusion data and output a decision action.

[0019] Optionally, the vehicle data of the target vehicle collected by the collection module at least includes a speed of the target vehicle and a distance between a geometric center of the target vehicle and a center line of a current lane, and the obstacle data at least includes a distance between the target vehicle and the obstacle and a speed of the obstacle.

[0020] Optionally, the decision action executed by the control module includes at least one of the following: an acceleration action, a deceleration action, a speed maintaining action, a left lane changing action and a right lane changing action.

[0021] According to an embodiment of the present application, a vehicle is also provided, including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the automatic driving decision method in any of the above.

[0022] According to an embodiment of the present application, a non-volatile storage medium is also provided, the non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the automatic driving decision method in any of the above when running on a computer or a processor.

[0023] In the embodiment of the present application, firstly, driving environment data of the target vehicle is collected, the driving environment data comprising vehicle data and obstacle data of the target vehicle, then the driving environment data is analyzed by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, the reward function is used to train the deep neural network model according to driving speeds and driving displacements of the target vehicle at least two time points, and finally the target vehicle is controlled to execute the decision action. By using the above method, the reward function based on the driving speeds and the driving displacements of the target vehicle at least two time points is constructed, and the training of the target neural network model is guided by the reward function, so that the decision result of the output of the target neural network model is more correct, thereby solving the technical problem that the reward function in the decision model is designed unreasonably, and the decision result output by the model is not reasonable. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings, which are included to provide a further understanding of the present application and are incorporated in and constitute a part of this application, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0025] Figure 1 is a flow chart of an automatic driving decision method according to an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of an initial deep neural network model according to an embodiment of the present application;

[0027] Figure 3 is a structural block diagram of an automatic driving decision device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", and the like in the description and in the claims of the application and in the above-described drawings are intended to distinguish similar objects and not necessarily to describe a particular sequential or chronological order. It is to be understood that the use of such terms herein is merely for distinguishing between objects in order to more clearly describe the embodiments of the application herein and the terms so used are not necessarily meant to or should be construed to limit the scope of the application to that which is presented by the terms exclusion. Furthermore, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, article, or apparatus that comprises a list of steps or units can not necessarily be limited to those steps or units which are expressly listed, but can include additional steps or units not expressly listed or inherent to such process, method, article, or apparatus.

[0030] According to an embodiment of the application, an embodiment of an automatic driving decision method is provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] The method embodiment can be executed in an electronic device including a memory and a processor, a similar control device or system. Taking the electronic device as an example, the electronic device can include one or more processors and a memory for storing data. Optionally, the above-mentioned electronic device can also include a communication device for communication function and a display device. Those skilled in the art can understand that the above-mentioned structural description is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the electronic device can also include more or less components than the above-mentioned structural description, or have a different configuration from the above-mentioned structural description.

[0032] The processor can include one or more processing units. For example, the processor can include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microcontroller unit (MCU), a field-programmable gate array (FPGA), a neural-network processing unit (NPU), a tensor processing unit (TPU), an artificial intelligent (AI) type processor, or the like. Different processing units can be independent components or integrated in one or more processors. In some examples, the electronic device can also include one or more processors.

[0033] The memory can be used to store a computer program, for example, a computer program corresponding to the automatic driving decision method in the embodiments of the present application. The processor realizes the automatic driving decision method described above by running the computer program stored in the memory. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the electronic device through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0034] The communication device is used to receive or send data via a network. Specific examples of the above-mentioned network can include a wireless network provided by a communication provider of a mobile terminal. In one example, the communication device includes a network interface controller (NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the communication device can be a radio frequency (RF) module, which is used to communicate with the Internet in a wireless manner.

[0035] The display device can be, for example, a liquid crystal display (LCD) and a touch display (also known as a "touch screen" or "touch display screen") in the form of a touch screen. The liquid crystal display can enable a user to interact with a user interface of the mobile terminal. In some embodiments, the mobile terminal described above has a graphical user interface (GUI), with which a user can interact with the GUI by finger contact and / or gesture on the touch-sensitive surface, where the human-machine interaction function can optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving e-mails, call interface, playing digital videos, playing digital music, and / or web browsing, etc., executable instructions for performing the above human-machine interaction functions are configured / stored in one or more computer program products or readable storage media executable by the processor.

[0036] Figure 1 An automatic driving decision method according to an embodiment of the present application is shown in FIG. 1, which comprises the following steps: Figure 1

[0037] Step S101, collecting driving environment data of the target vehicle.

[0038] The driving environment data includes vehicle data of the target vehicle and obstacle data.

[0039] During the automatic driving decision process of the target vehicle, real-time driving environment information needs to be collected. Based on the vehicle data of the target vehicle and the obstacle data around the target vehicle, the target vehicle can make an automatic driving decision.

[0040] Optionally, the driving data at least includes the speed of the target vehicle and the distance between the geometric center of the target vehicle and the current lane centerline, and the obstacle data at least includes the distance between the obstacle and the target vehicle and the speed of the obstacle.

[0041] It should be noted that the obstacle data can be obstacle data in eight directions of the target vehicle. The obstacle can be other vehicles in motion, or it can be a stationary object. The obstacle data in eight directions includes data of the nearest obstacle in front of the current lane, data of the nearest obstacle behind the current lane, data of the nearest obstacle in the left lane, data of the nearest obstacle in the right lane, distance of the nearest obstacle in front of the left lane, distance of the nearest obstacle behind the left lane, distance of the nearest obstacle in front of the right lane, and distance of the nearest obstacle behind the right lane.

[0042] Step S102, using a deep neural network model to analyze the driving environment data to obtain a decision action.

[0043] ​The deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, and the reward function is used to train the deep neural network model according to the driving speeds and driving displacements of the target vehicle at at least two time points.

[0044] For example, the two time points can be t and t-1, the driving speeds can be the driving speed at t and the driving speed at t-1, and the driving displacements can be the total mileage of the vehicle at t and the total mileage of the vehicle at t-1. The reward function is used to guide the training of the deep neural network model, and the reward value is affected by the two parameters of driving speed and driving displacement, so that the decision action output by the trained deep neural network model is more accurate.

[0045] In step S103, the target vehicle is controlled to perform the decision action.

[0046] When the deep neural network model outputs the decision action, the control module controls the target vehicle to perform the output decision action.

[0047] Optionally, the decision action includes at least one of the following: acceleration action, deceleration action, speed maintenance, left lane change action and right lane change action.

[0048] Through the above steps, first, the driving environment data of the target vehicle is collected, including vehicle data and obstacle data of the target vehicle, then the driving environment data is analyzed by using the deep neural network model to obtain the decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, the reward function is used to train the deep neural network model according to the driving speeds and driving displacements of the target vehicle at at least two time points, and finally the target vehicle is controlled to perform the decision action. By using the above method, a reward function based on the driving speeds and driving displacements of the target vehicle at at least two time points is constructed, and the training of the target neural network model is guided by the reward function, which can make the output decision result of the target neural network model more correct, thereby solving the technical problem that the reward function design in the decision model is not reasonable, and the output decision result of the model is not reasonable.

[0049] Optionally, the deep neural network model can be constructed and trained based on a reward function and a deep reinforcement learning algorithm, which can include the following steps:

[0050] In step S102a, the cumulative reward value is calculated according to the value of the reward function.

[0051] The reward function is constructed based on the driving speeds, driving displacements and collision penalty function of the target vehicle at at least two time points, and is used to evaluate the value of the target vehicle from the last state to the current state in the automatic driving decision process. The state is the image information collected by the vehicle. The expression of the reward function is as follows:

[0052] r t =(d t -d t-1 )+r col +(v t -v t-1 )

[0053] Where, d t The total distance traveled by the target vehicle at time t, d t-1 v is the total distance traveled by the target vehicle at time t. t Let v be the speed of the target vehicle at time t. t-1 Let t be the speed of the target vehicle at time t-1.

[0054] Where the collision penalty function r col The expression is as follows:

[0055]

[0056] The cumulative reward value is calculated using the cumulative reward function, which is used to evaluate the quality of decision-making actions during autonomous driving. The function expression of the cumulative reward function is as follows:

[0057]

[0058] Among them, R t Let r be the cumulative reward value at time t. t+1 Let t+1 be the return function value, γ be the discount factor, the discount factor ranges from [0,1), and L be the preset time length.

[0059] Step S102b: Construct an action value function based on the cumulative reward value and the deep reinforcement learning algorithm.

[0060] The deep reinforcement learning algorithm is a deep Q-network algorithm. The action-value function is used to represent the value of the current decision action. The function expression of the action-value function is as follows:

[0061] Q * (s,a)=E[R+γmax a′ Q * (s′,a′)∣s,a]

[0062] This function represents the action value of the target vehicle when it is in state s and takes decision action a. Here, R is the cumulative reward function value during the state transition, γ is the discount factor (ranging from [0,1)), s′ is the next state of the target vehicle, and a′ is the decision action to be performed in the next state.

[0063] Step S102c: Construct a loss function based on the action value function.

[0064] The loss function is used to represent the degree of difference between the prediction and the actual data, and then measure the goodness of the model.

[0065] Optionally, the step S102c of constructing the loss function according to the action value function can include the following steps:

[0066] In step S102c1, a deep network with a preset weight is set as a function approximator of the action value function.

[0067] Specifically, a Q neural network with a preset weight θ is used as a function approximator of the action value function, that is, Q (s, a; θ) ≈ Q* (s, a).

[0068] In step S102c2, the loss function is constructed according to the action value function and the function approximator.

[0069] The expression of the loss function is as follows:

[0070]

[0071] In step S102d, the initial deep neural network model is constructed according to the loss function and the action value function.

[0072] In step S102e, the initial deep neural network model is trained to obtain a deep neural network model according to the preset sample and the loss function.

[0073] The preset sample is a five-tuple data, and the representation form of the five-tuple data is e t = (s t , a t , r t , s t+1 , F). Wherein, s t is the state of the target vehicle and the obstacle at time t, a t is the decision action executed by the target vehicle at time t, r t is the cumulative reward function value of the target vehicle at time t, s t+1 is the next state of the target vehicle and the obstacle at time t, and F is a flag indicating whether s t+1 is the final state of the target vehicle following task decision sequence.

[0074] During the training process, the preset sample is input, and the initial deep neural network model is trained by adjusting the parameter θ i in the loss function at each iteration i, and the error is reduced by stochastic gradient descent. is the network parameter used to calculate the target at iteration i. In order to make the learning process more stable, a fixed step number C is set, and the latest network parameter

[0075] Optionally, the step S102e of obtaining the deep neural network model by training the initial deep neural network model according to the preset sample and the loss function can include the following steps.

[0076] In step S102e1, the initial deep neural network model is trained according to the preset sample to obtain a first output result.

[0077] In step S102e2, the first output result is put into a preset buffer queue.

[0078] In step S102e3, the first output result is selected as a training sample from the preset buffer queue by using a uniform random sampling method.

[0079] In step S102e4, the initial deep neural network model is trained by using the training sample to obtain the deep neural network model.

[0080] According to the steps S102e1, S102e2, S102e3 and S102e4, the data generated by training is put into the preset buffer queue, and then the training data is extracted from the preset buffer queue by using the random uniform sampling method. The extracted training data is used as a sample to train the initial deep neural network model, which can break the correlation between the data and further make the trained deep neural network model more stable.

[0081] Referring to Figure 2 Optionally, the step S102d of constructing the initial deep neural network model according to the loss function and the action value function can include the following steps.

[0082] In step S102d1, an input layer is constructed according to the driving environment data.

[0083] The input layer includes a first state space and a second state space, the first state space is used to input vehicle data of a target vehicle, and the second state space is used to input obstacle data.

[0084] The first state space is represented as:

[0085] s e =[v e ,d e ]

[0086] wherein v e is the speed of the target vehicle, and d e is the distance between the geometric center point of the target vehicle and the center line of the lane. The second state space is represented as:

[0087] s sur =[d f ,v f ,d b ,vb ,d lf ,v lf ,d lb ,v lb ,d rf ,v rf ,d rb ,v rb ]

[0088] wherein the distance of the nearest vehicle in front of the current lane is d f and the speed of the nearest vehicle in front of the current lane is v f the distance of the nearest vehicle behind the current lane is d b and the speed of the nearest vehicle behind the current lane is v b the distance of the nearest vehicle in front of the left lane is d lf and the speed of the nearest vehicle in front of the left lane is v lf the distance of the nearest vehicle behind the left lane is d lb and the speed of the nearest vehicle behind the left lane is v lb the distance of the nearest vehicle in front of the right lane is d rf and the speed of the nearest vehicle in front of the right lane is v rf the distance of the nearest vehicle behind the right lane is d rb and the speed of the nearest vehicle behind the right lane is v rb .

[0089] Step S102d2, constructing a first data extraction layer and a second data extraction layer according to the driving environment data.

[0090] wherein the first data extraction layer is connected with the first state space, and the second data extraction layer is connected with the second state space, the first extraction layer is used for data extraction on the vehicle data of the target vehicle to obtain first extraction data, and the second extraction layer is used for data extraction on the obstacle data to obtain second extraction data.

[0091] Step S102d3, constructing a first fusion layer and a second fusion layer according to the driving environment data.

[0092] wherein the first fusion layer is connected with the first data extraction layer and the second data extraction layer, and is used for fusing the first extraction data and the second extraction data to obtain first fusion data, and the second fusion layer is connected with the first data extraction layer, the second data extraction layer and the first fusion layer, and is used for fusing the first fusion data, the first extraction data and the second extraction data to obtain second fusion data.

[0093] Step S102d4, constructing an output layer based on a loss function and an action value function.

[0094] wherein the output layer is connected with the second data fusion layer, and the output layer is used for decision-making according to the second fusion data and outputting a decision action.

[0095] It should be noted that the first data extraction layer, the second data extraction layer, the first fusion layer, the second fusion layer, and the output layer are all full connection layers. The output dimension of the output layer is an action decision result of [1, 1].

[0096] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and a necessary general hardware platform, and of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the parts that contribute to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device) to execute the method described in each embodiment of the present application.

[0097] In this embodiment, an automatic driving decision device is also provided, which is used to implement the above embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0098] Figure 3 is a structural block diagram of an automatic driving decision device 200 according to an embodiment of the present application, as Figure 3 shown, taking the automatic driving decision device 200 as an example, the device includes: a collection module 201, the collection module 201 is used to collect driving environment data of a target vehicle, the driving environment data includes vehicle data and obstacle data of the target vehicle; a decision module 202, the decision module 202 is used to analyze the driving environment data by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, the reward function is used to train the deep neural network model according to the driving speed and the driving displacement of the target vehicle at least two times; a control module 203, the control module 203 is used to control the target vehicle to execute the decision action.

[0099] Optionally, the decision module 202 is further used to construct and train the deep neural network model based on the reward function and the deep reinforcement learning algorithm, including: calculating an accumulated reward value according to the value of the reward function; constructing an action value function according to the accumulated reward value and the deep reinforcement learning algorithm; constructing a loss function according to the action value function; constructing an initial deep neural network model according to the loss function and the action value function; training the initial deep neural network model according to a preset sample and the loss function to obtain the deep neural network model.

[0100] Optionally, the decision module 202 is further configured to construct the loss function according to the action value function, including: setting a deep network with a preset weight as a function approximator of the action value function; and constructing the loss function according to the action value function and the function approximator.

[0101] Optionally, the decision module 202 is further configured to train the initial deep neural network model to obtain the deep neural network model according to the preset sample and the loss function, including: training the initial deep neural network model according to the preset sample to obtain a first output result; placing the first output result into a preset buffer queue; selecting the first output result from the preset buffer queue as a training sample by using a uniform random sampling method; and training the initial deep neural network model by using the training sample to obtain the deep neural network model.

[0102] Optionally, the decision module 202 is further configured to construct the initial deep neural network model according to the loss function and the action value function, including: constructing an input layer according to the driving environment data, the input layer including a first state space and a second state space, the first state space being configured to input vehicle data of a target vehicle, and the second state space being configured to input obstacle data; constructing a first data extraction layer and a second data extraction layer according to the driving environment data, the first data extraction layer being connected to the first state space, the second data extraction layer being connected to the second state space, the first data extraction layer being configured to extract the vehicle data of the target vehicle to obtain first extraction data, and the second data extraction layer being configured to extract the obstacle data to obtain second extraction data; constructing a first fusion layer and a second fusion layer according to the driving environment data, the first fusion layer being connected to the first data extraction layer and the second data extraction layer, and being configured to fuse the first extraction data and the second extraction data to obtain first fusion data, the second fusion layer being connected to the first data extraction layer, the second data extraction layer and the first fusion layer, and being configured to fuse the first fusion data, the first extraction data and the second extraction data to obtain second fusion data; and constructing an output layer based on the loss function and the action value function, the output layer being connected to the second data fusion layer, and being configured to make a decision according to the second fusion data and output a decision action.

[0103] Optionally, the vehicle data of the target vehicle collected by the collection module 201 at least includes a speed of the target vehicle and a distance between a geometric center of the target vehicle and a current lane centerline, and the obstacle data at least includes a distance between the target vehicle and the obstacle and a speed of the obstacle.

[0104] Optionally, the decision action performed by the control module 203 includes at least one of the following: an acceleration action, a deceleration action, a speed maintaining action, a left lane changing action and a right lane changing action.

[0105] The embodiment of the present application also provides a vehicle, comprising a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in the above-mentioned embodiment of the automatic driving decision method.

[0106] Optionally, in the embodiment, the processor in the vehicle can be configured to execute the computer program to perform the following steps:

[0107] In step S101, driving environment data of a target vehicle is collected.

[0108] In step S102, the driving environment data is analyzed by using a deep neural network model to obtain a decision action.

[0109] In step S103, the target vehicle is controlled to perform the decision action.

[0110] Optionally, the specific examples in the embodiment can refer to the examples described in the above-mentioned embodiments and optional implementation manners, and the embodiment will not be described here again.

[0111] The embodiment of the present application also provides a non-volatile storage medium, which stores a computer program, wherein the computer program is configured to execute the steps in the above-mentioned embodiment of the automatic driving decision method when running on a computer or a processor.

[0112] Optionally, in the embodiment, the non-volatile storage medium can be configured to store a computer program for performing the following steps:

[0113] In step S101, driving environment data of a target vehicle is collected.

[0114] In step S102, the driving environment data is analyzed by using a deep neural network model to obtain a decision action.

[0115] In step S103, the target vehicle is controlled to perform the decision action.

[0116] Optionally, in the embodiment, the non-volatile storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage media that can store a computer program.

[0117] The above-mentioned embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0118] In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0119] In several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways. Among them, the above-described device embodiments are only illustrative, for example, the division of the units can be a logical function division, and actual implementation can have another division way, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or modules shown or discussed can be indirect coupling or communication connection between the units or modules through some interfaces, and can be electrical or other forms.

[0120] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed to multiple units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0121] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0122] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0123] The above is only the preferred embodiment of the present application, and it should be pointed out that for ordinary skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should be considered as the protection scope of the present application.

Claims

1. An automatic driving decision method characterized by, The method comprises: collecting driving environment data of a target vehicle, the driving environment data comprising vehicle data of the target vehicle and obstacle data, wherein the vehicle data of the target vehicle comprises a speed of the target vehicle and a distance between a geometric center of the target vehicle and a lane centerline, and the obstacle data comprises distances and speeds of nearest vehicles in front of a current lane, rear of the current lane, left of the current lane, right of the current lane, and rear of the left lane; analyzing the driving environment data by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, the reward function is used to train the deep neural network model according to driving speeds and driving displacements of the target vehicle at at least two time points, and the reward function is represented by the following formula: r t = (d t - d t-1 ) + r col + (v t - v t-1 ) where d t is the total distance traveled by the target vehicle t, d t-1 is the total distance traveled by the target vehicle t-1, v t is the speed of the target vehicle t, v t-1 is the speed of the target vehicle t-1, r col is the collision penalty function, where the collision penalty function r col is expressed as: controlling the target vehicle to perform the decision action; wherein the method further comprises: constructing an input layer according to the driving environment data, wherein the input layer comprises a first state space and a second state space, the first state space is used to input the vehicle data of the target vehicle, and the second state space is used to input the obstacle data; constructing a first data extraction layer and a second data extraction layer according to the driving environment data, wherein the first data extraction layer is connected with the first state space, the second data extraction layer is connected with the second state space, the first data extraction layer is used to extract data of the vehicle data of the target vehicle to obtain first extracted data, and the second data extraction layer is used to extract data of the obstacle data to obtain second extracted data; constructing a first fusion layer and a second fusion layer according to the driving environment data, wherein the first fusion layer is connected with the first data extraction layer and the second data extraction layer, is used to fuse the first extracted data and the second extracted data to obtain first fusion data, and the second fusion layer is connected with the first data extraction layer, the second data extraction layer and the first fusion layer, is used to fuse the first fusion data, the first extracted data and the second extracted data to obtain second fusion data; constructing an output layer based on a loss function and an action value function, wherein the action value function is constructed based on the reward function and the deep reinforcement learning algorithm, the loss function is constructed based on the action value function, the output layer is connected with the second fusion layer, and the output layer is used to make a decision according to the second fusion data and output the decision action.

2. The automatic driving decision method of claim 1, wherein, The deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, which comprises: calculating a cumulative reward value according to a value of the reward function; constructing the action value function according to the cumulative reward value and the deep reinforcement learning algorithm; constructing the loss function according to the action value function; constructing an initial deep neural network model according to the loss function and the action value function; The initial deep neural network model is trained according to the preset sample and the loss function to obtain the deep neural network model.

3. The automatic driving decision method of claim 2, wherein, The loss function is constructed according to the action-value function, and the method comprises the following steps: a deep network with a preset weight is set as a function approximator of the action-value function; the loss function is constructed according to the action-value function and the function approximator.

4. The automatic driving decision method of claim 2, wherein, The initial deep neural network model is trained according to the preset sample and the loss function to obtain the deep neural network model, and the method comprises the following steps: The initial deep neural network model is trained according to the preset sample to obtain a first output result; the first output result is put into a preset buffer queue; the first output result is selected as a training sample from the preset buffer queue by using a uniform random sampling method; the initial deep neural network model is trained by using the training sample to obtain the deep neural network model.

5. The automatic driving decision method of claim 1, wherein, The vehicle data of the target vehicle at least comprises the speed and the distance between the geometric center of the target vehicle and the centerline of the current lane, and the obstacle data at least comprises the distance between the obstacle and the target vehicle and the speed of the obstacle.

6. The automatic driving decision method of claim 1, wherein, The decision action comprises at least one of the following: an acceleration action, a deceleration action, a speed maintaining action, a left lane changing action and a right lane changing action.

7. An automatic driving decision device characterized by comprising: The method comprises the following steps: a collection module is configured to collect driving environment data of a target vehicle, wherein the driving environment data comprises vehicle data of the target vehicle and obstacle data; a decision module is configured to analyze the driving environment data by using a deep neural network model to obtain a decision action, wherein the deep neural network model is constructed and trained based on a reward function and a deep reinforcement learning algorithm, the reward function is used to train the deep neural network model according to driving speeds and driving displacements of the target vehicle at least two time points, and the reward function is represented by the following formula: r t = (d t - d t-1 ) + r col + (v t - v t-1 ) where d t is the total distance traveled by the target vehicle t, d t-1 is the total distance traveled by the target vehicle t-1, v t is the speed of the target vehicle t, v t-1 is the speed of the target vehicle t-1, r col is the collision penalty function, where the collision penalty function r col is expressed as: a control module is configured to control the target vehicle to execute the decision action. The method further comprises: constructing an input layer according to the driving environment data, wherein the input layer comprises a first state space and a second state space, the first state space is used for inputting vehicle data of the target vehicle, and the second state space is used for inputting the obstacle data; constructing a first data extraction layer and a second data extraction layer according to the driving environment data, wherein the first data extraction layer is connected with the first state space, the second data extraction layer is connected with the second state space, the first data extraction layer is used for data extraction on the vehicle data of the target vehicle to obtain first extraction data, and the second data extraction layer is used for data extraction on the obstacle data to obtain second extraction data; constructing a first fusion layer and a second fusion layer according to the driving environment data, wherein the first fusion layer is connected with the first data extraction layer and the second data extraction layer, is used for fusion on the first extraction data and the second extraction data to obtain first fusion data, and the second fusion layer is connected with the first data extraction layer, the second data extraction layer and the first fusion layer, is used for fusion on the first fusion data, the first extraction data and the second extraction data to obtain second fusion data; constructing an output layer based on a loss function and an action value function, wherein the action value function is constructed based on the reward function and the deep reinforcement learning algorithm, the loss function is constructed based on the action value function, the output layer is connected with the second fusion layer, and the output layer is used for decision-making according to the second fusion data and outputting the decision action.

8. A vehicle comprising a memory and a processor, characterized in that The memory stores a computer program, and the processor is configured to execute the computer program to perform the automatic driving decision-making method in any one of claims 1 to 6.

9. A non-volatile storage medium, comprising: The non-volatile storage medium stores a computer program, and the computer program is configured to execute the automatic driving decision-making method in any one of claims 1 to 6 when running on a computer or a processor.

Images