A trajectory prediction-based reinforcement learning automatic driving decision method and device

Abstract

The application provides a trajectory prediction-based reinforcement learning automatic driving decision method and device, and relates to the technical field of automatic driving. The method comprises the following steps: obtaining driving trajectory time series data of an automatic driving vehicle and other vehicles around the automatic driving vehicle in a historical time period; inputting the driving trajectory time series data into a trained Transformer model to predict the driving trajectories of the other vehicles around the automatic driving vehicle and obtain corresponding trajectory prediction information; adding the trajectory prediction information into a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes an automatic driving decision based on the target state space. It can be seen that the application adds the driving trajectory prediction information of the other vehicles into the state information of the reinforcement learning decision, thereby reducing the possibility of collision between the automatic driving vehicle and the other vehicles and achieving the purpose of improving the passing success rate of the automatic driving decision.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a trajectory prediction-based reinforcement learning automatic driving decision method and device. BACKGROUND

[0002] Automatic driving technology is an important technology in the field of motor vehicles and is also a direction that major manufacturers focus on. With the development of automatic driving technology, the automatic driving function of vehicles is becoming more and more mature. Vehicles running in automatic driving mode can free the driver from some driving-related responsibilities. Among them, automatic driving can include different stages of technical level such as L1-L5. With the improvement of the stage level, the automatic driving technology is more and more mature.

[0003] However, the current automatic driving model has the problem of unreasonable trajectory prediction of surrounding vehicles, which may cause traffic accidents. Reasonable trajectory prediction of the motion of other traffic participants by the automatic driving model will greatly improve the safety of automatic driving. Therefore, how to reasonably predict the motion of other traffic participants is a technical problem that needs to be solved in the field. SUMMARY

[0004] The purpose of the present application is to provide a trajectory prediction-based reinforcement learning automatic driving decision method and device, electronic equipment and storage medium, which can improve the accuracy of reinforcement learning automatic driving decision.

[0005] In a first aspect, the embodiments of the present application provide a trajectory prediction-based reinforcement learning automatic driving decision method, comprising:

[0006] Obtaining driving trajectory time series data of an automatic driving vehicle and other vehicles around the automatic driving vehicle in a historical time period;

[0007] Inputting the driving trajectory time series data into a trained Transformer model to predict the driving trajectory of the other vehicles around the automatic driving vehicle and obtaining corresponding trajectory prediction information;

[0008] Adding the trajectory prediction information into a state space corresponding to a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes automatic driving decisions based on the target state space.

[0009] In some embodiments of the present application, the other vehicles around the automatic driving vehicle include vehicles in the six directions of the front, left front, right front, back, left back and right back of the location of the automatic driving vehicle.

[0010] In some embodiments of the present application, the training process of the reinforcement learning model is as follows:

[0011] Initialize the parameters of the reinforcement learning model;

[0012] Acquire time-series data samples of the driving trajectories of autonomous vehicles and other vehicles in their vicinity;

[0013] The driving trajectory time series data sample is input into the trained Transformer model to predict the driving trajectories of other vehicles around the autonomous vehicle, and obtain the corresponding trajectory prediction information sample.

[0014] The trajectory prediction information samples are added to the state space corresponding to the reinforcement learning model to obtain state space samples:

[0015] s t =(x t y t x it y it x t+T y t+T x it+T y it+T v xt v yt v xit v yit );

[0016] Where i = 1, 2, 3, 4, 5, 6, representing the six directions: front, front left, front right, rear, rear left, and rear right, respectively; x and y represent the horizontal and vertical coordinates of the autonomous vehicle, respectively, and Δx it Δy it Let i represent the relative distance between the car and the autonomous vehicle, T represent the time duration, and v represent the value.

[0017] Through policy network a t ~π θ (a t |s t Generate action a t ;

[0018] Through the state transition probability s t ~p(s t+1 |s t a t Get s t+1 ;

[0019] s t a t 、r(s t a t ), s t+1 Store in the experience replay cache, where r represents the reward;

[0020] sample a training sample set with a batch size of β from the experience replay cache;

[0021] update each parameter of the reinforcement learning model based on the training sample set;

[0022] repeat the above steps multiple times to update each parameter of the reinforcement learning model by the N-step parameter update method, save the model parameters, and obtain a final reinforcement learning model.

[0023] In some embodiments of the present application, the parameters include: policy network parameters, value network parameters, target value network parameters, and Q function network parameters.

[0024] In some embodiments of the present application, the Transformer model includes two encoders and one decoder.

[0025] In a second aspect, the embodiments of the present application provide a trajectory prediction-based reinforcement learning automatic driving decision device, comprising:

[0026] an acquisition module configured to acquire driving trajectory time series data of an automatic driving vehicle and other vehicles around the automatic driving vehicle in a historical time period;

[0027] a trajectory prediction module configured to input the driving trajectory time series data into a trained Transformer model to predict driving trajectories of the other vehicles around the automatic driving vehicle and obtain corresponding trajectory prediction information;

[0028] an automatic driving decision module configured to add the trajectory prediction information into a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes an automatic driving decision based on the target state space.

[0029] In some embodiments of the present application, the other vehicles around the automatic driving vehicle include vehicles in six directions of the automatic driving vehicle, i.e., front, left front, right front, back, left back, and right back.

[0030] In some embodiments of the present application, the Transformer model includes two encoders and one decoder.

[0031] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of the first aspect.

[0032] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer readable instructions executable by a processor to implement the method of the first aspect.

[0033] Compared with the prior art, the trajectory prediction-based reinforcement learning automatic driving decision method provided by the present application acquires driving trajectory time series data of an automatic driving vehicle and other vehicles around the automatic driving vehicle in a historical time period; inputs the driving trajectory time series data into a trained Transformer model to predict driving trajectories of the other vehicles around the automatic driving vehicle and obtain corresponding trajectory prediction information; adds the trajectory prediction information into a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes automatic driving decisions based on the target state space. It can be seen that the trajectory prediction information of the other vehicles is added to the state information of the reinforcement learning decision, thereby reducing the possibility of collision between the automatic driving vehicle and the other vehicles and achieving the purpose of improving the passing success rate of the automatic driving decision. BRIEF DESCRIPTION OF DRAWINGS

[0034] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the present application. Moreover, the same reference numerals are used throughout the same figures. In the drawings:

[0035] Figure 1 A flowchart of a trajectory prediction-based reinforcement learning automatic driving decision method provided by an embodiment of the present application is shown;

[0036] Figure 2 A schematic diagram of a trajectory prediction-based reinforcement learning automatic driving decision device provided by an embodiment of the present application is shown;

[0037] Figure 3 A schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0039] It should be noted that the technical terms or scientific terms used in the present application should be understood as their general meanings understood by those skilled in the art, unless otherwise specified.

[0040] In addition, the terms "first" and "second" and the like are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed or can optionally include other steps or units inherent to such processes, methods, products or devices.

[0041] Reference is made to Figure 1 , Figure 1 is a flowchart of a trajectory prediction-based reinforcement learning automatic driving decision method provided by an embodiment of the present application, comprising the following S101-S103:

[0042] S101, obtaining driving trajectory time series data of an automatic driving vehicle and other vehicles around the automatic driving vehicle in a historical time period;

[0043] The other vehicles around the automatic driving vehicle include vehicles in six directions of the automatic driving vehicle, i.e., front, left front, right front, back, left back and right back.

[0044] The driving trajectory time series data is time series data formed by driving trajectories of the vehicle at a set time interval. The time interval can be set to a specific time interval such as seconds, minutes, etc.

[0045] S102, inputting the driving trajectory time series data into a trained Transformer model to predict the driving trajectories of the other vehicles around the automatic driving vehicle and obtaining corresponding trajectory prediction information;

[0046] The Transformer model is a deep learning model based on self-attention mechanism, which processes sequence data by relying entirely on self-attention mechanism. The Transformer model mainly consists of two parts: Encoder (encoder) and Decoder (decoder). The Transformer model of the present application is an improved Transformer model, which specifically includes two encoders and one decoder, and the encoding and decoding process is described in detail below.

[0047] S103, adding the trajectory prediction information into a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes automatic driving decisions based on the target state space.

[0048] The training process of the reinforcement learning model is as follows:

[0049] 1. Initialize parameters of the reinforcement learning model; the parameters include: policy network parameters, value network parameters, target value network parameters, Q function network parameters; specifically, the reinforcement learning model can use the SAC algorithm, and the policy network parameters θ, the value network parameters ψ, the target value network parameters Q function network parameters φ of the SAC algorithm are initialized.

[0050] SAC algorithm, full name Soft Actor-Critic, is a deep learning-based reinforcement learning method, aiming to achieve a balance between exploration and exploitation while maintaining high entropy (i.e. randomness) of the policy. SAC combines the maximum entropy reinforcement learning concept and the actor-critic framework, and is an off-policy and deep reinforcement learning method.

[0051] 2. Obtain driving trajectory time series data samples of the autonomous vehicle and other vehicles around it;

[0052] Specifically, vehicle driving data can be collected in a simulation platform to obtain historical trajectory information of the autonomous vehicle and other vehicles around it, and the historical trajectory information is organized into driving trajectory time series data samples.

[0053] The driving trajectory time series data samples are input into the trained Transformer model to predict the driving trajectory of the other vehicles around the autonomous vehicle, and obtain corresponding trajectory prediction information samples;

[0054] The processing process of the Transformer model for input data is as follows:

[0055] The driving trajectory time series data is positionally encoded to generate sequence information that the reinforcement learning model can recognize, which is used to input into the improved Transformer model to calculate the trajectory information of other vehicles; the specific formula of position encoding is as follows:

[0056] Location={S,ΔS|S=(x0,y0),ΔS=(Δx i ,Δy i )};

[0057] D=Position(Location);

[0058] Where (x0, y0) represents the position information of the autonomous vehicle in the monitored road segment, (Δx i , Δy irespectively represent the relative distance of the surrounding vehicles i (i = 1, 2, 3, 4, 5, 6, representing the front, left front, right front, back, left back, and right back, respectively) and the ego vehicle on the horizontal and vertical axes. The original vehicle trajectory information L is calculated by the position encoding formula Position() and then input into the next layer of spatial interaction layer to obtain the ego vehicle and surrounding vehicle trajectory information D.

[0059] The specific expression of the position encoding formula Position() is not described here.

[0060] 3. Input the data obtained in step 2 into the encoding layer. The Transformer model designed in this application uses two different encoders to extract information about other vehicles. First, in the first encoder, the vehicle spatial interaction layer and the time interaction layer are entered. In the vehicle spatial interaction layer, the spatial transformer is used to extract the influence information of the surrounding vehicles on the ego vehicle trajectory, and in the time interaction layer, the time series transformer is used to extract the time series information between the historical trajectories of the autonomous vehicle.

[0061] In the vehicle spatial interaction layer, the differential influence of the surrounding vehicles on the ego vehicle trajectory is measured by calculating the relative distance change between the ego vehicle and the surrounding vehicles, where the relative distance is the relative distance between each surrounding vehicle and the ego vehicle center position after establishing a coordinate system, i.e. the Euclidean distance Si. Its expression is:

[0062]

[0063] where Δx i and Δy i represent the relative distance of the surrounding vehicles i (i = 1, 2, 3, 4, 5, 6, representing the front, left front, right front, back, left back, and right back, respectively) and the ego vehicle on the horizontal and vertical axes.

[0064] The expression of the change amount ΔS i of the relative distance is:

[0065] ΔS i = S i，t - S i，t+1 ;

[0066] where S i，t represents the relative distance of the surrounding vehicle i and the ego vehicle at time t, and S i，t+1 represents the relative distance of the surrounding vehicle i and the ego vehicle at time t+1.

[0067] For each time t, a spatial graph is constructed according to the vehicle position information: if the spatial distance between the target vehicle and the ego vehicle is less than d, it is considered that there is an edge between the two. For any feature vector h i, extract its query vector q i , key vector k i , and value vector v i . Define the information from j node to i node in graph convolution as:

[0068]

[0069] Based on this, the self-attention mechanism of the Transformer is redefined as message passing:

[0070]

[0071] In the time interaction layer, a normal transformer encoder is used.

[0072]

[0073]

[0074] Wherein, head j = Att j (Q i , K i , V i ).

[0075] For the i-th vehicle, its historical position information is combined into a Tx2 matrix (T is the time length, and 2 represents the x and y coordinates), and different feature matrices are extracted through three different fully connected networks. Among them, Q represents the query matrix, K represents the key matrix, and V represents the value matrix. Through the above formula, the self-attention between the positions of different time points of vehicle i can be calculated, so as to extract the time dimension motion pattern of vehicle i and complete the time series modeling.

[0076] The method for assigning trajectory influence weight factors according to the relative distance change amount of surrounding vehicles and the ego vehicle is to perform Softmax function normalization calculation on ΔSi to obtain the weight factor α i , and its expression is:

[0077]

[0078] The vehicle interaction layer performs Hadamard multiplication on the total weight factor A i calculated and the trajectory information D of the ego vehicle and the surrounding vehicles input into the encoder to obtain the vehicle trajectory information D 1 containing vehicle spatial interaction information, and its expression is:

[0079] A i = {α0, αi}

[0080] D 1 = D O A i

[0081] Then in the second encoder, first extract the spatial information, fuse the time and spatial information extracted in the last step, and then use a time sequence Transformer to make the final prediction.

[0082] Finally, enter the feedforward layer and perform residual calculation and normalization again. The obtained result is combined with a Gaussian noise and input into the decoder.

[0083] In the decoder, first go through the operation similar to the position encoding in step 2, then enter the mask layer, get the mask information and perform residual calculation and normalization, output the data containing the mask information, and input it into the multi-head attention mechanism layer together with the result obtained by the above encoder to get the result containing the prediction information. Before output, perform a feedforward layer calculation, residual calculation and normalization; After linear transformation, the prediction trajectory distribution probability is calculated by the maximum probability output layer, and the maximum probability output is selected to obtain the final prediction trajectory information.

[0084] 4. Add the trajectory prediction information sample to the state space corresponding to the reinforcement learning model to obtain the state space sample:

[0085] s t = (x t , y t , x it , y it , x t+T , y t+T , x it+T , y it+T , v xt , v yt , v xit , v yit );

[0086] T = 2.5s, 5s;

[0087] Wherein, i = 1, 2, 3, 4, 5, 6, respectively represent the front, left front, right front, back, left back, right back six directions; x, y represent the horizontal and vertical coordinates of the autonomous vehicle, Δx it , Δy it respectively represent the relative distance between the i-th car and the autonomous vehicle, T = 2.5s / 5s represents the vehicle trajectory prediction information after 2.5s / 5s; v represents the value in reinforcement learning;

[0088] Through the policy network a t ~ π θ(a t |s t Generate action a t ;

[0089] Through the state transition probability s t ~p(s t+1 |s t a t Get s t+1 ;

[0090] s t a t 、r(s t a t ), s t+1 Store it in the experience replay cache, where r represents the reward in reinforcement learning;

[0091] Sample a training sample set with batch size β from the experience replay cache;

[0092] The parameters of the reinforcement learning model are updated based on the training sample set. The specific update process is as follows:

[0093] Use -logπ θ (a t |s t Update the value network parameters ψ;

[0094] use Update the network parameters φ1 and φ2 of the two functions;

[0095] use Update the policy network parameters θ;

[0096] use Update target value network parameters

[0097] Repeat the above steps multiple times to train the reinforcement learning model with importance sampling data. Update each parameter of the reinforcement learning model using the N-step parameter update method, save the model parameters, and obtain the final reinforcement learning model.

[0098] The reinforcement learning-based autonomous driving decision-making method based on trajectory prediction provided in this application has the following beneficial effects:

[0099] The application utilizes an improved Transformer model to predict the driving trajectories of other vehicles in the scene, and makes automatic driving decisions through the obtained prediction information. The trajectory prediction information of other vehicles is added to the state information of the reinforcement learning decision, thereby reducing the possibility of collision between the vehicle and other vehicles, and achieving the purpose of improving the success rate of driving decisions. Therefore, by using the method provided by the application, the safety of automatic driving decisions can be effectively improved.

[0100] In the above embodiment, a trajectory prediction-based reinforcement learning automatic driving decision method is provided, and a trajectory prediction-based reinforcement learning automatic driving decision device is also provided. The trajectory prediction-based reinforcement learning automatic driving decision device provided in the embodiments of the application can implement the trajectory prediction-based reinforcement learning automatic driving decision method described above. The trajectory prediction-based reinforcement learning automatic driving decision device can be implemented by software, hardware or a combination of software and hardware. For example, the trajectory prediction-based reinforcement learning automatic driving decision device can include integrated or separate functional modules or units to perform the corresponding steps in the above methods. Please refer to Figure 2 As shown in the figure, the trajectory prediction-based reinforcement learning automatic driving decision device 10 provided in the embodiments of the application includes:

[0101] The acquisition module 101 is configured to acquire driving trajectory time series data of an autonomous vehicle and other vehicles around the autonomous vehicle in a historical time period.

[0102] The trajectory prediction module 102 is configured to input the driving trajectory time series data into a trained Transformer model to predict the driving trajectories of other vehicles around the autonomous vehicle, and obtain corresponding trajectory prediction information.

[0103] The automatic driving decision module 103 is configured to add the trajectory prediction information to a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes automatic driving decisions based on the target state space.

[0104] In some embodiments of the application, the other vehicles around the autonomous vehicle include vehicles in six directions of the autonomous vehicle, i.e., front, left front, right front, back, left back and right back.

[0105] In some embodiments of the application, the Transformer model includes two encoders and one decoder.

[0106] In some embodiments of the application, the device further includes:

[0107] The model training module is configured to train the reinforcement learning model, and the specific training process is as follows:

[0108] Initialize the parameters of the reinforcement learning model;

[0109] Obtain the driving trajectory time series data samples of the autonomous vehicle and other vehicles around the autonomous vehicle;

[0110] Input the driving trajectory time series data samples into the trained Transformer model to predict the driving trajectories of the other vehicles around the autonomous vehicle, and obtain corresponding trajectory prediction information samples;

[0111] Add the trajectory prediction information samples to the corresponding state space of the reinforcement learning model to obtain state space samples:

[0112] s t =(x t ,y t ,x it ,y it ,x t+T ,y t+T ,x it+T ,y it+T ,v xt ,v yt ,v xit ,v yit );

[0113] wherein i = 1, 2, 3, 4, 5, 6, respectively representing the front, left front, right front, rear, left rear, and right rear six directions; x and y represent the horizontal and vertical coordinates of the autonomous vehicle, respectively; Δx it and Δy it represent the relative distance between the i-th vehicle and the autonomous vehicle, respectively; T represents the time length; and v represents the value.

[0114] An action a t is generated through a policy network a t ~ π θ (a t | s t );

[0115] s t+1 is obtained through a state transition probability s t ~ p(s t+1 | s t , a t );

[0116] s t , a t , r(s t , a t ), s t+1storing the experience replay buffer, r represents a reward;

[0117] sampling a training sample set with a batch size of β from the experience replay buffer;

[0118] updating each parameter of the reinforcement learning model based on the training sample set;

[0119] repeating the above steps multiple times to update each parameter of the reinforcement learning model through the N-step parameter update method, saving the model parameters, and obtaining the final reinforcement learning model.

[0120] In some embodiments of the present application, the parameters include: policy network parameters, value network parameters, target value network parameters, and Q function network parameters.

[0121] The trajectory prediction-based reinforcement learning automatic driving decision device provided by the embodiments of the present application and the trajectory prediction-based reinforcement learning automatic driving decision method provided by the embodiments of the present application have the same beneficial effects as the methods they employ, run or implement.

[0122] The embodiments of the present application also provide an electronic device corresponding to the method provided by the preceding embodiments, which can be an electronic device for a client, such as a vehicle-mounted computer, a mobile phone, a notebook computer, a tablet computer, a desktop computer, etc., to execute the trajectory prediction-based reinforcement learning automatic driving decision method.

[0123] Please refer to Figure 3 which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As Figure 3 shown, the electronic device 20 includes a processor 200, a memory 201, a bus 202 and a communication interface 203, the processor 200, the communication interface 203 and the memory 201 are connected through the bus 202; the memory 201 stores a computer program executable on the processor 200, and the processor 200 executes the computer program to perform the trajectory prediction-based reinforcement learning automatic driving decision method provided by any one of the preceding embodiments of the present application.

[0124] Among them, the memory 201 can contain a high-speed random access memory (RAM: Random Access Memory), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which can be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. can be used.

[0125] The bus 202 can be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. The memory 201 is configured to store programs, and the processor 200 executes the programs after receiving execution instructions. The trajectory prediction based reinforcement learning automatic driving decision method disclosed in any of the embodiments of the present application can be applied to the processor 200 or implemented by the processor 200.

[0126] The processor 200 can be an integrated circuit chip with processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit or an instruction in the form of software in the processor 200. The processor 200 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), and the like; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a ready programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and combines the hardware to complete the steps of the above method.

[0127] The electronic device provided by the embodiments of the present application has the same beneficial effects as the trajectory prediction based reinforcement learning automatic driving decision method provided by the embodiments of the present application, and has the same beneficial effects as the method adopted, run or implemented by the electronic device.

[0128] The embodiments of the present application also provide a computer readable storage medium corresponding to the trajectory prediction based reinforcement learning automatic driving decision method provided by the preceding embodiments, and a computer program (i.e. program product) is stored on the computer readable storage medium. When the computer program is run by a processor, the trajectory prediction based reinforcement learning automatic driving decision method provided by any of the preceding embodiments is executed.

[0129] It should be noted that examples of the computer-readable storage medium can also include, but are not limited to, a phase change memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or other optical, magnetic storage medium, and the like, which will not be listed one by one here.

[0130] The computer-readable storage medium provided by the above embodiments of the present application has the same beneficial effects as the method adopted, run or implemented by the application program stored therein, based on the same inventive concept as the trajectory prediction-based reinforcement learning automatic driving decision method provided by the embodiments of the present application.

[0131] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should be covered in the scope of the claims and the specification of the present application.

Claims

1. A trajectory prediction based reinforcement learning automatic driving decision method, characterized in that, The method comprises the following steps: obtaining driving trajectory time series data of an autonomous vehicle and other vehicles around the autonomous vehicle in a historical time period; inputting the driving trajectory time series data into a trained Transformer model to predict the driving trajectories of the other vehicles around the autonomous vehicle, and obtaining corresponding trajectory prediction information; adding the trajectory prediction information into a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes autonomous driving decisions based on the target state space; the other vehicles around the autonomous vehicle include vehicles in six directions, i.e., front, left front, right front, back, left back and right back, of a position where the autonomous vehicle is located; the training process of the reinforcement learning model is as follows: initializing parameters of the reinforcement learning model; obtaining driving trajectory time series data samples of the autonomous vehicle and the other vehicles around the autonomous vehicle; inputting the driving trajectory time series data samples into the trained Transformer model to predict the driving trajectories of the other vehicles around the autonomous vehicle, and obtaining corresponding trajectory prediction information samples; adding the trajectory prediction information samples into a corresponding state space of the reinforcement learning model to obtain state space samples: ； Wherein, i = 1, 2, 3, 4, 5, 6, respectively represent the front, left front, right front, back, left back, right back six direction; x, y respectively represent the horizontal and vertical coordinates of the autonomous vehicle, T represents the length of time; Indicates the speed; By the policy network Generating actions ; By state transition probabilities Acquisition ; will be stored in the experience replay buffer, representing a reward, representing a reward; sample a batch size of training samples from the experience replay buffer; updating the parameters of the reinforcement learning model based on a training sample set; repeating the above steps multiple times, updating the parameters of the reinforcement learning model through an N-step parameter updating method, saving model parameters, and obtaining a final reinforcement learning model.

2. The method of claim 1, wherein, The parameters include policy network parameters, value network parameters, target value network parameters and Q function network parameters.

3. The method of claim 1, wherein, The Transformer model comprises two encoders and one decoder.

4. A trajectory prediction-based reinforcement learning automatic driving decision device, characterized by, The method comprises the following steps: a obtaining module is configured to obtain driving trajectory time series data of an autonomous vehicle and other vehicles around the autonomous vehicle in a historical time period; a trajectory prediction module is configured to input the driving trajectory time series data into a trained Transformer model to predict the driving trajectories of the other vehicles around the autonomous vehicle, and obtain corresponding trajectory prediction information; an autonomous driving decision module is configured to add the trajectory prediction information into a corresponding state space of a trained reinforcement learning model to obtain a target state space, so that the reinforcement learning model makes autonomous driving decisions based on the target state space; the other vehicles around the autonomous vehicle include vehicles in six directions, i.e., front, left front, right front, back, left back and right back, of a position where the autonomous vehicle is located; the training process of the reinforcement learning model is as follows: initializing parameters of the reinforcement learning model; obtaining driving trajectory time series data samples of the autonomous vehicle and the other vehicles around the autonomous vehicle; inputting the driving trajectory time series data samples into the trained Transformer model to predict the driving trajectories of the other vehicles around the autonomous vehicle, and obtaining corresponding trajectory prediction information samples; adding the trajectory prediction information samples into a corresponding state space of the reinforcement learning model to obtain state space samples: ； Wherein, i = 1, 2, 3, 4, 5, 6, respectively represent the front, left front, right front, back, left back, right back six direction; x, y respectively represent the horizontal and vertical coordinates of the autonomous vehicle, T represents the length of time; Indicates the speed; By the policy network Generating actions ; By state transition probabilities Acquisition ; will be stored in the experience replay buffer, representing a reward; representing a reward; sample a batch size of training samples from the experience replay buffer; updating the parameters of the reinforcement learning model based on a training sample set; The above steps are repeated multiple times to update each parameter of the reinforcement learning model through an N-step parameter updating method, save the model parameters, and obtain a final reinforcement learning model.

5. The apparatus of claim 4, wherein, The Transformer model comprises two encoders and one decoder.

6. An electronic device comprising: Memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 3.

7. A computer readable storage medium characterized in that, A computer readable medium having stored thereon computer readable instructions executable by a processor to implement the method of any one of claims 1 to 3.

Images