Vehicle state representation method and behavior decision method based on deep reinforcement learning

Abstract

This application relates to the field of autonomous driving decision-making technology, and particularly to a vehicle state representation method and behavior decision-making method based on deep reinforcement learning. The representation method includes: acquiring the state information of the vehicle itself and other vehicles, and concatenating them into a complete state vector; constructing a state encoding module to encode the complete state vector, obtaining multiple query, key, and value vectors, and calculating the correlation coefficient between the query vector and the key vector; adding a virtual vehicle to the complete state vector, and performing a weighted summation of the correlation coefficient and multiple value vectors to obtain a vehicle state vector that integrates all vehicle information; verifying the permutation invariance of the vehicle state vector, and after successful verification, inputting the vehicle state vector into a deep reinforcement learning network to obtain a deep reinforcement learning vehicle state representation model. This method can help autonomous vehicles better utilize reinforcement learning methods to make action decisions, thereby helping autonomous vehicles navigate more complex traffic scenarios.

Description

Technical Field

[0001] This application relates to the field of autonomous driving decision-making technology, and in particular to a vehicle state representation method and behavior decision-making method based on deep reinforcement learning. Background Technology

[0002] With the development of autonomous driving decision-making algorithms, an increasing number of end-to-end algorithm models are being applied in both academic and commercial fields of autonomous vehicle behavior decision-making. Currently, existing decision-making algorithms in the field of autonomous driving often struggle to meet the performance requirements of complex traffic scenarios. Especially in scenarios involving interaction with other random dynamic vehicles, traditional decision-making methods often fail to find the optimal solution for the vehicle's behavior due to excessive constraints. While existing end-to-end methods can handle relatively complex autonomous driving traffic scenarios, the sheer number of vehicle state and surrounding environment parameters makes it difficult to represent and process them as input to reinforcement learning models. Summary of the Invention

[0003] This application provides a vehicle state representation method and device based on deep reinforcement learning, a behavior decision-making method, device, vehicle, and storage medium based on the vehicle state representation method based on deep reinforcement learning, in order to solve the problem that existing decision-making methods for random dynamic vehicle interactions are difficult to solve for the optimal solution of the vehicle behavior, and the problem that existing end-to-end processing of complex autonomous driving traffic scenarios has too many vehicle state and surrounding vehicle environment parameters, which are difficult to represent.

[0004] The first aspect of this application provides a vehicle state representation method based on deep reinforcement learning, including the following steps: acquiring the state information of the vehicle itself and the state information of at least one other vehicle, and concatenating the state information of the vehicle itself and the state information of the other vehicle into a complete state vector;

[0005] A state encoding module is constructed, and the complete state vector is input into the state encoding module for encoding processing to obtain multiple query vectors, multiple key vectors and multiple value vectors.

[0006] Calculate the correlation coefficient between each query vector and each key vector;

[0007] A padding mask mechanism is used to add virtual vehicles to the complete state vector so that the sequence length of the complete state vector remains fixed;

[0008] The correlation coefficient and the plurality of value vectors are weighted and summed to obtain a vehicle state vector that integrates all vehicle information.

[0009] The permutation invariance of the vehicle state vector is verified. After the verification is passed, the vehicle state vector is input into the deep reinforcement learning network for training to obtain a deep reinforcement learning vehicle state representation model.

[0010] Optionally, the status information of the vehicle includes its lateral position, longitudinal position, speed, acceleration, and heading angle; the status information of the other vehicle includes the lateral position, longitudinal position, speed, acceleration, and heading angle of at least one other vehicle within the preset observation range of the vehicle.

[0011] Optionally, the state encoding module includes a fully connected encoding network and a self-attention mechanism unit, wherein the first layer is the fully connected encoding network, which encodes the complete state vector to obtain a vehicle encoding vector; the second layer is the self-attention mechanism unit, which calculates the vehicle encoding vector to obtain the plurality of query vectors, the plurality of key vectors and the plurality of value vectors respectively.

[0012] Optionally, the formula for calculating the correlation coefficient is:

[0013]

[0014] Among them, α i,j Let q be the correlation coefficient between the q-vector of the i-th vehicle and the k-vector of the j-th vehicle. i Let q be the query vector for the i-th vehicle. i ∈[1, 2, ..., N], where N is the maximum observable number of vehicles, k j Let k be the key vector of the j-th vehicle. i ∈[1, 2, ..., N], For q i and k j Dimensions.

[0015] Optionally, the step of adding virtual vehicles to the complete state vector using a padding mask mechanism to keep the sequence length of the complete state vector fixed includes:

[0016] Add the virtual vehicle to the complete state vector to obtain the filled state vector;

[0017] The mask vector in the padding masking mechanism is used to record the positions of the real vehicle and the virtual vehicle in the padding state vector, so that the sequence length of the complete state vector remains fixed.

[0018] Optionally, the step of weighted summing of the correlation coefficients and the plurality of value vectors to obtain a vehicle state vector that integrates all vehicle information includes:

[0019] The Softmax function is used to align the correlation coefficients between the same query vector and different key vectors;

[0020] The aligned correlation coefficients are weighted and summed with the multiple value vectors to obtain the vehicle state vector that integrates all vehicle information.

[0021] Optionally, the vehicle state vector is:

[0022]

[0023]

[0024] Among them, s' i This is the vehicle state vector. The correlation coefficient after alignment, v j Let α be the value vector of the j-th vehicle, j∈[1,2,...,N], where N is the maximum observable number of vehicles. i,j Let q be the correlation coefficient between the q vector of the i-th vehicle and the k vector of the j-th vehicle, ∑ z exp(α i,z ) is the correlation coefficient generated by the q vector of the i-th car and the z vector of all cars, z∈[1,2,...,N], and the sum after processing by the exp() function.

[0025] A second aspect of this application provides a vehicle state representation device based on deep reinforcement learning, comprising:

[0026] The first acquisition module is used to acquire the status information of the vehicle itself and the status information of at least one other vehicle, and to concatenate the status information of the vehicle itself and the status information of the other vehicle into a complete status vector.

[0027] A construction module is used to construct a state encoding module, which inputs the complete state vector into the state encoding module for encoding processing to obtain multiple query vectors, multiple key vectors and multiple value vectors.

[0028] The calculation module is used to calculate the correlation coefficient between each query vector and each key vector;

[0029] An addition module is used to add virtual vehicles to the complete state vector using a padding mask mechanism, so that the sequence length of the complete state vector remains fixed;

[0030] The fusion module is used to perform a weighted summation of the correlation coefficients and the multiple value vectors to obtain a vehicle state vector that fuses all vehicle information;

[0031] The training module is used to verify the permutation invariance of the vehicle state vector, so as to input the vehicle state vector into the deep reinforcement learning network for training and obtain a deep reinforcement learning vehicle state representation model.

[0032] A third aspect of this application provides a behavior decision-making method based on a deep reinforcement learning-based vehicle state representation method, comprising the following steps:

[0033] Obtain the status information of your own vehicle, the status information of at least one other vehicle, and its historical trajectory information;

[0034] The historical trajectory information of the other vehicle is input into the LSTM network to extract the hidden state of the other vehicle.

[0035] The hidden states of other vehicles are input into the deep reinforcement learning vehicle state representation model to extract the observable states of other vehicles.

[0036] The vehicle's state information is encoded using a fully connected layer. The encoded vehicle state information is then combined with the observable states of other vehicles and input into a deep reinforcement learning network to obtain the vehicle's decision-making actions in the current environment.

[0037] A fourth aspect of this application provides a behavior decision-making device based on a deep reinforcement learning-based vehicle state representation method, comprising:

[0038] The second acquisition module is used to acquire the status information of the vehicle itself, the status information of at least one other vehicle, and its historical trajectory information.

[0039] The first extraction module is used to input the historical trajectory information of the other vehicle into the LSTM network to extract the hidden state of the other vehicle.

[0040] The second extraction module is used to input the hidden state of other vehicles into the deep reinforcement learning vehicle state representation model to extract the observable state of other vehicles.

[0041] The decision-making module is used to encode the vehicle's state information using a fully connected layer, combine the encoded vehicle state information with the observable state of other vehicles, and input it into a deep reinforcement learning network to obtain the vehicle's decision-making action in the current environment.

[0042] A fifth aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a behavior decision method based on a deep reinforcement learning-based vehicle state representation method as described in the above embodiments.

[0043] A sixth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the behavior decision method of the vehicle state representation method based on deep reinforcement learning as described above.

[0044] The vehicle state representation method and apparatus based on deep reinforcement learning, the behavior decision-making method, apparatus, vehicle, and storage medium proposed in this application, acquire and process driving data of the vehicle and vehicles in the surrounding environment, and represent an observable state that can be used as input to a deep reinforcement learning network through dimension determination and self-attention processing modules. This solves the problem that existing decision-making methods for random dynamic vehicle interactions are difficult to solve for the optimal solution of the vehicle's behavior, as well as the problem that existing end-to-end processing of complex autonomous driving traffic scenarios has too many parameters of the vehicle state and the surrounding vehicle environment, making it difficult to represent. It also helps autonomous vehicles to better use reinforcement learning methods to make action decisions, thereby helping autonomous vehicles to pass through more complex traffic scenarios.

[0045] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0046] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0047] Figure 1 This is a flowchart of a vehicle state representation method based on deep reinforcement learning provided according to an embodiment of this application;

[0048] Figure 2 This is a schematic diagram illustrating the mechanism of a vehicle state representation method based on deep reinforcement learning provided according to an embodiment of this application.

[0049] Figure 3 The reward curves for the SE-DQN algorithm (observation state of the vehicle), SA-DQN algorithm (observation state of the vehicle after fusing the weekly vehicle states) and DQN algorithm provided in the embodiments of this application are shown.

[0050] Figure 4 A block diagram of a vehicle state representation device based on deep reinforcement learning according to an embodiment of this application;

[0051] Figure 5 This is a flowchart of a behavior decision-making method based on a deep reinforcement learning-based vehicle state representation method provided in an embodiment of this application.

[0052] Figure 6 This is a schematic diagram illustrating the mechanism of a behavior decision-making method based on a deep reinforcement learning-based vehicle state representation method provided in an embodiment of this application.

[0053] Figure 7A block diagram of a behavior decision-making device based on a deep reinforcement learning-based vehicle state representation method according to an embodiment of this application;

[0054] Figure 8 This is a structural schematic diagram of a vehicle according to an embodiment of this application. Detailed Implementation

[0055] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0056] Figure 1 This is a flowchart of a vehicle state representation method based on deep reinforcement learning provided according to an embodiment of this application.

[0057] like Figure 1 As shown, this vehicle state representation method based on deep reinforcement learning includes the following steps:

[0058] In step S101, the state information of the vehicle itself and the state information of at least one other vehicle are obtained, and the state information of the vehicle itself and the state information of the other vehicles are concatenated into a complete state vector.

[0059] The status information of the vehicle includes its lateral position, longitudinal position, speed, acceleration, and heading angle; the status information of other vehicles includes the lateral position, longitudinal position, speed, acceleration, and heading angle of at least one other vehicle within the preset observation range of the vehicle.

[0060] Specifically, the physical state characteristics of the vehicles are selected as the vehicle state s for each vehicle. i i = (0, 1, 2KK N), s0 represents the vehicle state of the vehicle, s1 to s N This represents the vehicle states of N other vehicles within the observation range of the vehicle itself; and sets the state vectors of all vehicles as s. i The vectors are concatenated to form a feature vector, which is then used as the observation vector S of the vehicle.

[0061] For example, in an intersection scenario, the vehicle state s of each vehicle can be characterized by its lateral position x, longitudinal position y, velocity v, acceleration a, and heading angle θ. i s i =[x i ,y i ,v i ,a i ,θ i i∈[0,N], and then concatenate them into a complete state vector S, S=[s0,s1,......sN ].

[0062] In step S102, a state encoding module is constructed. The complete state vector is input into the state encoding module for encoding processing to obtain multiple query vectors, multiple key vectors, and multiple value vectors.

[0063] Furthermore, the state encoding module includes a fully connected encoding network and a self-attention mechanism unit. The first layer is a fully connected encoding network, which encodes the complete state vector to obtain the vehicle encoding vector. The second layer is a self-attention mechanism unit, which calculates the vehicle encoding vector to obtain multiple query vectors, multiple key vectors, and multiple value vectors.

[0064] Specifically, such as Figure 2 As shown, in this embodiment of the application, the state s of each vehicle is first set... i After encoding by a fully connected coding network, the encoded vector a is obtained. i The encoded vector a i The three fully connected networks in the self-attention mechanism unit produce three different vectors, namely the query vector (q). i Key vector (key) k i Value vector (v) i ,in,

[0065] a i =W a s i

[0066] q i =W q a i

[0067] k i =W k a i

[0068] v i =W v a i .

[0069] In step S103, the correlation coefficient between each query vector and each key vector is calculated.

[0070] Specifically, the embodiments of this application use scaled dot-product to calculate each query vector q. i With key vector (key)k i The correlation coefficient α between them i,j The calculation formula is:

[0071]

[0072] Where, α i,j Let q be the correlation coefficient between the q-vector of the i-th vehicle and the k-vector of the j-th vehicle. i Let q be the query vector for the i-th vehicle. i ∈[1, 2, ..., N], where N is the maximum observable number of vehicles, k j Let k be the key vector of the j-th vehicle. i ∈[1, 2, ..., N], For q i and k j This dimension is used to prevent different key vectors from having significantly different correlation coefficients with the same query vector.

[0073] In step S104, a padding mask mechanism is used to add virtual vehicles to the complete state vector so that the sequence length of the complete state vector remains fixed.

[0074] Furthermore, a padding mask mechanism is used to add virtual vehicles to the complete state vector to keep the sequence length of the complete state vector fixed, including:

[0075] Add virtual vehicles to the complete state vector to obtain the filled state vector;

[0076] The mask vector in the padding masking mechanism is used to record the positions of real and virtual vehicles in the padding state vector, so that the sequence length of the complete state vector remains fixed.

[0077] Specifically, in order to remove the information redundancy introduced by the added virtual vehicles, the embodiments of this application, in vector S = [s0, s1, ... s N The area is filled with O = [s0, s1, ... s]. N s N+1 , ......, s′ M During the process, a padding mask mechanism is used, with a mask vector P = [p0, ..., p0] M This is used to record the positions of the real vehicles and the filled virtual vehicles in state O, ensuring that the length of the vehicle state sequence O input to the state encoding module remains a fixed value M.

[0078]

[0079] Additionally, the mask vector P is used to adjust the coefficient α. i,jThe correlation coefficient between all vehicles and the filled virtual vehicles is set to negative infinity. This ensures that after subsequent processing by the Softmax function, the weight coefficient of the virtual vehicles is 0, and they will not affect the output vehicle state vector.

[0080] In step S105, the correlation coefficient and multiple value vectors are weighted and summed to obtain a vehicle state vector that integrates all vehicle information.

[0081] Specifically, in the embodiments of this application, the Softmax function is used for the same query vector q. i With different key vectors k i Align the correlation coefficients between them; then align the aligned correlation coefficients. With multiple value vectors v i By performing a weighted summation, we obtain a vehicle state vector s' that incorporates information from all vehicles after processing with a self-attention mechanism. i Wherein, the vehicle state vector s' i for:

[0082]

[0083]

[0084] in, The correlation coefficient after alignment, v j Let α be the value vector of the j-th vehicle, j∈[1,2,...,N], where N is the maximum observable number of vehicles. i,j Let q be the correlation coefficient between the q vector of the i-th vehicle and the k vector of the j-th vehicle, ∑ z exp(α i,z ) is the correlation coefficient generated by the q vector of the i-th car and the z vector of all cars, z∈[1,2,...,N], and the sum after processing by the exp() function.

[0085] In step S106, the permutation invariance of the vehicle state vector is verified. After the verification is passed, the vehicle state vector is input into the deep reinforcement learning network for training to obtain the deep reinforcement learning vehicle state representation model.

[0086] Specifically, in this embodiment, only the output s'0 of the encoding module is used as the input of the subsequent policy network. Due to the processing of the self-attention mechanism, s'0 no longer contains only the vehicle state information of the vehicle itself, but is the sum of the vehicle state information and the state information of other vehicles under different weight coefficients:

[0087]

[0088] From the above formula, it can be seen that when the input sequence of the input encoding module is S = [s0, s2, ... s...], the input sequence is... N When the order of the vehicle state vectors changes, only the formula is altered. The order of the items does not affect the final summation result. Therefore, s'0 is used as the observation state O of the vehicle. stage_ego It possesses permutation invariance, meaning that under the same conditions, there is only one unique observed state O. stage_ego Meanwhile, the dimension of s'0 is the same as the encoded vector a obtained after linear layer encoding. i The dimensions are equal and are a fixed value. For the policy network, it is a fixed-dimensional input that can be used to train the network.

[0089] The state-around representation method, which integrates the coded states of surrounding vehicles, is based on the representation method that focuses on the vehicle's own state. i (i∈[1,N]) are accumulated, and the resulting vector is used as the observed state O of the vehicle. stage_around :

[0090] O stage_around =∑ i s′ i =s′0+s′1+……+s′ N

[0091] As can be seen from the above, even if the input sequence S = [s0, s1, ... s2]... N The order of the vehicle state vectors in the diagram changes, and the encoded state s' obtained after encoding by the state encoding module is obtained. i It will not change, similarly, by s' i The accumulated O stage_around It will not change, O stage_around With O stage_ego It also has the property of substitution invariance, O stage_around The dimension is also the same as that of the encoding vector a i The dimensions are equal and are constant.

[0092] like Figure 2 As shown, in this embodiment of the application, three actions are selected as the output of the algorithm: {acceleration, holding, and deceleration}, where the acceleration during acceleration is 3 m / s². 2 The acceleration during deceleration is -3m / s². 2It is important to note that only the vehicle's acceleration is controlled, not the steering wheel angle; the vehicle follows a fixed path through the intersection. In the scenario of an unprotected intersection, the basic DQN algorithm is used as a control algorithm, and a comparative analysis is performed with the two models proposed in this application: SE-DQN (state-ego) and SA-DQN (state-around). The performance parameters are shown in the table below, and the resulting reward curves are shown in the figure. Figure 3 As shown.

[0093] Algorithm performance at three-lane intersections without traffic lights

[0094] DQN 0.90 50.79 40.07s SE-DQN 0.75 30.23 5.51s SA-DQN 0.99 70.65 6.66s

[0095] In summary, this application proposes a vehicle state representation method based on deep reinforcement learning. By acquiring driving data of the vehicle and vehicles in the surrounding environment and performing data processing and analysis, and through dimension determination and self-attention processing modules, an observable state that can be used as input to a deep reinforcement learning network is represented. This helps autonomous vehicles to better use reinforcement learning methods to make action decisions, thereby helping autonomous vehicles to navigate more complex traffic scenarios.

[0096] Next, referring to the accompanying drawings, a vehicle state representation device based on deep reinforcement learning proposed according to an embodiment of this application is described.

[0097] Figure 4 This is a block diagram of a vehicle state representation device based on deep reinforcement learning according to an embodiment of this application.

[0098] like Figure 4 As shown, the vehicle state representation device 40 based on deep reinforcement learning includes: a first acquisition module 401, a construction module 402, a calculation module 43, an addition module 404, a fusion module 405, and a training module 406.

[0099] The system comprises the following modules: a first acquisition module 401, which acquires the state information of the vehicle itself and at least one other vehicle, and concatenates the state information of the vehicle itself and the other vehicles into a complete state vector; a construction module 402, which constructs a state encoding module, inputting the complete state vector into the state encoding module for encoding processing to obtain multiple query vectors, multiple key vectors, and multiple value vectors; a calculation module 403, which calculates the correlation coefficient between each query vector and each key vector; an addition module 404, which adds virtual vehicles to the complete state vector using a padding mask mechanism to keep the sequence length of the complete state vector fixed; a fusion module 405, which performs a weighted summation of the correlation coefficients and multiple value vectors to obtain a vehicle state vector that fuses the information of all vehicles; and a training module 406, which verifies the permutation invariance of the vehicle state vector, inputting the vehicle state vector into a deep reinforcement learning network for training to obtain a deep reinforcement learning vehicle state representation model.

[0100] It should be noted that the foregoing explanation of the vehicle state representation method based on deep reinforcement learning also applies to the vehicle state representation device based on deep reinforcement learning in this embodiment, and will not be repeated here.

[0101] According to the vehicle state representation device based on deep reinforcement learning proposed in the embodiments of this application, by acquiring driving data of the vehicle and vehicles in the surrounding environment and performing data processing and analysis, and through dimension determination and self-attention processing modules, an observable state that can be used as input to a deep reinforcement learning network is represented, which helps autonomous vehicles to better use reinforcement learning methods to make action decisions, thereby helping autonomous vehicles to pass through more complex traffic scenarios.

[0102] Figure 5 This is a flowchart of a behavior decision-making method based on a deep reinforcement learning-based vehicle state representation method according to an embodiment of this application.

[0103] like Figure 5 As shown, the behavior decision-making method based on deep reinforcement learning for vehicle state representation includes the following steps:

[0104] Step S501: Obtain the status information of the vehicle itself, the status information of at least one other vehicle, and its historical trajectory information.

[0105] Specifically, it acquires the lateral position, longitudinal position, speed, acceleration, and heading angle of its own vehicle and other vehicles, and acquires the historical trajectory information of other vehicles in T+1 frames.

[0106] Step S502: Input the historical trajectory information of other vehicles into the LSTM network to extract the hidden state of other vehicles.

[0107] Specifically, such as Figure 6As shown, in autonomous driving scenarios, other traffic participants often possess hidden states that are not directly observable, such as intentions to go straight, turn left, or turn right, or mild or aggressive driving strategies. This forces autonomous vehicles to confront a partially observable environment. Therefore, an LSTM network is used to obtain the hidden states of other traffic participants, transforming the partially observable environment into a fully observable one. Then, based on the MDP framework, a deep reinforcement learning algorithm is used to solve the problem. First, a vehicle state prediction module based on a long short-term memory network and attention mechanism is designed. This module combines the long short-term memory network with the aforementioned deep reinforcement learning-based vehicle state representation model to form the vehicle state prediction module. Features of the observable and hidden states of other vehicles are extracted using historical trajectory information.

[0108] Step S503: Input the hidden state of the other vehicle into the deep reinforcement learning vehicle state representation model to extract the observable state of the other vehicle.

[0109] In other words, the historical state information of other vehicles is processed and integrated into a form that can be used as input to a deep reinforcement learning vehicle state representation model to extract the observable state of other vehicles.

[0110] Step S504: Encode the vehicle state information using a fully connected layer, combine the encoded vehicle state information with the observable states of other vehicles, and input it into a deep reinforcement learning network to obtain the vehicle's decision-making action in the current environment.

[0111] In other words, the vehicle learns from the historical behavior of other vehicles and outputs the decisions that the vehicle needs to make in the current environment.

[0112] In summary, the behavior decision-making method based on deep reinforcement learning for vehicle state representation proposed in the embodiments of this application, which fuses and represents the states of the vehicle and other vehicles as suitable inputs for reinforcement learning models, can help autonomous vehicles make better action decisions using reinforcement learning methods, thereby enabling autonomous vehicles to navigate more complex traffic scenarios.

[0113] Figure 7 This is a block diagram of a behavior decision-making device based on a deep reinforcement learning-based vehicle state representation method according to an embodiment of this application.

[0114] like Figure 7 As shown, the behavior decision-making device 70 based on the deep reinforcement learning-based vehicle state representation method includes: a second acquisition module 701, a first extraction module 702, a second extraction module 703, and a decision-making module 704.

[0115] The second acquisition module 701 is used to acquire the vehicle's state information, the state information of at least one other vehicle, and its historical trajectory information. The first extraction module 702 is used to input the historical trajectory information of other vehicles into an LSTM network to extract their hidden states. The second extraction module 703 is used to input the hidden states of other vehicles into a deep reinforcement learning vehicle state representation model to extract their observable states. The decision module 704 is used to encode the vehicle's state information using a fully connected layer, combine the encoded vehicle state information with the observable states of other vehicles, and input this combination into a deep reinforcement learning network to obtain the vehicle's decision-making action in the current environment.

[0116] It should be noted that the foregoing explanation of the behavior decision-making method embodiment of the vehicle state representation method based on deep reinforcement learning also applies to the behavior decision-making device of the vehicle state representation method based on deep reinforcement learning in this embodiment, and will not be repeated here.

[0117] The behavior decision-making device based on the vehicle state representation method of deep reinforcement learning proposed in the embodiments of this application integrates the states of the vehicle itself and other vehicles and represents them as suitable inputs for reinforcement learning models. This can help autonomous vehicles make better action decisions using reinforcement learning methods, thereby helping autonomous vehicles to navigate more complex traffic scenarios.

[0118] Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. The electronic device may include:

[0119] The memory 801, the processor 802, and the computer program stored on the memory 801 and capable of running on the processor 802.

[0120] When the processor 802 executes the program, it implements the behavior decision method based on the deep reinforcement learning-based vehicle state representation method provided in the above embodiments.

[0121] Furthermore, electronic devices also include:

[0122] Communication interface 803 is used for communication between memory 801 and processor 802.

[0123] The memory 801 is used to store computer programs that can run on the processor 802.

[0124] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0125] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0126] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0127] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0128] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described behavior decision-making method based on deep reinforcement learning for vehicle state representation.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0131] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0133] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0134] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0136] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A vehicle state representation method based on deep reinforcement learning, characterized in that, Includes the following steps: Obtain the status information of your own vehicle and the status information of at least one other vehicle, and concatenate the status information of your own vehicle and the status information of the other vehicle into a complete state vector; A state encoding module is constructed, and the complete state vector is input into the state encoding module for encoding processing to obtain multiple query vectors, multiple key vectors, and multiple value vectors. The state encoding module includes a fully connected encoding network and a self-attention mechanism unit. The first layer is the fully connected coding network, which encodes the complete state vector to obtain the vehicle coding vector; The second layer is the self-attention mechanism unit, which calculates the vehicle encoding vector to obtain the plurality of query vectors, the plurality of key vectors and the plurality of value vectors respectively; Calculate the correlation coefficient between each query vector and each key vector; A padding mask mechanism is used to add virtual vehicles to the complete state vector so that the sequence length of the complete state vector remains fixed; The correlation coefficient and the plurality of value vectors are weighted and summed to obtain a vehicle state vector that integrates all vehicle information. The permutation invariance of the vehicle state vector is verified. After the verification is passed, the vehicle state vector is input into the deep reinforcement learning network for training to obtain a deep reinforcement learning vehicle state representation model.

2. The vehicle state representation method based on deep reinforcement learning according to claim 1, characterized in that, The vehicle's status information includes its lateral position, longitudinal position, speed, acceleration, and heading angle; The status information of other vehicles includes the lateral position, longitudinal position, speed, acceleration, and heading angle of at least one other vehicle within the preset observation range of the own vehicle.

3. The vehicle state representation method based on deep reinforcement learning according to claim 1, characterized in that, The formula for calculating the correlation coefficient is: in, For the first The car Vector and the first The car The correlation coefficient of vectors, For the first The query vector for vehicles. , To a considerable maximum number of vehicles, For the first The key vector of the vehicle. , for and Dimensions.

4. The vehicle state representation method based on deep reinforcement learning according to claim 1, characterized in that, The step of adding virtual vehicles to the complete state vector using a padding mask mechanism to keep the sequence length of the complete state vector fixed includes: Add the virtual vehicle to the complete state vector to obtain the filled state vector; The mask vector in the padding masking mechanism is used to record the positions of the real vehicle and the virtual vehicle in the padding state vector, so that the sequence length of the complete state vector remains fixed.

5. The vehicle state representation method based on deep reinforcement learning according to claim 1, characterized in that, The step of weighted summing of the correlation coefficients and the multiple value vectors to obtain a vehicle state vector that integrates all vehicle information includes: The Softmax function is used to align the correlation coefficients between the same query vector and different key vectors; The aligned correlation coefficients are weighted and summed with the multiple value vectors to obtain the vehicle state vector that integrates all vehicle information.

6. The vehicle state representation method based on deep reinforcement learning according to claim 5, characterized in that, The vehicle state vector is: in, This is the vehicle state vector. The correlation coefficient after alignment. For the first The value vector of the vehicle. , To a considerable maximum number of vehicles, For the first The car Vector and the first The car The correlation coefficient of vectors, For the first Vehicle generated Vector and all cars The correlation coefficient generated by vector calculation ,go through The sum after function processing.

7. A vehicle state representation device based on deep reinforcement learning, characterized in that, include: The first acquisition module is used to acquire the status information of the vehicle itself and the status information of at least one other vehicle, and to concatenate the status information of the vehicle itself and the status information of the other vehicle into a complete status vector. A construction module is used to construct a state encoding module. The complete state vector is input into the state encoding module for encoding processing to obtain multiple query vectors, multiple key vectors, and multiple value vectors. The state encoding module includes a fully connected encoding network and a self-attention mechanism unit. The first layer is the fully connected coding network, which encodes the complete state vector to obtain the vehicle coding vector; The second layer is the self-attention mechanism unit, which calculates the vehicle encoding vector to obtain the plurality of query vectors, the plurality of key vectors and the plurality of value vectors respectively; The calculation module is used to calculate the correlation coefficient between each query vector and each key vector; An addition module is used to add virtual vehicles to the complete state vector using a padding mask mechanism, so that the sequence length of the complete state vector remains fixed; The fusion module is used to perform a weighted summation of the correlation coefficients and the multiple value vectors to obtain a vehicle state vector that fuses all vehicle information; The training module is used to verify the permutation invariance of the vehicle state vector, so as to input the vehicle state vector into the deep reinforcement learning network for training and obtain a deep reinforcement learning vehicle state representation model.

8. A behavior decision-making method based on deep reinforcement learning for vehicle state representation, characterized in that, The vehicle state representation method based on deep reinforcement learning, as described in any one of claims 1-6, includes the following steps: Obtain the status information of your own vehicle, the status information of at least one other vehicle, and its historical trajectory information; The historical trajectory information of the other vehicle is input into the LSTM network to extract the hidden state of the other vehicle. The hidden states of other vehicles are input into the deep reinforcement learning vehicle state representation model to extract the observable states of other vehicles. The vehicle's state information is encoded using a fully connected layer. The encoded vehicle state information is then combined with the observable states of other vehicles and input into a deep reinforcement learning network to obtain the vehicle's decision-making actions in the current environment.

9. A behavior decision-making device based on a deep reinforcement learning-based vehicle state representation method, characterized in that, The vehicle state representation method based on deep reinforcement learning, as described in any one of claims 1-6, comprises: The second acquisition module is used to acquire the status information of the vehicle itself, the status information of at least one other vehicle, and its historical trajectory information. The first extraction module is used to input the historical trajectory information of the other vehicle into the LSTM network to extract the hidden state of the other vehicle. The second extraction module is used to input the hidden state of other vehicles into the deep reinforcement learning vehicle state representation model to extract the observable state of other vehicles. The decision-making module is used to encode the vehicle's state information using a fully connected layer, combine the encoded vehicle state information with the observable state of other vehicles, and input it into a deep reinforcement learning network to obtain the vehicle's decision-making action in the current environment.

10. A vehicle, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the behavior decision method of the vehicle state representation method based on deep reinforcement learning as described in claim 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the behavior decision method of the vehicle state representation method based on deep reinforcement learning as described in claim 8.

Images