An autonomous driving system, method and medium based on dynamic road structure information

Abstract

The application belongs to the technical field of automatic driving and relates to an automatic driving system, method and medium based on dynamic road structure information, the automatic driving system based on dynamic road structure information comprising: a road information construction module configured to extract road structure information; a normalized decision generation module configured to generate an automatic driving strategy according to the road structure information and send a strategy action corresponding to the automatic driving strategy to an automatic driving environment; the automatic driving environment configured to automatically drive according to the strategy action, obtain a state and a reward function at a next moment, and send the state and the reward function at the next moment to the road information construction module and the normalized decision generation module; and the normalized decision generation module configured to adjust the automatic driving strategy according to the real-time road structure, the state at the next moment and the reward function. The application can improve the result of the reinforcement learning decision algorithm for the combined effect of various road structures.

Description

TECHNICAL FIELD

[0001] The present application relates to an automatic driving system, method and medium based on dynamic road structure information, and belongs to the technical field of automatic driving. BACKGROUND

[0002] The decision-making technology of automatic driving needs to consider the long-term interaction with the traffic environment to perform decision actions. Existing automatic driving decision-making methods can be divided into two categories according to the interaction mode. In the current mainstream automatic driving decision-making algorithm, it is mainly divided into interactive or semi-interactive. The semi-interactive follows the two-layer logic of predicting first and then deciding, considers the corresponding factors of the environment, but ignores the interaction with the vehicle itself. And the interactive method based on the model, such as model predictive control, partially observable Markov decision or reinforcement learning method, obtains the hidden probability distribution from the historical data or obtains the corresponding neural network parameters.

[0003] For automatic driving decision-making, road structure has a significant impact on the behavior and decision action of environmental traffic participants. In engineering, compared with environmental targets with clear description and independent physical state (such as the position, speed and size of environmental people and vehicles), road structure with variable length, complex connection and difficult to describe is usually not added to the state input, but is ignored through normalized expression, or the preference for the road is added to the design of the loss function or the reward function. This method will ignore the influence of road structure, or can only be used under fixed structure. Therefore, different decision algorithms are designed for different road structures according to the actual performance of the decision, and the road structures with large differences are processed modularly. SUMMARY

[0004] In view of the above problems, the purpose of the present application is to provide an automatic driving system, method and medium based on dynamic road structure information, which can extract the influence of road geometric structure on decision-making under different road structures, and further realize the result improvement of reinforcement learning decision-making algorithm under the combined action of multiple road structures.

[0005] To achieve the above purpose, the present application provides the following technical scheme: an automatic driving system based on dynamic road structure information, comprising: a road information construction module, a normalized decision generation module and an automatic driving environment; the road information construction module is used for extracting road structure information; the normalized decision generation module is used for generating an automatic driving strategy according to the road structure information, and sending a strategy action corresponding to the automatic driving strategy to the automatic driving environment; the automatic driving environment is used for automatically driving according to the strategy action, obtaining a state and a reward function at the next moment, and sending the state and the reward function at the next moment to the road information construction module and the normalized decision generation module; the normalized decision generation module adjusts the automatic driving strategy according to the real-time road structure, the state at the next moment and the reward function.

[0006] Further, the state is described in a lane coordinate system, and the lane coordinate system takes the direction along the lane as the horizontal coordinate and the direction perpendicular to the lane as the vertical coordinate.

[0007] Further, the road information construction module comprises a road information coding submodule and an autonomous driving map storage submodule.

[0008] Further, the information generated by the road information coding submodule comprises road nodes and normalized decision state nodes.

[0009] Further, the method for updating the node information comprises completing two-hop road node information updating and performing linkage of the normalized decision state node and two road nodes.

[0010] x k+1 =σ(W.CONCAT(Agg(neighbour(x k ),x k ))

[0011] wherein x represents each node to be updated, W is a neural network parameter matrix, neighbour indicates a neighboring function, Agg is an aggregation function, k is the number of encoding aggregation and is set to 2, sigma is a nonlinear function, for example, a Relu function, and CONCAT represents matrix splicing.

[0012] Further, the road information construction module comprises a data caching submodule, a decision state coding submodule and a reinforcement learning strategy improvement submodule.

[0013] Further, the state tuple comprises a state at time t, a policy action, a reward function and a state at the next time.

[0014] Further, the decision state encoding submodule jointly encodes the environment original node and the node information aggregated with the road structure information through a multi-layer perception machine, inputs the encoded state into a reinforcement learning strategy promotion submodule to generate a policy action, the reinforcement learning adopts a neural network model, and a loss function thereof is:

[0015] L=L rl +αL s

[0016] Wherein, L rl is a loss function of reinforcement learning update, alpha is a change with training step number, L s is a state encoding auxiliary function;

[0017] L rl =r t +maxQ(s t+1 ,a)-Q(s t ,a)

[0018] Wherein, r t is the reward given by the environment to the reinforcement learning at t moment, s t is t moment state, a is a policy action, and Q is a reinforcement learning value function;

[0019]

[0020] Wherein, H function is information entropy, The state encoded through the road graph network, and p represents a probability distribution.

[0021] The application also discloses an automatic driving method based on dynamic road structure information, which is used for any one of the automatic driving systems based on dynamic road structure information, and comprises the following steps: extracting road structure information; generating an automatic driving policy according to the road structure information, and generating a corresponding policy action according to the automatic driving policy; performing automatic driving according to the policy action, obtaining a state and a reward function at a next moment, and feeding back the state and the reward function at the next moment to a road information construction module and a normalized decision generation module; the road information construction module judges whether the decision needs to be adjusted according to the feedback result, and if so, sends corresponding road information encoding to the normalized decision generation module to generate a new policy action.

[0022] The application also discloses a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0023] The application has the following advantages due to the above technical scheme.

[0024] 1. The application can realize the influence of road geometry extraction on decision-making under different road structures, and further realize the result improvement of the reinforcement learning decision-making algorithm of multiple road structure combination.

[0025] 2. After the scheme training is completed in the application, when the normalized decision-making algorithm is used, the road structure can not be considered, the state is directly input into the state coding module and then input into the reinforcement learning module, and the normalized decision-making result is output. When the decision-making method with road structure influence information is used, according to the current position of the vehicle, the road information in the automatic driving map is read, the state coding module is performed after the aggregation of the ego state point, and finally the reinforcement learning module is input to output the decision-making result considering the road information. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a structure schematic diagram of an autonomous driving system based on dynamic road structure information in an embodiment of the application;

[0027] Figure 2 is a road network structure schematic diagram in an embodiment of the application;

[0028] Figure 3 is a road node and normalized decision-making state node connection schematic diagram in an embodiment of the application. DETAILED DESCRIPTION

[0029] In order for those skilled in the art to better understand the technical solutions of the application, the application is described in detail through specific embodiments. However, it should be understood that the specific embodiments are provided only for better understanding of the application, and they should not be understood as a limitation on the application. In the description of the application, it should be understood that the terms used are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0030] In existing technologies, autonomous driving methods, such as reinforcement learning, require pre-defined fixed state spaces and input dimensions. The complex and variable road structures make these methods difficult to represent, limiting their generalization and transferability. Therefore, effectively acquiring the impact of road structure information on decision-making and traffic participants is crucial for improving overall decision-making performance. To address these technical issues, this invention proposes an autonomous driving system, method, and medium based on dynamic road structure information. The policy generation process is divided into a normalized decision generation module and a road information construction module. The normalized decision generation module includes three sub-modules: a data cache, decision state encoding, and reinforcement learning strategy. The road information construction module includes two sub-modules: road information encoding and autonomous driving map storage. Both modules interact with the autonomous driving environment simultaneously. This enables the extraction of the impact of road geometry on decision-making under different road structures, thereby improving the results of reinforcement learning decision-making algorithms that leverage the combined effects of multiple road structures. Furthermore, after the scheme training is complete, when using the normalized decision-making algorithm, the road structure can be disregarded; the state can be directly input into the state encoding module and then into the reinforcement learning module, outputting a normalized decision result. When using a decision-making method incorporating road structure influence information, the road information in the autonomous driving map is read based on the vehicle's current location, aggregated with the vehicle's state points, processed by a state encoding module, and finally input into a reinforcement learning module to output a decision result that considers the road information. The following detailed description of the invention, with reference to the accompanying drawings, illustrates the solution in detail through embodiments.

[0031] Example 1

[0032] This invention discloses an autonomous driving system based on dynamic road structure information, such as... Figure 1 As shown, it includes: a road information construction module, a normalized decision generation module, and an autonomous driving environment; the road information construction module is used to extract road structure information; the normalized decision generation module is used to generate an autonomous driving strategy based on the road structure information and send the strategy actions corresponding to the autonomous driving strategy to the autonomous driving environment; the autonomous driving environment is used to perform autonomous driving based on the strategy actions, obtain the state and reward function of the next moment, and send the state and reward function of the next moment to the road information construction module and the normalized decision generation module; the normalized decision generation module adjusts the autonomous driving strategy based on the real-time road structure, the state and reward function of the next moment.

[0033] The road information construction module includes a road information encoding submodule and an autonomous driving map storage submodule. The road information encoding submodule is used to extract the differences in vehicle behavior in the state space caused by differences in road structure. The autonomous driving map storage submodule is used to store the information generated by the road information encoding submodule. In this embodiment, the state is described using a lane coordinate system, with the direction along the lane as the horizontal axis and the direction perpendicular to the lane as the vertical axis.

[0034] The road information construction module includes: a data caching submodule, a decision state encoding submodule, and a reinforcement learning policy enhancement submodule. The data caching submodule stores the state tuples used to update decisions. The decision state encoding submodule aggregates the original environmental nodes and road structure information to generate node information and performs joint encoding on the node information. The reinforcement learning policy enhancement submodule performs reinforcement learning on the encoded states to ultimately generate policy actions. In this embodiment, the state tuple includes: the state at time t, the policy action, the reward function, and the state at the next time step, i.e., the state tuple is (s...). t ,a t ,r t ,s t+1 ).

[0035] like Figure 1 and 2 As shown, the state of the autonomous driving environment includes the physical state attributes of the autonomous vehicle, surrounding vehicles, environmental targets, traffic rules, and other targets, such as their position, speed, and acceleration. This is labeled as s. t The subscript 't' represents time, and multiple graph network nodes are used to represent this. In this embodiment, the speed and position of the autonomous vehicle and the surrounding vehicle are used as examples of the state space. The autonomous driving environment receives a decision action 'a'. t Then, output the state and reward r at the next time step. t The reward function remains unchanged across all environments. All states are described using a lane coordinate system, which describes the state along the lane and perpendicular to the lane. It should be noted that although this embodiment uses the speed and position of the vehicle and the surrounding vehicles as the state space, this is only an example and does not preclude the possibility that the state space may include other states.

[0036] The system comprises a road information encoding submodule and an autonomous driving map storage submodule. The road information encoding submodule extracts the differences in vehicle behavior in the state space caused by differences in road structure. The autonomous driving map storage submodule stores the information generated by the road information encoding submodule. First, a memory for storing this information is constructed within the autonomous driving map storage submodule. A dynamic information storage layer is added to the autonomous driving map storage submodule. A schematic diagram of the road network structure in this embodiment is shown below.Figure 2 The road information encoding sub-module generates information including road nodes and normalized decision state nodes, the road nodes being a lane-level road network, nodes established in the middle of each road, each node having a serial number and attributes, Figure 2 The serial number of the node is the serial number of the node, and the attributes consist of road information encoding and connection mode. The connection mode is the link of each node with all drivable direction nodes, which is shown by the thick black lines in Figure 2 . The nodes where the two thick black lines intersect can connect the current lane and the left forward lane, and can be further connected with the next node. In this embodiment, two-hop connection is used as the encoding input. For a given node node i , the dimension is V∈R N×2 , where N is the number of nodes involved, and 2 is the lane coordinate position relative to the node node i . The adjacency matrix Adj∈R n×n , when two nodes are connected, Adj[i,j]=1, and the rest are 0. On the basis of the above, the data buffer module is used to store the state tuples used to update the decision, which has a maximum storage capacity of N, and the state tuple is (s t , a t , r t , s t+1 ), respectively, the state at time t, the action, the reward and the state at the next time.

[0037] Figure 3 is a connection diagram of road nodes and normalized decision state nodes, and the dimension of the normalized decision state node is N v ×4, N v is the number of environment vehicles and the number of nodes. The normalized decision state node is connected with the two nearest road nodes.

[0038] The road nodes and the normalized decision state nodes are encoded into the same dimension using a multi-layer perception, and in this embodiment, each node is defined to be encoded into 4 dimensions, and the parameters of the two multi-layer perceptions are θ r , θ n . After initial state encoding, the node information is updated using the graphSage method. The specific method for updating the node information is:

[0039] Two-hop road node information is updated;

[0040] The normalized decision state node is linked with the two road nodes, and the update function of the two-hop road node information is:

[0041] x k+1 =σ(W.CONCAT(Agg(neighbour(x​k ),x k ))

[0042] wherein, x represents each node that needs to be updated, W is a neural network parameter matrix, neighbour refers to a neighboring function, Agg is an aggregation function, an average function is used as the aggregation function in the embodiment, k is the number of encoding aggregation and is set to 2, sigma is a nonlinear function, for example, a Relu function, and CONCAT means that the matrices are spliced. The autonomous driving map storage submodule in the road information construction module stores the encoding result of each road node before aggregation with the normalized decision state node into the autonomous driving map.

[0043] The decision state encoding submodule jointly encodes the environment original node and the node information aggregated with the road structure information through a multilayer perception machine, and the parameters of the multilayer perception machine are theta s , and the purpose is to unify the decision state space. The encoded state is input into the reinforcement learning policy improvement submodule to generate a policy action, and the reinforcement learning takes deep Q learning as an example, and the reinforcement learning network parameters are theta rl , but the reinforcement learning model in the present application is not limited to deep Q learning.

[0044] Deep Q learning is a kind of neural network model, in which all neural network parameters are associated with each other and can be trained end-to-end as a whole. In the present embodiment, the scene distinguishing function module and the reinforcement learning traditional reward module are used for training, and the loss function is:

[0045] L=L rl +alpha L s

[0046] wherein, L rl is the loss function of reinforcement learning update, alpha is a variable that gradually converges to 0 with the training step number, L s is a state encoding auxiliary function; L rl is the loss function of reinforcement learning update, and taking deep Q learning as an example, the function is specifically:

[0047] L rl =r t +maxQ(s t+1 ,a)-Q(s t ,a)

[0048] wherein, r t is the reward given by the environment to the reinforcement learning at time t, s t is the state at time t, a is the policy action, and Q is the reinforcement learning value function.

[0049]

[0050] Where H is the information entropy. The state refers to the state after the road map network is encoded, and p represents the probability distribution.

[0051] After training, when using a normalized decision-making algorithm, road structure can be disregarded; the state is directly input into the state encoding module and then into the reinforcement learning module, outputting a normalized decision result. When using a decision-making method that incorporates road structure information, road information from the autonomous driving map is read based on the vehicle's current position, aggregated with the vehicle's state points, processed by the state encoding module, and finally input into the reinforcement learning module, outputting a decision result that considers road information.

[0052] Example 2

[0053] Based on the same inventive concept, this embodiment discloses an autonomous driving method based on dynamic road structure information, used in any of the above-mentioned autonomous driving systems based on dynamic road structure information, comprising the following steps:

[0054] S1 extracts road structure information;

[0055] S2 generates an autonomous driving strategy based on road structure information and generates corresponding strategy actions based on the autonomous driving strategy.

[0056] S3 performs autonomous driving based on policy actions, obtains the state and reward function of the next moment, and feeds back the state and reward function of the next moment to the road information construction module and the normalized decision generation module;

[0057] The S4 road information construction module determines whether the decision needs to be adjusted based on the feedback results. If so, it sends the corresponding road information code to the normalized decision generation module to generate a new strategy action.

[0058] Example 3

[0059] Based on the same inventive concept, this embodiment discloses a computer-readable storage medium storing a computer program, which is executed by a processor to implement the above-described autonomous driving method based on dynamic road structure information.

[0060] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0062] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0063] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific embodiments of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the claims of the present invention. The above content is only a specific embodiment of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An autonomous driving system based on dynamic road structure information, characterized in that, include: Road information construction module, normalized decision generation module, and autonomous driving environment; The road information construction module is used to extract road structure information; The state of the autonomous driving environment includes the physical state attributes of the autonomous vehicle, the surrounding vehicle, the environmental target, and the traffic rule target, such as the position, speed, and acceleration. The normalized decision generation module is used to generate an autonomous driving strategy based on the road structure information and send the strategy actions corresponding to the autonomous driving strategy to the autonomous driving environment. The autonomous driving environment is used to perform autonomous driving according to the policy action, obtain the state and reward function of the next moment, and send the state and reward function of the next moment to the road information construction module and the normalized decision generation module; The normalized decision generation module adjusts the autonomous driving strategy based on the real-time road structure, the state at the next moment, and the reward function; The normalized decision generation module includes: a data caching submodule, a decision state encoding submodule, and a reinforcement learning policy enhancement submodule. The data caching submodule is used to store state tuples used to update decisions. The decision state encoding submodule is used to aggregate the original environmental nodes and road structure information to generate node information, and to jointly encode the node information. The reinforcement learning policy enhancement submodule is used to perform reinforcement learning on the encoded state to finally generate policy actions. The road information construction module includes: a road information encoding submodule and an autonomous driving map storage submodule. The road information encoding submodule is used to extract the differences in vehicle behavior in the state space caused by differences in road structure. The autonomous driving map storage submodule is used to store the information generated by the road information encoding submodule. The information generated by the road information coding submodule includes: road nodes and normalized decision state nodes. The road nodes are lane-level road networks, with nodes established in the middle of each road. Each node has a sequence number and attributes, and the attributes consist of road information coding and connection methods. The connection methods are links between each node and all nodes in the drivable directions. The road nodes and normalized decision state nodes are initially coded to the same dimension using a multilayer perceptron. After the initial state coding, the node information is updated using the graphSage method. The decision state encoding submodule jointly encodes the original environmental node information and the node information aggregated with road structure information using a multilayer perceptron. The encoded state is then input into the reinforcement learning policy enhancement submodule to generate policy actions. The reinforcement learning employs a neural network model with the following loss function: in, To reinforce the loss function for learning updates, To vary with the number of training steps, It is a state coding auxiliary function; in, It is the environment that provides the reward for reinforcement learning at time t. It is the state at time t. Q is the policy action, and Q is the reinforcement learning value function. It is the state at the next moment; in, The function is information entropy. This refers to the state after road map network encoding. This represents a probability distribution.

2. The autonomous driving system based on dynamic road structure information as described in claim 1, characterized in that, The state is described using a lane coordinate system, where the horizontal axis is the direction along the lane and the vertical axis is the direction perpendicular to the lane.

3. The autonomous driving system based on dynamic road structure information as described in claim 2, characterized in that, The method for updating node information is as follows: Complete the update of road node information for the two jumps; Establish normalized links between the decision state node and the two road nodes. The update function for two-hop road node information is: in, This represents each node that needs to be updated. For the neural network parameter matrix, Adjacent functions For aggregate functions, The number of encoding aggregations is set to 2. It is a nonlinear function. This indicates that a matrix is ​​being concatenated.

4. The autonomous driving system based on dynamic road structure information as described in claim 3, characterized in that, The state tuple includes: the state at time t, the policy action, the reward function, and the state at the next time step.

5. An autonomous driving method based on dynamic road structure information, characterized in that, An autonomous driving system based on dynamic road structure information as described in any one of claims 1-4, comprising the following steps: Extract road structure information; An autonomous driving strategy is generated based on the road structure information, and corresponding strategy actions are generated based on the autonomous driving strategy. The system performs autonomous driving based on the strategy actions, obtains the state and reward function at the next moment, and feeds back the state and reward function at the next moment to the road information construction module and the normalized decision generation module. The road information construction module determines whether the decision needs to be adjusted based on the feedback results. If so, it sends the corresponding road information code to the normalized decision generation module to generate a new strategy action.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the autonomous driving method based on dynamic road structure information as described in claim 5.

Images