Road train dynamic road right distribution method based on reinforcement learning and rule constraint

Abstract

The invention relates to the technical field of intelligent traffic systems and automatic driving, in particular to a road train dynamic road right distribution method based on reinforcement learning and rule constraint, which comprises the following steps: acquiring state information of each vehicle in a road environment where a road train is located, and establishing state characteristics of each vehicle; conflict detection is carried out, and a conflict matrix representing the conflict relation between the road train and other vehicles is obtained; graph attention mechanism processing is carried out on the state features of all the vehicles, and global state features are obtained based on the cooperative relation between the road train and other vehicles; establishing a multi-vehicle collaborative decision-making model, inputting the global state features into the multi-vehicle collaborative decision-making model, and obtaining a driving decision-making intention of each vehicle; the driving decision intentions of all the vehicles are screened according to the conflict matrix, breadth-first search is carried out, and a driving path planned for the road train is obtained; according to the method, the road right optimization efficiency and safety in the multi-vehicle conflict scene can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent transportation systems and automatic driving, in particular to a highway train dynamic right-of-way allocation method based on reinforcement learning and rule constraints. BACKGROUND

[0002] With the development of intelligent transportation systems, unmanned driving technology in highway scenarios is increasingly becoming a key to improving the operational efficiency and safety of highway trains. However, the current traditional autonomous vehicles have problems such as rigid right-of-way allocation, low traffic efficiency, and high safety risks, which cannot meet the requirements of complex road operating scenarios and all-weather high-speed operation, and lack real-time collaborative decision-making capabilities. By combining reinforcement learning with traditional rule constraints, the real-time allocation of right-of-way is realized while meeting the need to improve traffic efficiency.

[0003] In the prior art, the Chinese patent application with publication number CN120199073A discloses a networked intersection ecological guidance cooperative control method based on a deep Q network, which outputs a phase release sequence and a green light start time by combining a deep reinforcement learning model, thereby optimizing intersection traffic efficiency. The Chinese patent application with publication number CN119479283A discloses a vehicle right-of-way decision and speed guidance method for a signal-free intersection, which divides vehicle fleets and establishes a right-of-way optimization model to determine the traffic order, thereby realizing conflict-free vehicle traffic. However, the real traffic environment is complex, and only the environmental information of the vehicle is considered, ignoring the interaction between the environment and the vehicle, making it difficult to dynamically adjust the vehicle right-of-way in different scenarios. SUMMARY

[0004] In view of the above problems, the present application provides a highway train dynamic right-of-way allocation method based on reinforcement learning and rule constraints, which solves the technical problem of insufficient right-of-way optimization efficiency and safety in the prior art in the multi-vehicle conflict scenario.

[0005] The present application provides a highway train dynamic right-of-way allocation method based on reinforcement learning and rule constraints, comprising the following steps: Step S1, obtaining the state information of each vehicle in the road environment of the highway train, the state information including a sequence of the position and heading of the vehicle before the current time; establishing the state features of each vehicle according to the state information; Step S2, performing conflict detection based on the state information to obtain a conflict matrix representing the conflict relationship between the highway train and other vehicles; Step S3, performing graph attention mechanism processing on the state features of each vehicle to obtain global state features based on the cooperative relationship between the highway train and other vehicles; Step S4, a multi-vehicle cooperative decision-making model is established, global state features are input into the multi-vehicle cooperative decision-making model, and driving decision intentions of each vehicle are obtained; Step S5, the driving decision intentions of each vehicle are screened by the conflict matrix, a breadth-first search is performed according to the screened driving decision intentions and state information of each vehicle, and a driving path planned for the highway train is obtained.

[0006] Preferably, step S1 specifically comprises: Step S1-1, a historical trajectory sequence is formed based on a plurality of headings and positions of the vehicle before the current time; Step S1-2, the position coordinates of the vehicle in the historical trajectory sequence are converted from a global coordinate system to a vehicle current position coordinate system; Step S1-3, the converted historical trajectory sequence is processed to obtain state features of each vehicle.

[0007] Preferably, in step S1-1, the expression of the heading and the position is:

[0008]

[0009]

[0010]

[0011]

[0012] wherein, represents the Euclidean distance between the vehicle and the vehicle , represents the current position vector of the vehicle , represents the current position vector of the vehicle , represents the modulus of the vector; represents the heading difference between the vehicle and the vehicle , and respectively represent the direction vectors of the historical trajectories of the vehicle and the vehicle , represents the cross product of vectors, represents the azimuth angle of the vehicle relative to the vehicle , represents the position vector from the vehicle to the vehicle ; In step S1-2, the expression of coordinate system conversion is:

[0013]

[0014] wherein, is the coordinate and heading angle in the global coordinate system of the autonomous vehicle, is the coordinate and heading angle in the global coordinate system of the traffic participant, is the coordinate and heading angle in the local coordinate system of the traffic participant.

[0015] Preferably, the step S2 specifically comprises: Step S2-1, conflict detection based on the state information, determining and recording the conflict relationship between other vehicles and the highway train; Step S2-2, establishing a conflict matrix from the conflict relationship, including: if there is a conflict between other vehicles and the highway train, set the corresponding element value in the conflict matrix to 1, otherwise set it to 0.

[0016] Preferably, the step S3 specifically comprises: Step S3-1, obtaining the attention weight between each vehicle based on the state characteristics of each vehicle; Step S3-2, constructing a dynamic collaboration graph using the attention weight; Step S3-3, obtaining global state features based on the dynamic collaboration graph.

[0017] Preferably, the step S3-2 specifically comprises: constructing a dynamic collaboration graph using the calculated attention weight The dynamic collaboration graph consists of a node set and an edge set The node represents the th vehicle in the scene, wherein indicates the index of the node, is the total number of vehicles; When the attention weight is greater than or equal to a preset threshold, it is considered that there is an effective collaboration relationship between the two intelligent agents, and the corresponding edge is established in the graph; when the attention weight is less than the threshold, no edge connection is established.

[0018] Preferably, the step S4 specifically comprises: Step S4-1, determining the reward function of the multi-vehicle cooperative decision-making model, including: Step S4-1-1, initializing the state and action space for each vehicle in the current traffic scene; Step S4-1-2, according to the real-time position and speed of the vehicle in the state and action space, calculating the collision risk, and calculating the collision reward value accordingly; Step S4-1-3, according to the deviation degree of the current speed of the vehicle from the ideal speed, calculating the speed reward value; Step S4-1-4, according to the first-in first-out principle, obtaining the right-of-way sequence of the vehicle, and determining the right-of-way reward value; Step S4-1-5, weighting and summing the collision reward, the speed reward and the right-of-way reward to obtain the final reward function; Step S4-2, establishing a multi-vehicle cooperative decision-making model based on the reward function, inputting global state features into the multi-vehicle cooperative decision-making model, and obtaining driving decision intentions of each vehicle.

[0019] Preferably, in step S4-1-4, the expression of the right-of-way reward value is:

[0020] Among them, represents the right-of-way reward value, represents the low-priority right-of-way reward, represents the high-priority right-of-way reward; In step S4-1-5, the expression of the reward function is:

[0021] Among them, 、 、 respectively represent the weight coefficients of the collision reward, the speed reward and the right-of-way reward, respectively represent the collision reward, the speed reward and the right-of-way reward, represents the total reward value of the intelligent agent.

[0022] Preferably, the step S5 specifically comprises: Step S5-1, screening the driving decision intentions of each vehicle according to the conflict matrix, including: if there is no conflict between a vehicle and the host vehicle, i.e. the value of the corresponding element in the conflict matrix is 0, the driving decision intention of the vehicle is removed; Step S5-2, performing breadth-first search according to the screened driving decision intentions to obtain a driving path planned for the highway train.

[0023] Preferably, the step S5-2 specifically comprises: Step S5-2-1, generating a candidate driving path based on breadth-first search according to the state information of each vehicle, and for the current ​a certain key point pair of a lane is found a key point pair of a lane; the key point pair represents a sampling point on a lane center line; step S5-2-2, judging whether the key point pair of a lane meets the driving decision intention of a vehicle, if not, returning to step S5-2-1; otherwise, adopting the key point pair of a lane to update the key point pair of a lane to update a track sequence where the key point pair of a lane is located; the track sequence represents a sequence composed of continuous key point pairs; step S5-2-3, when all key point pairs do not have a connected key point pair, the obtained track sequence is saved as a driving path of a highway train planning.

[0024] Compared with the prior art, the present application has at least the following beneficial effects: (1) The present application obtains a conflict matrix through conflict detection, establishes a multi-vehicle cooperative decision-making model, designs a multi-objective reward function, and combines a conflict prediction mechanism. By inputting the state information of the vehicle and the conflict relationship between vehicles into the decision-making model in real time, the distribution of road rights can be dynamically adjusted according to the information, overcoming the problem of rigid allocation scheme of traditional methods, so as to better adapt to the complex and changeable road environment, and improve the rationality of allocation and the safety of traffic.

[0025] (2) The present application uses a graph attention mechanism to process the state features of the vehicle, and combines a multi-head self-attention mechanism, which can comprehensively and efficiently extract and fuse the state information of multiple vehicles, and obtain the global state features of the highway train and other vehicles. This mechanism significantly improves the understanding ability of the model for the relationship between multiple vehicles in a complex traffic environment, speeds up the decision-making process, and effectively improves the real-time performance and decision-making efficiency of road right allocation.

[0026] (3) The present application uses the conflict matrix to screen the driving decision intention of each vehicle, and combines the state information to optimize the driving path of the highway train using breadth-first search. Through dynamic planning and screening of the driving path of the vehicle, effective coordination can be carried out according to the driving intention, state and priority between vehicles, so as to realize efficient avoidance and smooth passing between multiple vehicles, improve the overall road use efficiency, and reduce the probability of traffic conflicts. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated herein and constitute a part of this application.

[0028] Figure 1 The flowchart of the highway train dynamic road right allocation method based on reinforcement learning and rule constraints provided by the present application.

[0029] Figure 2A coordinate conversion schematic diagram provided by the present application.

[0030] Figure 3 A breadth-first search algorithm flowchart provided by the present application. DETAILED DESCRIPTION

[0031] In order to enable persons skilled in the art to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0032] The present application proposes a highway train dynamic right allocation method based on reinforcement learning and rule constraints. Through multi-agent reinforcement learning model construction, conflict prediction, graph attention mechanism and path planning steps, efficient and accurate dynamic right allocation is realized in complex traffic scenarios, and the highway train passing efficiency and safety are improved.

[0033] In order to illustrate the effectiveness of the method proposed in the present application, the above technical solutions of the present application will be described in detail below through a specific embodiment. As shown in Figure 1 The present application discloses a highway train dynamic right allocation method based on reinforcement learning and rule constraints, and the specific implementation steps are as follows: Step S1, obtaining the state information of each vehicle in the road environment where the highway train is located, the state information including the sequence of the position and heading of the vehicle before the current time; establishing the state features of each vehicle according to the state information; In this step, the state information of each vehicle in the road environment where the highway train is located is obtained. The state information of each vehicle can include the position information, speed information, acceleration information, heading angle information and other key features of the vehicle. These environmental data provide basic information support for subsequent right allocation. In actual application, the above road environment data can be obtained through vehicle-mounted sensors, roadside equipment, vehicle networking communication and other ways. The obtained data can fully reflect the traffic environment state of the highway train currently located, and provide accurate data basis for intelligent decision-making.

[0034] After obtaining the road environment data, the state features of the vehicle need to be extracted and processed. This step completes the vehicle coordinate conversion, and extracts the vehicle state features through the multi-head self-attention mechanism. The processed data is used as the input of the subsequent model, which is described in detail as follows.

[0035] The present application constitutes a sequence of multiple headings of the vehicle before the current time; the present application constitutes a sequence of multiple positions of the vehicle before the current time, which is called the historical trajectory sequence of each vehicle.

[0036] The vehicle position and heading of the present application are described in detail as follows.

[0037] The vehicle position and map information in the current state are represented by the current position P and the heading vector V, where the heading vector is represented by the trajectory direction. In order to describe the position relationship between the vehicle and the vehicle , three characteristic quantities of distance , heading difference , and relative azimuth angle are used to describe.

[0038] The specific calculation formula is as follows:

[0039]

[0040]

[0041]

[0042]

[0043] wherein, represents the Euclidean distance between the vehicle and the vehicle , represents the current position vector of the vehicle , represents the current position vector of the vehicle , represents the modulus of the vector; represents the heading difference between the vehicle and the vehicle , and respectively represent the direction vectors of the historical trajectories of the vehicle and the vehicle , represents the vector cross product, represents the azimuth angle of the vehicle relative to the vehicle , represents the position vector from the vehicle to the vehicle .

[0044] By calculating the sine and cosine values ​​of the heading difference and relative azimuth angle, the relative position and direction of motion between vehicles can be fully characterized, avoiding the periodicity problem in angle representation. This feature information provides important state input for subsequent multi-vehicle cooperative decision-making.

[0045] The vehicle's position coordinates in the historical trajectory of the global coordinate system are transformed into coordinates based on the target vehicle's current position coordinate system, such as... Figure 2 The diagram shows a coordinate transformation, and the coordinate transformation formula is as follows:

[0046]

[0047] in, These are the coordinates and heading angles in the global coordinate system of the autonomous vehicle. The coordinates and heading angles of the traffic participants in the global coordinate system. This refers to the coordinates and heading angles of the traffic participants in the local coordinate system.

[0048] This coordinate transformation converts vehicle position information in a global coordinate system into position information in a local coordinate system centered on the target vehicle, facilitating subsequent feature extraction and decision analysis.

[0049] This invention processes the historical trajectory sequences of vehicles to obtain the state characteristics of each vehicle. Specifically, it processes the historical trajectory sequences of traffic participants. The definition is as follows:

[0050]

[0051] Let represent the set of historical trajectory sequences of all m vehicles. Indicates vehicle exist The historical trajectory sequence of moments Indicates vehicle exist The historical trajectory sequence of moments Indicates vehicle exist Historical trajectory points at any given time; each trajectory point can include its location and heading. Indicates the total number of vehicles. Indicates the width of the time window.

[0052] When the trajectory sequence of a vehicle is less than the preset time window width At this time, fill it to the specified length by using the numerical value 0. At the same time, in order to identify whether there is real trajectory data for each step, a binary dimension mask vector is added, the elements in the mask vector take values of 0 or 1, wherein 1 indicates that there is real trajectory data at the time step, and 0 indicates that the time step is padding data. The mask vector is spliced with the trajectory data, and finally an input vector with a size of is obtained, which contains the position information, heading information and data validity information of the traffic participants.

[0053] The input vector is determined as the state feature of each vehicle.

[0054] Step S2, based on the state information, conflict detection is performed to obtain a conflict matrix representing the conflict relationship of the highway train and other vehicles; In order to reduce the redundancy of decision data and improve the decision efficiency, the application introduces a conflict prediction mechanism. The mechanism screens the surrounding vehicles that do not conflict with the host vehicle by constructing a conflict matrix, so that the decision process is more focused on the vehicles that truly have conflict risks, thereby improving the real-time performance and accuracy of the road right allocation.

[0055] Step S2-1, based on the state information, conflict detection is performed to judge and record the conflict relationship of other vehicles and the highway train; In this step, it is judged whether the surrounding vehicles have conflicts with the host vehicle. The specific judgment criteria include: calculating the minimum distance between the surrounding vehicles and the host vehicle within the prediction time range, if the minimum distance is greater than the preset safety distance threshold, it is considered that the surrounding vehicle does not have conflict with the host vehicle; if the minimum distance is less than or equal to the safety distance threshold, it is considered that there is a conflict risk.

[0056] Step S2-2, a conflict matrix is established from the conflict relationship, including: if the other vehicles have conflicts with the highway train, the corresponding element value in the conflict matrix is set to 1, otherwise it is set to 0.

[0057] The conflict matrix is a matrix, wherein is the total number of vehicles in the scene. The elements in the matrix represent whether there is a conflict relationship between vehicle i and vehicle j, when there is a conflict, the value is 1; when there is no conflict, the value is 0. The diagonal elements of the conflict matrix are all 0, because the vehicle cannot conflict with itself. By maintaining and updating the conflict matrix, the conflict relationship between all vehicles in the current scene can be clearly represented, which provides an important basis for subsequent road right allocation and cooperative decision-making.

[0058] Through the above steps, the conflict matrix representing the conflict relationship between the highway train and other vehicles is established. For the surrounding vehicles judged as having no conflict, the influence of these vehicles will not be considered in the subsequent decision intention calculation. In this way, the amount of data to be processed can be significantly reduced, the calculation complexity can be reduced, and the decision speed can be improved.

[0059] Step S3, performing graph attention mechanism processing on the state features of each vehicle, and obtaining global state features based on the cooperative relationship between the highway train and other vehicles; In order to construct the dynamic cooperative relationship between vehicles and optimize the group decision performance, the graph attention mechanism is introduced into the centralized Critic network. The framework can adaptively learn the cooperative relationship strength between vehicles through the graph attention mechanism, so as to better perform global value evaluation.

[0060] Step S3-1, obtaining attention weights between each vehicle based on the state features of each vehicle; The calculation of the attention weights adopts the mechanism of a graph attention network (GAT). Specifically, for vehicle i and vehicle j, the input is the vehicle state features extracted by step S1 for the two vehicles, denoted as , respectively, where F is the feature dimension, represents a real field composed of feature dimensions. First, the state features of them are mapped through a shared linear transformation layer to obtain high-dimensional feature representations and : , , W∈

[0061] wherein, and represent the original state features of vehicle i and vehicle j, respectively; and represent the mapped high-dimensional feature vectors of vehicle i and vehicle j, respectively; represents a shared linear transformation weight matrix, which upgrades the original features to dimensions; is the original feature dimension; is the mapped feature dimension, represents a real field with a dimension of .

[0062] Then, the unnormalized attention score is calculated, which reflects the decision influence degree of vehicle j on vehicle i:

[0063] wherein, denotes the unnormalized attention score of vehicle j to vehicle i; is an activation function to prevent gradient vanishing; || denotes vector concatenation operation; a e denotes the learnable attention vector; Then, softmax normalization is performed on all neighbor nodes (including itself) to obtain the final attention weight :

[0064] where, is the normalized attention weight of the output; denotes the set of all vehicles within the perception range of vehicle i (including itself). denotes the unnormalized attention score of vehicle k to vehicle i. The value range of the attention weight is between 0 and 1, and the greater the weight, the stronger the cooperative relationship between the two vehicles and the greater the mutual influence. Through this mechanism, the model can automatically learn which vehicles need to be focused on and coordinated, and which vehicles have less influence.

[0065] Optionally, the present application adopts a multi-head attention mechanism, that is, M groups of the above attention weights are calculated in parallel, and the results are averaged or spliced to enhance the modeling ability of the model for different interaction modes:

[0066] where, is the attention weight after multi-head averaging; M represents the number of attention heads; denotes the weight calculated by the mth head; Step S3-2, using the attention weight, constructing a dynamic cooperation graph; In this step, the dynamic cooperation graph is constructed using the calculated attention weight, denoted as . The dynamic cooperation graph consists of a node set and an edge set . Node represents the th vehicle in the scene, that is, the th agent, where denotes the index of the node, is the total number of agents. The undirected edge indicates that the agent has a cooperative relationship with the agent .

[0067] In some embodiments, the weight of an edge is determined by the attention weight calculated in step S3-2. For any two agents i and j, if the calculated attention weight is greater than or equal to a preset threshold, it is determined that there is a valid cooperative relationship between the two agents, and then at node... and Establish an undirected edge between them When the attention weight is less than a threshold, no edge is established. The resulting collaboration graph is dynamically changing; as the traffic scenario evolves and vehicle states are updated, the graph's structure adjusts accordingly. This dynamic collaboration graph accurately depicts the relationships between vehicles in complex traffic scenarios, providing crucial structured information for global decision-making.

[0068] Step S3-3: Obtain global state features based on the dynamic collaboration graph.

[0069] In this step, graph convolution or graph attention aggregation operations are used to aggregate the features of each node with the features of its neighboring nodes.

[0070] During the aggregation process, the contributions of neighboring nodes are weighted by attention weights. By stacking multiple graph attention layers, indirect influence relationships between vehicles at greater distances can be captured. Finally, the features of all nodes are aggregated into a global state feature. This vector is then input into subsequent layers of the Critic network.

[0071] Step S4: Establish a multi-vehicle collaborative decision-making model. Input the global state features into the multi-vehicle collaborative decision-making model to obtain the driving decision intentions of each vehicle. This invention constructs a multi-vehicle cooperative decision-making model based on the Multi-Agent Proximal Policy Optimization algorithm (MAPPO algorithm). For example... Figure 3 The diagram illustrates the MAPPO algorithm, which enables collaborative decision-making among multiple agents. In this step, the invention establishes a multi-objective reward function encompassing collision rewards, speed rewards, and right-of-way rewards, thereby building a multi-vehicle collaborative decision-making model. Then, global state features are input into the multi-vehicle collaborative decision-making model to obtain the driving intentions of each vehicle. A detailed description follows.

[0072] Step S4-1: Determine the reward function for the multi-vehicle collaborative decision-making model.

[0073] Step S4-1-1: For each vehicle in the current traffic scenario, initialize its state and action space.

[0074] The state space includes information such as the position, speed, acceleration, heading angle of the vehicle, and the relative position relationship with surrounding vehicles. The action space can be defined as a discrete action space or a continuous action space according to the actual application scenario. In this embodiment, the action space can include operations such as acceleration, deceleration, maintaining the current speed, lane changing, etc. Each vehicle is an agent, which makes decisions in its respective state space and action space.

[0075] Step S4-1-2, according to the real-time position and speed of the vehicle in the state and action space, the collision risk is calculated, and the collision reward value is calculated accordingly.

[0076] In some embodiments, the calculation of the collision risk can be based on a comprehensive evaluation of factors such as the distance between vehicles, relative speed, vehicle size, etc. When the distance between vehicles is less than a safety distance threshold, it is considered that there is a high collision risk. According to the size of the collision risk, the collision reward value is adjusted.

[0077] Step S4-1-3, according to the deviation of the current speed of the vehicle from the ideal speed, the speed reward value is calculated.

[0078] In some embodiments, the ideal speed can be determined according to factors such as road speed limit, traffic flow state, vehicle target, etc. When the current speed of the vehicle is close to the ideal speed, a higher positive reward is given; when the vehicle speed deviates from the ideal speed greatly, a smaller reward or a negative reward is given. The design of the speed reward aims to guide the vehicle to travel at a reasonable speed, both to ensure the traffic efficiency and to avoid excessive speeding or low-speed driving.

[0079] Step S4-1-4, according to the first-in first-out principle, the right-of-way order of the vehicle is obtained, and the right-of-way reward value is determined. The calculation formula of the right-of-way reward is as follows:

[0080] wherein, represents the right-of-way reward value, represents the low-priority right-of-way reward, represents the high-priority right-of-way reward.

[0081] According to the order of the vehicles reaching the conflict area, the first-arriving vehicle has a higher right-of-way priority and should be given a higher right-of-way reward; the later-arriving vehicle has a lower right-of-way priority and its right-of-way reward is relatively low. Through this right-of-way reward mechanism based on the FIFO rule, the right-of-way can be fairly and reasonably allocated, avoiding conflicts between vehicles and improving traffic efficiency.

[0082] Step S4-1-5, the collision reward, the speed reward and the right-of-way reward are weighted and summed to obtain the final reward function value. The expression is:

[0083] wherein, 、 、 respectively represent the weight coefficients of the collision reward, the speed reward and the right reward, respectively represent the collision reward, the speed reward and the right reward, represent the total reward value of the intelligent agent .

[0084] Step S4-2, a multi-vehicle cooperative decision-making model is established based on the reward function, global state features are input into the multi-vehicle cooperative decision-making model, and driving decision intentions of each vehicle are obtained.

[0085] In this step, the multi-vehicle cooperative decision-making model outputs the strategy probability distribution of the optional action of each intelligent agent (vehicle) at the current time through forward transmission, or directly outputs a specific action decision. Then, according to the result output by the strategy network, the optimal action is sampled or selected from the candidate action according to the state of each vehicle, and the driving decision intention of each vehicle is determined, such as acceleration, deceleration, lane keeping, lane changing, etc.

[0086] Step S5, screening the driving decision intention of each vehicle by the conflict matrix, breadth-first search is performed according to the screened driving decision intention and the state information of each vehicle, and a driving path planned for the highway train is obtained.

[0087] In this step, the driving decision intention of each vehicle is screened according to the conflict matrix, including: If there is no conflict between a vehicle and the host vehicle, that is, the value of the corresponding element in the conflict matrix is 0, the driving decision intention of the vehicle is removed, and the vehicle is no longer analyzed.

[0088] Through this conflict prediction and screening mechanism, the number of vehicles that need to be processed by the decision system can be effectively reduced, and the decision efficiency is greatly improved while ensuring the accuracy of the decision.

[0089] Breadth-first search is performed according to the screened driving decision intention, including: (1) generating a candidate driving path based on breadth-first search according to the state information of each vehicle, for a certain key point pair of the current , the key point pair of the is searched.

[0090] Key points refer to important location nodes on the road, such as sampling points on the lane centerline, lane change points, merging points, etc. This invention employs a breadth-first search algorithm starting from the vehicle's current position, using discrete lane centerlines as a basis, and sampling reachable key points layer by layer at equal intervals. The road space is discretized into a multi-row set of key points, each row corresponding to a time step or spatial step. A key point pair consists of two key points from two adjacent rows. The row represents the set of key points at the i-th time step or spatial step. Find the first key point pair in the row. A pair of keypoints can be reached from this keypoint pair in the row. Reachability is determined based on vehicle kinematic constraints, such as maximum steering angle, maximum acceleration, and maximum deceleration. For each... The key points of the row are correct, and the traversal is performed. For all key points of a row, determine whether there is a path connection that meets the kinematic constraints.

[0091] (2) Judgment The key points of the line and The key points are checked against whether they match the vehicle's driving decision intent. If they do not, the process returns to step (1); otherwise, the connection is established. Key points to add The key points are located in the orbital sequence.

[0092] The trajectory sequence represents a continuous pair of keypoints arranged in time-space step order from the vehicle's current position to the target position. Its generation process involves: starting with the vehicle's initial keypoint, expanding line by line to satisfy kinematic constraints and align with the decision-making intent. Figure 1 The key point pairs are matched and qualified key point pairs are added to the sequence in row order to form a spatiotemporal path that can be used for vehicle trajectory tracking.

[0093] If the A key point in the line relative to the first If a key point pair is not connected, meaning it does not conform to the vehicle's driving decision intent, then return to step (1) to continue searching for other possible key point pairs. If it is determined to be connected, meaning it satisfies all constraints, then connect the connected key point pair. Key points to add to the first The key points of each row are located in the track sequence. The track sequence records the complete path information from the starting point to the current row. By expanding and filtering row by row, multiple candidate paths that meet the constraints can be generated.

[0094] (3) When there are no connected key point pairs among all key point pairs, the resulting track sequence is saved as the driving path for the highway train planning.

[0095] The detailed description of the application describes and illustrates with reference to certain specific embodiments. However, the description and illustrations are intended to be merely illustrative and not restrictive of the application. While the application has been described and illustrated with reference to specific embodiments, it will be recognized that variations and modifications can be made by persons skilled in the art depending upon the overall teachings of the present application. In particular, those skilled in the art will recognize that elements of the present application can readily be combined to provide further embodiments of the present application. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is, therefore, to be understood that the application is not to be limited to the specific embodiments disclosed and that modifications and / or substitutions are intended to be included within the scope of the present application. Such equivalents are considered within the scope of the present application.

[0096] The detailed description of the application describes and illustrates with reference to certain specific embodiments. However, the description and illustrations are intended to be merely illustrative and not restrictive of the application. While the application has been described and illustrated with reference to specific embodiments, it will be recognized that variations and modifications can be made by persons skilled in the art depending upon the overall teachings of the present application. In particular, those skilled in the art will recognize that elements of the present application can readily be combined to provide further embodiments of the present application. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. It is, therefore, to be understood that the application is not to be limited to the specific embodiments disclosed and that modifications and / or substitutions are intended to be included within the scope of the present application. Such equivalents are considered within the scope of the present application.

Claims

1. A method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints, characterized in that, Includes the following steps: Step S1: Obtain the status information of each vehicle in the road environment where the highway train is located. The status information includes a sequence of vehicle positions and headings before the current moment. Establish the status characteristics of each vehicle based on the status information. Step S2: Based on the state information, perform conflict detection to obtain a conflict matrix representing the conflict relationship between the highway train and other vehicles; Step S3: Process the state features of each vehicle using a graph attention mechanism to obtain global state features based on the cooperative relationship between the road train and other vehicles. Step S4: Establish a multi-vehicle collaborative decision-making model. Input the global state features into the multi-vehicle collaborative decision-making model to obtain the driving decision intentions of each vehicle. Step S5: Filter the driving decision intentions of each vehicle using the conflict matrix, and perform a breadth-first search based on the filtered driving decision intentions and the status information of each vehicle to obtain the planned travel path for the highway train.

2. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 1, characterized in that, Step S1 specifically includes: Step S1-1: Construct a historical trajectory sequence based on multiple headings and positions of the vehicle before the current moment; Step S1-2: Convert the vehicle's position coordinates in the historical trajectory sequence from the global coordinate system to the vehicle's current position coordinate system; Steps S1-3: Process the converted historical trajectory sequence to obtain the state characteristics of each vehicle.

3. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 2, characterized in that, In step S1-1, the expressions for the heading and position are: in, Indicates vehicle With vehicles Euclidean distance, Indicates vehicle The current position vector, Indicates vehicle The current position vector, Represents the magnitude of a vector; Indicates vehicle With vehicles The difference in heading, and Representing vehicles respectively With vehicles The direction vector of the historical trajectory Represents the cross product of vectors. Indicates vehicle Relative to vehicles The azimuth angle, Indicates from vehicle Pointing to the vehicle The position vector; In step S1-2, the expression for coordinate system transformation is: in, These are the coordinates and heading angles in the global coordinate system of the autonomous vehicle. The coordinates and heading angles of the traffic participants in the global coordinate system. This refers to the coordinates and heading angles of the traffic participants in the local coordinate system.

4. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 3, characterized in that, Step S2 specifically includes: Step S2-1: Based on the status information, perform conflict detection, determine and record the conflict relationship between other vehicles and the road train; Step S2-2: Establish a conflict matrix based on the conflict relationship, including: if other vehicles conflict with the road train, set the corresponding element value in the conflict matrix to 1, otherwise set it to 0.

5. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 4, characterized in that, Step S3 specifically includes: Step S3-1: Obtain the attention weights between each vehicle based on the state features of each vehicle; Step S3-2: Construct a dynamic collaboration graph using attention weights; Step S3-3: Obtain global state features based on the dynamic collaboration graph.

6. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 5, characterized in that, Step S3-2 specifically includes: Construct a dynamic collaboration graph using the calculated attention weights. A dynamic collaboration graph consists of a set of nodes. Sum of edges Composition, Node The first in the representative scene Vehicles, among which Indicates the subscript index of the node. This represents the total number of vehicles. When the attention weight is greater than or equal to a preset threshold, it is considered that there is an effective cooperative relationship between the two agents, and the corresponding edge is established in the graph; when the attention weight is less than the threshold, no edge connection is established.

7. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 6, characterized in that, Step S4 specifically includes: Step S4-1: Determine the reward function for the multi-vehicle cooperative decision-making model, including: Step S4-1-1: For each vehicle in the current traffic scenario, initialize the state and action space; Step S4-1-2: Calculate the collision risk based on the real-time position and speed of the vehicle in the state and action space, and calculate the collision reward value accordingly. Step S4-1-3: Calculate the speed bonus value based on the degree of deviation between the vehicle's current speed and the ideal speed; Step S4-1-4: Obtain the right-of-way order of vehicles according to the first-in-first-out principle, and determine the right-of-way reward value; Step S4-1-5: Sum the collision reward value, speed reward value, and right-of-way reward value in a weighted manner to obtain the final reward function; Step S4-2: Establish a multi-vehicle collaborative decision-making model based on the reward function, input the global state features into the multi-vehicle collaborative decision-making model, and obtain the driving decision intentions of each vehicle.

8. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 7, characterized in that, In step S4-1-4, the expression for the right-of-way reward value is: in, This represents the right-of-way reward value. This indicates a reward for low-priority road rights. This indicates a high-priority right-of-way reward; In step S4-1-5, the expression for the reward function is: in, 、 、 These represent the weighting coefficients for collision reward, speed reward, and right-of-way reward, respectively. These represent collision bonus, speed bonus, and right-of-way bonus, respectively. Represents intelligent agents The total reward value.

9. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 8, characterized in that, Step S5 specifically includes: Step S5-1: Filter the driving decision intentions of each vehicle according to the conflict matrix, including: if a vehicle does not conflict with the main vehicle, that is, the value of the corresponding element in the conflict matrix is ​​0, then remove the driving decision intention of that vehicle. Step S5-2: Perform a breadth-first search based on the filtered driving decision intentions to obtain the planned driving path for the highway train.

10. The method for dynamic right-of-way allocation of highway trains based on reinforcement learning and rule constraints according to claim 9, characterized in that, Step S5-2 specifically includes: Step S5-2-1: Generate candidate driving paths based on breadth-first search according to the status information of each vehicle. For the current... Search for a key point pair in the row. Key point pairs; the key point pairs represent sampling points on the center line of the lane; Step S5-2-2, Judgment The key points of the line and The key points are checked to see if they match the vehicle's driving decision intent. If not, the process returns to step S5-2-1; otherwise, the process continues. Key points of the line for update The orbital sequence in which the keypoint pairs are located; the orbital sequence represents a sequence of consecutive keypoint pairs; Step S5-2-3: When there are no connected key point pairs among all key point pairs, save the obtained track sequence as the travel path for the highway train planning.

Images