Vehicle steering control method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a vehicle steering control method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-modal data of a target vehicle in a target scene, the target scene being a scene for track prediction and steering control of the target vehicle; performing trajectory prediction based on the multi-modal data to obtain target trajectory information; determining predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data; and controlling the target vehicle according to the predicted steering control information. According to the embodiment of the invention, the method achieves the prediction of the vehicle track in combination with the multi-modal data in the target scene needing the steering control, and achieves the reasonable and accurate vehicle steering prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle control, and particularly to a vehicle steering control method and device, an electronic device, and a storage medium. Background Art

[0002] A roundabout is a complex traffic scenario, characterized by intertwined traffic flows, diverse dynamic traffic participants, and complex road geometric structures. Vehicles need to perform complex path planning and execution between specific entrances and exits, while interacting with dynamic traffic participants in real time. There are the following challenges in the current vehicle passing in roundabout scenarios: First, the diverse dynamic traffic environment will affect the decision-making and trajectory planning of autonomous vehicles; second, the complex road geometric features will increase the difficulty of trajectory planning; third, it is necessary to ensure a safe distance from dynamic traffic participants within the roundabout to avoid collisions and sudden braking, and the safety requirements are high.

[0003] Currently, the trajectory prediction methods adopted by autonomous vehicles in roundabout scenarios mainly rely on deep learning methods, but this method has problems of insufficient rationality and accuracy of trajectory prediction, and current trajectory prediction methods usually have difficulty in real-time evaluating dynamic safety in complex roundabout scenarios, such as the safe distance between vehicles and other traffic participants. At the same time, steering control is an important part of roundabout passing. Currently, there is a lack of human-likeness in vehicle roundabout passing. The vehicle steering behavior fails to imitate the natural behavior of human drivers, such as adaptively adjusting the steering angle and acceleration changes, thus affecting the passing efficiency and riding experience of the vehicle. In addition, high-precision maps contain detailed road information (such as lane boundaries, road curvature, slope, etc.), which is particularly important for trajectory planning and steering control in roundabout scenarios. However, the existing technology has insufficient utilization of high-precision map information, and the lack of fusion of map and dynamic information leads to incomplete environmental understanding. Summary of the Invention

[0004] In view of the above problems, a vehicle steering control method and device, an electronic device, and a storage medium are proposed to overcome or at least partially solve the above problems, including: A vehicle steering control method, the method includes: Obtain multi-modal data of a target vehicle in a target scenario, where the target scenario is a scenario for the target vehicle to perform trajectory prediction and steering control; Perform trajectory prediction based on the multi-modal data to obtain target trajectory information; Determine predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data; Control the target vehicle according to the predicted steering control information.

[0005] Optionally, performing trajectory prediction based on the multimodal data to obtain target trajectory information, including: Performing feature extraction on the multimodal data to obtain multimodal features; Using a preset attention mechanism to perform weighted fusion on the multimodal features to obtain fused features; Performing trajectory prediction based on the fused features to obtain multiple pieces of candidate trajectory information; Determining target trajectory information from the multiple pieces of candidate trajectory information.

[0006] Optionally, the determining target trajectory information from the multiple pieces of candidate trajectory information includes: Determining the trajectory score corresponding to each piece of candidate trajectory information according to a preset trajectory evaluation index; Determining target trajectory information from the multiple pieces of candidate trajectory information based on the trajectory scores.

[0007] Optionally, the trajectory evaluation index includes any one or more of the following: Deviation of the trajectory from the center line of the lane, trajectory smoothness score, trajectory prediction probability score, minimum distance between the trajectory and dynamic traffic participants.

[0008] Optionally, the method further includes: Constructing a target model of the target vehicle, where the target model includes a target trajectory prediction sub-model for performing trajectory prediction on the target vehicle and a target steering control sub-model for performing steering control on the target vehicle; The performing trajectory prediction based on the multimodal data to obtain target trajectory information includes Inputting the multimodal data into the target trajectory prediction sub-model and outputting the target trajectory information of the target vehicle; The determining the predicted steering control information of the target vehicle according to the target trajectory information and the multimodal data includes: Inputting the target trajectory information and the multimodal features corresponding to the multimodal data into the target steering control sub-model and outputting the predicted steering control information of the target vehicle.

[0009] Optionally, the training process of the target model is as follows: Constructing an initial trajectory prediction sub-model and an initial steering control sub-model; Training the initial trajectory prediction sub-model based on preset training data to obtain a candidate trajectory prediction sub-model; Training the initial steering control sub-model based on the preset training data and the frozen candidate trajectory prediction sub-model to obtain a candidate steering control sub-model; Adjust the parameters of the trajectory prediction sub-model and the candidate steering control sub-model using the preset training data to obtain the target model.

[0010] Optionally, it further includes: Obtain the real-time steering control information of the target vehicle; Determine the loss function of the target model based on the real-time steering control information and the predicted steering control information; Adjust the parameters of the target trajectory prediction sub-model and the target steering control sub-model according to the loss function.

[0011] A vehicle steering control device, the device includes: A multi-modal data acquisition module, configured to acquire multi-modal data of the target vehicle in a target scenario, where the target scenario is a scenario in which the target vehicle performs trajectory prediction and steering control; A target trajectory information determination module, configured to perform trajectory prediction based on the multi-modal data to obtain target trajectory information; A predicted steering control information determination module, configured to determine the predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data; A vehicle steering control module, configured to control the target vehicle according to the predicted steering control information.

[0012] An electronic device, including a processor, a memory, and a computer program stored on the memory and capable of running on the processor, where when the computer program is executed by the processor, it implements the vehicle steering control method as described above.

[0013] A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the vehicle steering control method as described above.

[0014] The embodiments of the present invention have the following advantages: The embodiments of the present invention can acquire multi-modal data of the target vehicle in the target scenario where steering control is required, and then can perform trajectory prediction based on the multi-modal data to obtain target trajectory information; thus, determine the predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data, and control the target vehicle according to the predicted steering control information. By combining multi-modal data analysis, the vehicle driving environment can be comprehensively understood, and then reasonable and accurate steering control information can be obtained, so as to facilitate vehicle decision-making and trajectory planning in diverse dynamic traffic environments, and trajectory planning can also be achieved for complex road geometric features. At the same time, it can ensure a safe distance from dynamic traffic participants in a loaded road to avoid collisions. Description of the Drawings

[0015] To more clearly illustrate the technical solution of the present invention, the accompanying drawings required for the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0016] Figure 1 is a flowchart of the steps of a vehicle steering control method provided by an embodiment of the present invention; Figure 2a is a flowchart of the steps of another vehicle steering control method provided by an embodiment of the present invention; Figure 2b is a schematic diagram of a trajectory prediction process provided by an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a vehicle steering control device provided by an embodiment of the present invention. Specific Embodiments

[0017] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0018] Referring to Figure 1 , a flowchart of the steps of a vehicle steering control method provided by an embodiment of the present invention is shown, which may specifically include the following steps: Step S101, obtain multi-modal data of the target vehicle in the target scenario, where the target scenario is a scenario for the target vehicle to perform trajectory prediction and steering control; In practical applications, in the steering scenario of the target vehicle, to ensure the safety of the target vehicle, it is necessary to predict the future steering control information of the vehicle based on the current state of the vehicle. Therefore, the target vehicle can collect multi-modal data in this scenario to achieve reasonable and accurate steering control based on the multi-modal data. The multi-modal data is data associated with vehicle steering. Among them, the multi-modal data may include at least two of human driver data, dynamic traffic participant data, and road structure data. Specifically, the human driver data may include data such as speed, acceleration, and steering angle; the dynamic traffic participant data may include data such as other vehicles, pedestrians, and bicycles; the road structure data includes data such as road shape, signs and markings, and obstacles.

[0019] Among them, the target scenario can be a scenario that requires trajectory prediction and steering control, such as a curved road scenario, a roundabout scenario, etc.

[0020] Step S102, perform trajectory prediction based on multi-modal data to obtain target trajectory information; After obtaining the multi-modal data, trajectory prediction can be performed based on the correlation relationship between the multi-modal data to obtain target trajectory information, which is the vehicle trajectory information predicted according to the multi-modal data.

[0021] In an embodiment of the present invention, after collecting the multi-modal data and before performing trajectory prediction, data preprocessing can be performed on the multi-modal data, and the preprocessing can include, but is not limited to, any one or more of data cleaning, data synchronization, and other processing processes.

[0022] Step S103, determine the predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data; After predicting the target trajectory information, the target vehicle can determine the predicted steering control information of the target vehicle by combining the target trajectory information and the multi-modal data. Among them, the predicted steering control information can include any one of the steering angle and the acceleration.

[0023] Step S104, control the target vehicle according to the predicted steering control information.

[0024] After the target vehicle obtains the predicted steering control information, it can control the target vehicle to steer according to the predicted steering control information.

[0025] In an embodiment of the present invention, a target model for the target vehicle can be pre-constructed, and the target model can include a target trajectory prediction sub-model for performing trajectory prediction on the target vehicle and a target steering control sub-model for performing steering control on the target vehicle. Furthermore, the target trajectory prediction sub-model and the target steering control sub-model in the target model can be used to implement the vehicle steering control in the embodiment of the present invention. Specifically, in the vehicle trajectory prediction process, the multi-modal data can be input into the target trajectory prediction sub-model, and the target trajectory information of the target vehicle is output; in the vehicle steering control process, the target trajectory information and the multi-modal features corresponding to the multi-modal data can be input into the target steering control sub-model, and the predicted steering control information of the target vehicle is output.

[0026] In an embodiment of the present invention, training data can be collected to train the target model, and the training data can include at least two of human driver data, dynamic traffic participant data, and road structure data.

[0027] In practical applications, to capture the dynamic characteristics of traffic in the target scenario, it is necessary to collect human driver data (such as speed, acceleration, steering angle, etc.), road structure data (road shape, signs and markings, obstacles), and dynamic traffic participant data (other vehicles, pedestrians, bicycles, etc.). At the same time, to improve the human-like driving performance of autonomous vehicles in the target scenario, it is necessary to collect human driving data for steering optimization and trajectory prediction model training. The purpose is to enable the vehicle to learn the speed control and yielding decisions of human drivers when entering a roundabout, the smooth steering adjustment and speed management within the roundabout, and the acceleration and path selection when exiting the roundabout.

[0028] The data collection sources and collection devices are as follows: 1) Vehicle sensor data: including GPS module (obtaining the real-time position and heading angle of the vehicle, accurate to the driving path within the roundabout), inertial navigation system (providing high-frequency acceleration and angular velocity data to capture the steering dynamics of the vehicle within the roundabout), wheel speed sensor (recording the vehicle speed changes and analyzing the acceleration or deceleration behavior when entering and exiting the roundabout), steering wheel angle sensor (collecting the steering wheel angle and angular velocity to reflect the steering operation of the vehicle), etc.

[0029] 2) Environmental perception data: cameras (collecting roundabout environment images, including visual information such as lane markings, traffic signs, pedestrians, bicycles, obstacles, etc.), lidar (generating high-precision three-dimensional point clouds to obtain the roundabout road geometry, the movement trajectories of other vehicles, and the positions of dynamic obstacles), radar (capturing the speed and distance information of other vehicles in the roundabout for modeling traffic interaction relationships); 3) Map and structure information: high-precision map (used to obtain road curvature, slope, lane width, number of lanes, number of entrances and exits, and relative positions, etc.).

[0030] Taking the roundabout scenario as an example, the data collection process can be refined into the following methods: 1) Collection scenario planning: The collected roundabout scenario data should cover different types of roundabouts (single-lane, small multi-lane, large complex roundabouts), different traffic flows (low, medium, high traffic flows), and different driving behaviors (going straight, yielding, merging, exiting).

[0031] 2) Collect data at different times (daytime, night, dusk) and different weather conditions.

[0032] 3) Collect the dynamic behaviors of the vehicle in the roundabout, such as entering the roundabout (speed control, steering adjustment, interaction with main loop vehicles), passing through the roundabout (path selection, turning control, avoiding pedestrians and vehicles), and exiting the roundabout (route adjustment, acceleration or deceleration, etc.).

[0033] By collecting and utilizing human driving data, a trained model can learn the complex decision-making logic and operating habits of human driving, thereby achieving a more human-like autonomous driving performance, which can not only improve driving comfort but also enhance the safety and efficiency of roundabout passage.

[0034] Based on the above-collected training data, the training of the target model can be realized. The specific training process includes the following steps: Step S11, construct an initial trajectory prediction sub-model and an initial steering control sub-model; Among them, the model types of the initial trajectory prediction sub-model and the initial steering control sub-model can be selected according to actual needs. For example, the model type of the initial trajectory prediction sub-model can be a conditional variational autoencoder (PCVAE) model using particle swarm optimization, and the model type of the initial steering control sub-model is a deep neural network (DNN) model.

[0035] Step S12, train the initial trajectory prediction sub-model based on the preset training data to obtain a candidate trajectory prediction sub-model; After selecting the initial model, the training data can be input into the initial trajectory prediction sub-model for iterative training. When the preset training termination condition is reached, the corresponding candidate trajectory prediction sub-model is output to generate a high-precision future trajectory.

[0036] Step S13, train the initial steering control sub-model based on the preset training data and the frozen candidate trajectory prediction sub-model to obtain a candidate steering control sub-model; During the training of the initial steering control sub-model, the model parameters of the candidate trajectory prediction sub-model need to be frozen. Then, the training data can be input into the candidate trajectory prediction sub-model for trajectory prediction, and the estimated results can be input into the initial steering control sub-model for iterative training. During the iterative training process, the actual steering control in the training data can be compared with the initial steering control sub-model, and then the model parameters in the initial steering control sub-model can be feedback-adjusted to make the steering result output by the steering control sub-model approach the expected steering control in the training data, thereby realizing iterative training. When the preset training termination condition is reached, the candidate steering control sub-model is output.

[0037] Step S14, adjust the parameters of the candidate trajectory prediction sub-model and the candidate steering control sub-model using the preset training data to obtain the target model.

[0038] Furthermore, the candidate trajectory prediction sub-model and the candidate steering control sub-model can be combined for end-to-end fine-tuning to obtain the final target model to further improve system consistency.

[0039] Through the above step-by-step training and joint optimization strategy, the efficient combination of trajectory prediction and steering control is achieved, the collaborative ability of control and trajectory prediction is improved, and the vehicle exhibits natural human-like driving behavior in the target scenario.

[0040] In an embodiment of the present invention, during the vehicle steering training and application process using the target model, a loss function can be introduced to adjust the model in real time, so that the target model is closer to the actual situation.

[0041] The specific process of fine-tuning the loss function is as follows: Obtain the real-time steering control information of the target vehicle; Determine the loss function of the target model based on the real-time steering control information and the predicted steering control information; Adjust the parameters of the target trajectory prediction sub-model and the target steering control sub-model according to the loss function.

[0042] In practical applications, the loss function can be set in advance. Then, by obtaining the real-time steering control information and the predicted steering control information, and calculating the loss function, the model parameters of the target trajectory prediction sub-model and the target steering control sub-model are feedback-adjusted, so that the adjusted model parameters can make the real-time steering control information approach the predicted steering control information. Update the target trajectory prediction sub-model and the target steering control sub-model through the above process, thereby improving the data output accuracy of the target trajectory prediction sub-model and the target steering control sub-model.

[0043] To enhance the human-likeness of vehicle steering control, the loss function can be set as any one or more of the control error loss, trajectory deviation loss, and smoothness loss, and the loss function value is the sum of all set loss functions.

[0044] Among them, 1) Control error loss: The purpose is to make the predicted steering angle and acceleration close to the data of human drivers, and its calculation method is:

[0045] In the formula, is the control error loss, N represents the total number of training samples, represents the steering angle of the i-th predicted sample, is the actual steering angle of the i-th sample of the human driver. is the acceleration or deceleration of the i-th sample predicted by the model, is the actual acceleration of the i-th sample of the human driver. is the weight coefficient of the acceleration loss term.

[0046] 2) Trajectory deviation loss: Ensure that the predicted trajectory can smoothly follow the center line of the lane. Its calculation method is:

[0047] Wherein, is the trajectory deviation loss, (X pred，i , Y pred，i ) are the predicted point coordinates of the predicted trajectory, and (X center，i , Y center，i ) the center point is the point coordinates on the center line of the lane.

[0048] 3) Smoothness loss: Constraining the rate of change of the steering angle to avoid excessive steering fluctuations. Its calculation method is:

[0049] Wherein, is the smoothness loss, N represents the total number of training samples, represents the steering angle of the i-th predicted sample, represents the steering angle of the (i - 1)-th predicted sample.

[0050] Therefore, the total loss function can be expressed as:

[0051] Wherein, represents the total loss function, is the control error loss, is the trajectory deviation loss, is the smoothness loss.

[0052] Meanwhile, a feedback loop is introduced into the above-mentioned steering optimization control module to compare the actually executed steering angle with the predicted steering angle, and the multi-modal feature extraction module, the trajectory prediction module, and the imitation learning module are adjusted and optimized in real time according to the error information.

[0053] In the embodiment of the present invention, multi-modal data of the target vehicle in a target scenario where steering control is required can be obtained, and then trajectory prediction can be performed based on the multi-modal data to obtain target trajectory information; thereby, the predicted steering control information of the target vehicle can be determined according to the target trajectory information and the multi-modal data, and the target vehicle can be controlled in accordance with the predicted steering control information. By combining multi-modal data analysis, reasonable and accurate steering control information can be obtained.

[0054] Referring to Figure 2a , a flowchart of steps of another vehicle steering control method provided by an embodiment of the present invention is shown, which may specifically include the following steps: Step S201, obtaining multi-modal data of the target vehicle in a target scenario, where the target scenario is a scenario for the target vehicle to perform trajectory prediction and steering control; Step S202, extracting features from the multi-modal data to obtain multi-modal features; In practical applications, different feature extraction methods can be adopted according to the types of different modality data. The extracted features can effectively eliminate the interference of impurity data and reflect the characteristics of the data.

[0055] The following describes the feature extraction process for various modality data: (1) Historical trajectory feature extraction based on human driver data: In the embodiments of the present invention, a Bidirectional Gate Recurrent Unit based on Symmetric Residual Bidirectional Cross-Attention (BGRU-SRBCA) can be used for trajectory feature extraction. The residual bidirectional cross-attention module is used to capture the associations between different time steps and features in the trajectory, and the bidirectional gate recurrent network layer further extracts the temporal features of the trajectory. The network input is the trajectory information of the past period T, mainly the two-dimensional coordinates of the vehicle. The historical trajectory sequence can be expressed as , where represents the historical trajectory sequence, and represents the two-dimensional coordinates on the trajectory point of the vehicle.

[0056] The trajectory feature extraction process based on BGRU-SRBCA is as follows: 1) First, normalize the historical trajectory data input to the network and scale the data to a unified interval.

[0057] 2) Secondly, to reduce the computational cost and memory consumption during training, the present solution proposes a new symmetric bidirectional cross-attention embedded in the feature extraction network structure. By constructing a symmetric interaction structure, the natural attention symmetry between the latent vectors and the input tokens is utilized to achieve efficient information exchange. This can not only reduce the computational complexity, but also compared with the traditional bidirectional attention, this method only needs to calculate the attention matrix once, the model is more lightweight, reducing the number of parameters and the computational cost. The calculation process of the bidirectional cross-attention is as follows: Slice the trajectory data to convert it into a token sequence. For example, slice it at a fixed time interval Δt, and each slice contains data of n time steps. Suppose the obtained token sequence is, , L is the number of tokens, and D is the token feature dimension. At the same time, initialize a set of learnable latent vectors , M is the number of latent vectors, and M is much smaller than L. These latent vectors are used to capture high-level features. When calculating the bidirectional cross-attention, reference-value pairs can be created through linear projection. Obtain from the latent vector lat, and project from the token sequence tok to get . The similarity between the latent vector and the token is calculated using scaled dot product, and the calculation formula is:

[0058] where represents the attention weight matrix from the lateral input (such as other positions or features from the same network layer) to the token input (such as words or data points in the sequence). represents the attention weight matrix from the token input to the lateral input. is the transpose of, ensuring symmetry. represents the representation matrix of the lateral input. represents the representation matrix of the token input.

[0059] The above formula shows that whether calculating the attention from the latent vector to the input token or from the input token to the latent vector, the same similarity matrix is used, realizing symmetric calculation of two-way information exchange. This symmetric calculation method can optimize both the latent vector and the input token simultaneously, and only needs to calculate the attention matrix once, reducing the computational amount and the number of parameters.

[0060] The attention updates of the latent vector and the token are calculated respectively, and the calculation formula is:

[0061]

[0062] where represents the information (the result after weighted summation) extracted from the token by the lateral feature through the attention mechanism; represents the information extracted from the lateral by the token feature through the attention mechanism. softmax() defaults to normalizing each row of the matrix (i.e., normalizing the key dimension).

[0063] This two-way attention mechanism can reduce the complexity of the traditional attention mechanism from quadratic to linear without sacrificing performance or adding constraints to the input modality (for example, the traditional attention complexity is O(ML), when using self-attention in Transformer, M = L, and the complexity at this time is O(N 2 ), as the length of the input sequence increases, the computational amount will increase quadratically, which makes the computational cost extremely high when processing long sequences and limits its application. In this solution, M is much smaller than L, and the complexity can be approximated as O(L) at this time, effectively solving the problem of high computational complexity of the traditional attention mechanism when processing long sequences.

[0064] 3) Add the attention output to the original input to form a residual, which serves as the new input for the next layer of the network to alleviate the vanishing gradient problem:

[0065]

[0066] where, is the original lateral feature (such as auxiliary feature, context information or features of other modalities); is the updated lateral feature; is the original token feature (such as words in a sequence, local regions in an image, etc.); is the updated lateral feature.

[0067] 4) Finally, input the updated token sequence into a bidirectional gated recurrent network, which can utilize both forward and backward information simultaneously to generate a feature representation containing richer temporal dependencies. For data with obvious time series characteristics such as historical trajectories, this comprehensive information can more comprehensively characterize the motion patterns of vehicles. For example, whether there are repeated speed or direction adjustments of the vehicle within a roundabout, and how these adjustments are related to the vehicle's positions before and after. Let and be the hidden states of the forward GRU and backward GRU at time step t respectively. After being updated by the network, the last layer of the network will output a feature vector of the historical trajectory, denoted as , which can effectively capture the changing trends and patterns of the vehicle's past motion states.

[0068] (2) Extract dynamic traffic participant features based on dynamic traffic participant data Use a fully connected neural network FCNN to extract dynamic traffic participant information. The input of the network is the factual information of dynamic traffic participants such as other vehicles, pedestrians, non-motor vehicles, etc. within the roundabout, including the relative positions, relative speeds, relative motion directions and types of traffic participants. Integrating these data can be represented as a set of dynamic traffic participant information , where the traffic participant type is represented by one-hot encoding. For example, for the car type, it can be represented as (0, 1, 0, 0,...).

[0069] After performing feature transformation and fusion on all traffic participant information through the fully connected network, the final output of the network is a higher-level feature representation of dynamic traffic participants, which can be denoted as .

[0070] (3)High-precision map feature extraction based on road structure data Use Graph Neural Networks (GNN) to extract the geometric information of roundabout lanes (including lane lines and road curvature), the coordinate information of roundabout entrances / exits, the lane width information, and the connection relationships between lanes. Consider map elements (such as lanes, intersections, traffic signs, etc.) as nodes in the graph, and their relationships (such as connection relationships, adjacent relationships, etc.) as edges in the graph. GNN can effectively process this type of graph-structured data and learn the feature representations of nodes and edges. The high-precision map feature extraction process based on graph neural networks is as follows: 1) Define nodes: For lane segments, the continuous part of each lane can be used as a node, and the node attributes can include several types of information such as the length, width, and curvature of the lane. For the entrances, exits, and internal intersections of roundabouts, they can be used as nodes, and the node attributes include information such as the type and coordinates of the intersection.

[0071] 2) Define edges: For lane connection relationships, if two lane segments are connected, add an edge between their corresponding nodes. The attributes of the edge can include information such as the type of connection (e.g., direct connection, turning connection) and the steering angle. For node adjacent relationships, if two intersections are adjacent or an intersection is adjacent to a lane segment, add an edge between their corresponding nodes. The attributes of the edge can include information such as distance and traffic rules.

[0072] 3) Node feature extraction: For lane segment nodes, extract their geometric features from the preprocessed data, such as the start and end coordinates of the lane, the curvature of the lane line, and the lane width. At the same time, extract the semantic features of the lane, such as solid-line lanes and dashed-line lanes. For intersection nodes, extract features such as their coordinates, the number of connected lanes, and traffic rules. For traffic sign nodes, extract features such as their type, location, and orientation.

[0073] 4) Node feature representation: Normalize the extracted node features to have the same scale range. Then, combine these features into a vector as the feature representation of the node. For example, for a lane segment node, its feature vector can be represented as: . where length is the length of the lane segment, width is the lane width, curvature is the curvature of the lane line, and lane_type is the type of the lane (such as solid-line lane, dashed-line lane).

[0074] 5) In the embodiments of the present invention, the graph attention network (GAT) in the graph convolutional network can be adopted. This network can adaptively assign different weights to the neighbors of nodes, can more effectively capture the important relationships between nodes, and finally obtain high-precision map features after being processed by GAT. .

[0075] Step S203: Weightedly fuse the multi-modal features by using a preset attention mechanism to obtain fused features. Since different modal features have different spatial or temporal scales, it is necessary to unify them into the same representation space through feature alignment manipulation. In the embodiments of the present invention, an attention mechanism can be used to assign weights to different modalities, and the fused features are used as the global feature representation and input to the subsequent decoder module for trajectory prediction.

[0076]

[0077] Step S204: Perform trajectory prediction based on the fused features to obtain multiple pieces of candidate trajectory information. In practical applications, the fused features can be input to a decoder for trajectory prediction. The decoder can adopt a conditional variational autoencoder (CVAE). The CAVE can generate the trajectory coordinates of the vehicle at the next N time steps. At the same time, to improve the performance of the CVAE, such as the reconstruction accuracy and the quality of the generated trajectories, a particle swarm optimization (PSO) algorithm can be used to optimize the optimal hyperparameters and model parameters of the CVAE, such as the learning rate, batch size, regularization coefficient, etc.

[0078] Step S205: Determine the target trajectory information from the multiple pieces of candidate trajectory information.

[0079] After generating multiple pieces of candidate trajectory information, each piece of generated candidate trajectory information can be analyzed to select the most reasonable and accurate candidate trajectory information as the target trajectory information. This target trajectory information can then be used to generate candidate steering control information.

[0080] In an embodiment of the present invention, the predicted generated trajectory should be smooth and safe, capable of adapting to the complex road geometry in the target scenario, while considering the movement behaviors of dynamic obstacles and other traffic participants, and potential collision risks should be avoided. Thus, in the embodiment of the present invention, an optimal trajectory selection strategy can be designed to ensure the vehicle passes through the target scenario safely and efficiently in a complex scenario by using comprehensive indicators for trajectory scoring and selection. Specifically, the process of determining the target trajectory information from multiple candidate trajectory information may include: determining the trajectory score corresponding to each candidate trajectory information according to a preset trajectory evaluation index; and determining the target trajectory information from multiple candidate trajectory information based on the trajectory score.

[0081] Among them, the trajectory evaluation index includes any one or more of the following: The deviation of the trajectory from the center line of the lane, the trajectory smoothness score, the trajectory prediction probability score, and the minimum distance between the trajectory and dynamic traffic participants.

[0082] Specifically, the above various indicators are further explained and analyzed as follows: 1) The deviation of the trajectory from the center line of the lane: It is defined as the average distance from the trajectory point to the corresponding center line of the lane, and the calculation method is as follows:

[0083] In the formula, where is the deviation of the trajectory from the center line of the lane, T represents the number of time steps of the trajectory (total number of trajectory points), represents the coordinates of the t-th trajectory point in the trajectory, Θ is the set of discrete points on the center line of the lane, is the coordinate of a point on the center line of the lane. Min represents finding the point on the center line of the lane closest to . It can be understood as finding the point on the center line of the lane closest to the trajectory point and calculating the Euclidean distance between the two points. The smaller the deviation, the closer the trajectory is to the center line of the lane.

[0084] 2) Trajectory smoothness score: It measures the change rate of the heading angular velocity between adjacent trajectory points. Ideally, the change in the heading angle of a smooth trajectory should be continuous and small, which makes vehicle driving smoother and more comfortable. Its calculation method is as follows:

[0085] In the formula, represents the trajectory smoothness score, T is the total number of trajectory points, and are the heading angles of the t-th and t + 1-th trajectory points respectively, and Δt is the sampling time interval of the trajectory points.

[0086] 3) Trajectory prediction probability score: Its calculation method is as follows:

[0087] In the formula, is the k-th predicted trajectory, is the true trajectory, is the input feature for trajectory prediction, is the probability density of the predicted trajectory. A higher probability score means the predicted trajectory is more reasonable.

[0088] 4) Minimum distance from dynamic traffic participants: A larger safety score indicates a safer trajectory. Within the time range of trajectory t, compare the coordinates of all dynamic traffic participants to find the minimum distance between the vehicle trajectory point and any dynamic traffic participant.

[0089]

[0090] In the formula, represents the minimum distance between the vehicle trajectory point and any dynamic traffic participant; is the k-th predicted trajectory, which consists of a series of trajectory points constituting, represents the coordinates of a certain dynamic traffic participant. , and O is the set of coordinates of all dynamic traffic participants.

[0091] Taking into account the above indicators comprehensively, the trajectory evaluation model can be expressed as:

[0092] In the formula, w is the weight, reflecting the importance of each indicator.

[0093] Therefore, the optimal trajectory is the trajectory with the largest J, that is:

[0094] Finally, the optimal trajectory can be output for the downstream steering control module to use.

[0095] Step S206, determine the predicted steering control information of the target vehicle according to the target trajectory information and multi-modal data; Step S207, control the target vehicle according to the predicted steering control information.

[0096] Referring to Figure 2b , a schematic diagram of a trajectory prediction process is shown. Among them, the trajectory prediction module includes an encoder module and a decoder module, and the specific content is as follows: The encoder can collect historical trajectory data, dynamic traffic participants, and high-precision maps, and then perform historical trajectory feature extraction based on BGRU-SRBCA, dynamic traffic participant feature extraction based on FCNN, and high-precision map feature extraction based on GAT. For the proposed features, multi-modal feature fusion is performed based on the attention mechanism. The fused features are combined with the decoder, and trajectory generation is performed based on PCVAE. Among the generated multiple trajectories, the optimal trajectory is selected according to the optimal trajectory screening function.

[0097] Referring to Figure 2b , a steering control schematic diagram is shown, and the steering control signal can be predicted based on Figure 2b the optimal trajectory, the extracted dynamic traffic participant features, and the high-precision map features.

[0098] Specifically, the steering optimization control module can combine the original multi-modal feature extraction module and the trajectory prediction module, introduce an imitation learning module, and learn the steering angle control behavior of human drivers through behavior cloning to optimize the steering angle control. Behavior cloning is an imitation learning method, which is a supervised learning-based imitation learning method. Its core is to directly map environmental features to control outputs by learning the historical data of human drivers.

[0099] The overall imitation learning adopts a deep neural network framework. The inputs for the steering optimization control based on imitation learning are: Vehicle state (current speed, heading angle, heading angular velocity, acceleration); Optimal predicted trajectory features ; Dynamic traffic participant features ; High-precision map features .

[0100] Similarly, based on attention, the above inputs are fused into a multi-modal feature to obtain a new multi-modal feature vector .

[0101] The output of the steering optimization control based on imitation learning is: Vehicle control signals are generated through a multi-layer fully connected network: steering angle δ and acceleration a. The steering angle δ ensures that the vehicle travels along the planned trajectory, and the acceleration ensures a smooth passage through the roundabout.

[0102] In an embodiment of the present invention, multi-modal data of a target vehicle in a target scenario where steering control is required can be obtained, feature extraction is performed on the multi-modal data to obtain multi-modal features; a preset attention mechanism is used to perform weighted fusion on the multi-modal features to obtain fusion features; in step S204, trajectory prediction is performed based on the fusion features to obtain multiple pieces of candidate trajectory information; the target trajectory information is determined from the multiple pieces of candidate trajectory information, so that the predicted steering control information of the target vehicle is determined according to the target trajectory information and the multi-modal data, and the target vehicle is controlled in accordance with the predicted steering control information. By combining multi-modal data analysis, reasonable and accurate steering control information can be obtained.

[0103] It should be noted that for the method embodiments, for the sake of simple description, they are expressed as a series of action combinations. However, those skilled in the art should know that the embodiments of the present invention are not limited by the described action sequences, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0104] Referring to Figure 3 , a schematic structural diagram of a vehicle steering control device provided by an embodiment of the present invention is shown, which may specifically include the following modules: The multi-modal data acquisition module 301 is used to acquire the multi-modal data of the target vehicle in the target scenario, and the target scenario is the scenario where the target vehicle performs trajectory prediction and steering control; The target trajectory information determination module 302 is used to perform trajectory prediction based on the multi-modal data to obtain the target trajectory information; The predicted steering control information determination module 303 is used to determine the predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data; The vehicle steering control module 304 is used to control the target vehicle in accordance with the predicted steering control information.

[0105] In an embodiment of the present invention, the target trajectory information determination module 302 may include: The multi-modal feature determination sub-module is used to perform feature extraction on the multi-modal data to obtain multi-modal features; The fusion feature determination sub-module is used to perform weighted fusion on the multi-modal features by using a preset attention mechanism to obtain fusion features; The candidate trajectory information determination sub-module is used to perform trajectory prediction based on the fusion features to obtain multiple pieces of candidate trajectory information; The target trajectory information determination sub-module is used to determine the target trajectory information from the multiple pieces of candidate trajectory information.

[0106] In one embodiment of the present invention, the target trajectory information determination sub-module may include: A trajectory score determination unit, configured to determine a trajectory score corresponding to each piece of candidate trajectory information according to a preset trajectory evaluation index; A target trajectory information determination unit, configured to determine target trajectory information from the multiple pieces of candidate trajectory information based on the trajectory score.

[0107] In one embodiment of the present invention, the trajectory evaluation index includes any one or more of the following: The deviation of the trajectory from the center line of the lane, the trajectory smoothness score, the trajectory prediction probability score, the minimum distance between the trajectory and the dynamic traffic participants.

[0108] In one embodiment of the present invention, the device further includes: A target model construction module, configured to construct a target model for a target vehicle, where the target model includes a target trajectory prediction sub-model for predicting the trajectory of the target vehicle and a target steering control sub-model for controlling the steering of the target vehicle; When the target trajectory information determination module 302 is used to perform trajectory prediction based on the multi-modal data to obtain target trajectory information, it is specifically configured to: input the multi-modal data into the target trajectory prediction sub-model and output the target trajectory information of the target vehicle; When the predicted steering control information determination module 303 is used to determine the predicted steering control information of the target vehicle according to the target trajectory information and the multi-modal data, it is specifically configured to: Input the multi-modal features corresponding to the target trajectory information and the multi-modal data into the target steering control sub-model and output the predicted steering control information of the target vehicle.

[0109] In one embodiment of the present invention, the device further includes: An initial model construction module, configured to construct an initial trajectory prediction sub-model and an initial steering control sub-model; A predicted trajectory sub-model training module, configured to train the initial trajectory prediction sub-model based on preset training data to obtain a candidate trajectory prediction sub-model; A steering control sub-model training module, configured to train the initial steering control sub-model based on the preset training data and the frozen candidate trajectory prediction sub-model to obtain a candidate steering control sub-model; A parameter fine-tuning module, configured to adjust the parameters of the trajectory prediction sub-model and the candidate steering control sub-model by using the preset training data to obtain a target model.

[0110] In an embodiment of the present invention, the device may further include: A real-time steering control information acquisition module, configured to acquire the real-time steering control information of the target vehicle; A loss function determination module, configured to determine the loss function of the target model based on the real-time steering control information and the predicted steering control information; A parameter adjustment module, configured to adjust the parameters of the target trajectory prediction sub-model and the target steering control sub-model according to the loss function.

[0111] In an embodiment of the present invention, multi-modal data of the target vehicle in a target scenario where steering control is required may be acquired, and then trajectory prediction may be performed based on the multi-modal data to obtain target trajectory information; thereby, the predicted steering control information of the target vehicle may be determined according to the target trajectory information and the multi-modal data, and the target vehicle may be controlled according to the predicted steering control information. By combining multi-modal data analysis, reasonable and accurate steering control information may be obtained.

[0112] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the above vehicle steering control method is implemented.

[0113] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above vehicle steering control method is implemented.

[0114] For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiment.

[0115] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments may be referred to each other.

[0116] Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0117] Embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing terminal device generate a device for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks

[0118] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or the functions specified in multiple blocks

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one flow Figure 1 one flow or multiple flows and / or blocks Figure 1 or the functions specified in multiple blocks

[0120] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention

[0121] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising said element.

[0122] The above provides a detailed introduction to a vehicle steering control method and device, an electronic device, and a storage medium. In this article, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A vehicle steering control method, characterized in that, The method includes: Obtaining multimodal data of a target vehicle in a target scenario, where the target scenario is a scenario for the target vehicle to perform trajectory prediction and steering control; Performing trajectory prediction based on the multimodal data to obtain target trajectory information; Determining predicted steering control information of the target vehicle according to the target trajectory information and the multimodal data; Controlling the target vehicle according to the predicted steering control information.

2. The method according to claim 1, characterized in that, The performing trajectory prediction based on the multimodal data to obtain target trajectory information includes: Performing feature extraction on the multimodal data to obtain multimodal features; Using a preset attention mechanism to perform weighted fusion on the multimodal features to obtain fused features; Performing trajectory prediction based on the fused features to obtain multiple pieces of candidate trajectory information; Determining target trajectory information from the multiple pieces of candidate trajectory information.

3. The method according to claim 2, wherein The determining target trajectory information from the multiple pieces of candidate trajectory information includes: Determining a trajectory score corresponding to each piece of candidate trajectory information according to a preset trajectory evaluation index; Determining target trajectory information from the multiple pieces of candidate trajectory information based on the trajectory scores.

4. The method according to claim 3, wherein The trajectory evaluation index includes any one or more of the following: The deviation of the trajectory from the center line of the lane, the trajectory smoothness score, the trajectory prediction probability score, the minimum distance between the trajectory and dynamic traffic participants.

5. The method according to any one of claims 1 to 4, characterized in that The method further includes: Constructing a target model of the target vehicle, where the target model includes a target trajectory prediction sub-model for performing trajectory prediction on the target vehicle and a target steering control sub-model for performing steering control on the target vehicle; The performing trajectory prediction based on the multimodal data to obtain target trajectory information includes Inputting the multimodal data into the target trajectory prediction sub-model and outputting the target trajectory information of the target vehicle; The determining the predicted steering control information of the target vehicle according to the target trajectory information and the multimodal data includes: Inputting the target trajectory information and the multimodal features corresponding to the multimodal data into the target steering control sub-model and outputting the predicted steering control information of the target vehicle.

6. The method according to claim 5, characterized in that, The training process of the target model is as follows: Constructing an initial trajectory prediction sub-model and an initial steering control sub-model; Training the initial trajectory prediction sub-model based on preset training data to obtain a candidate trajectory prediction sub-model; Training the initial steering control sub-model based on the preset training data and the frozen candidate trajectory prediction sub-model to obtain a candidate steering control sub-model; Adjusting the parameters of the candidate trajectory prediction sub-model and the candidate steering control sub-model using the preset training data to obtain a target model.

7. The method according to claim 5, wherein The method further includes: Obtaining the real-time steering control information of the target vehicle; Determining a loss function of the target model based on the real-time steering control information and the predicted steering control information; Adjusting the parameters of the target trajectory prediction sub-model and the target steering control sub-model according to the loss function.

8. A vehicle steering control device, characterized in that, The device includes: A multimodal data acquisition module, configured to acquire multimodal data of a target vehicle in a target scenario, where the target scenario is a scenario in which the target vehicle performs trajectory prediction and steering control; A target trajectory information determination module, configured to perform trajectory prediction based on the multimodal data to obtain target trajectory information; A predicted steering control information determination module, configured to determine predicted steering control information of the target vehicle according to the target trajectory information and the multimodal data; A vehicle steering control module, configured to control the target vehicle according to the predicted steering control information.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the vehicle steering control method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements the vehicle steering control method according to any one of claims 1 to 7.

Images