Driving behavior prediction method and device, vehicle, medium and product

Abstract

The embodiment of the invention provides a driving behavior prediction method and device, a vehicle, a medium and a product. The method comprises the steps that in the vehicle driving process, environment time sequence information, vehicle time sequence information of a vehicle and adjacent vehicle time sequence information are collected; and then, inputting the environment time sequence information, the vehicle time sequence information of the vehicle and the adjacent vehicle time sequence information into a driving behavior prediction model, and obtaining the driving behavior of at least one adjacent vehicle output by the driving behavior prediction model. Wherein the driving behavior prediction model is obtained by training sample environment time sequence information, sample vehicle time sequence information, sample adjacent vehicle time sequence information and labeling information in advance, and the labeling information comprises sample driving behaviors of sample adjacent vehicles. According to the technical scheme, the accuracy and timeliness of driving behavior prediction can be improved, and the driving safety of the vehicle is ensured.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and particularly relates to a driving behavior prediction method, device, vehicle, medium and product. Background Art

[0002] In the modern transportation field, autonomous driving systems are gradually becoming an important development direction for future travel. During autonomous driving, a vehicle needs to detect and predict the driving behaviors of adjacent vehicles in real time, so as to make corresponding countermeasures in advance to ensure the safety and efficient driving of the vehicle.

[0003] Currently, to predict the driving behaviors of adjacent vehicles, a Gaussian mixture model can be used to construct a vehicle driving behavior classifier, solve the probabilities of adjacent vehicles belonging to each driving behavior category, and the driving behavior with the maximum probability is the predicted driving behavior of the adjacent vehicle.

[0004] However, the existing technologies have problems of low accuracy and low real-time performance. Summary of the Invention

[0005] Embodiments of the present application provide a driving behavior prediction method, device, vehicle, medium and product, so as to achieve the effect of improving prediction accuracy and real-time performance.

[0006] In a first aspect, an embodiment of the present application provides a driving behavior prediction method, including:

[0007] During the driving of a vehicle, collect environmental time series information, the vehicle's own time series information, and the time series information of adjacent vehicles;

[0008] Input the environmental time series information, the vehicle's own time series information, and the time series information of adjacent vehicles into a driving behavior prediction model, and obtain at least one driving behavior of an adjacent vehicle output by the driving behavior prediction model. The driving behavior prediction model is pre-trained through sample environmental time series information, sample vehicle time series information, sample adjacent vehicle time series information, and annotation information, and the annotation information includes the sample driving behaviors of sample adjacent vehicles.

[0009] In a possible implementation manner, before collecting the environmental time series information, the vehicle's own time series information, and the time series information of adjacent vehicles during the driving of the vehicle, the method further includes:

[0010] During the driving of a sample vehicle, collect a plurality of training samples, and each training sample includes sample environmental time series information, sample vehicle time series information, and sample adjacent vehicle time series information;

[0011] Obtain the annotation information corresponding to each sample adjacent vehicle time series information.

[0012] Train the initial model according to the multiple training samples and the annotation information corresponding to each training sample to generate the driving behavior prediction model.

[0013] In a possible implementation manner, the loss function of the initial model includes a cross-entropy loss sub-function and a false detection suppression sub-function, and the false detection suppression sub-function is the product of the cross-entropy loss sub-function and a first weight;

[0014] Among them, the first weight corresponding to when the sample vehicle changes lanes or drives on a curve is greater than the first weight corresponding to when the sample vehicle is in other driving scenarios.

[0015] In a possible implementation manner, the loss function of the initial model further includes an improvement continuity sub-function, and the improvement continuity sub-function is the product of the cross-entropy loss sub-function and a second weight; the annotation information further includes the start period, the middle period, and the end period of the sample driving behavior;

[0016] If the prediction result of the target period is consistent with the corresponding sample driving behavior in the annotation information, or the prediction result of the target moment is undetected, then the second weight of the target period is the preset weight of the target period;

[0017] If the prediction result of the target period is inconsistent with the corresponding sample driving behavior in the annotation information, then the second weight of the target period is the sum of the preset weight of the target period and the weight sum of the second weight corresponding to the middle period;

[0018] Among them, the target period is the start period or the end period, and the preset weight of the target period is less than the second weight corresponding to the middle period.

[0019] In a possible implementation manner, the preset weight of the start period is positively correlated with time, and the preset weight of the end period is negatively correlated with time.

[0020] In a possible implementation manner, the initial model includes a single-layer encoder and a Top-K structure, and the Top-K structure is used to determine K target feature maps from multiple feature maps in the time dimension according to the attention scores of each feature map determined by the single-layer encoder, where K is a positive integer.

[0021] In a possible implementation manner, the method further includes:

[0022] Train the initial model according to the multiple training samples and the annotation information corresponding to each training sample to generate an initial driving behavior prediction model;

[0023] Verify the initial driving behavior prediction model within a first time period, and obtain the prediction results of the initial driving behavior prediction model for each time frame within a second time period, where the second time period belongs to the first time period;

[0024] According to the prediction results of each time frame within the second time period and the preset prediction results, determine whether the initial driving behavior prediction model meets the preset verification conditions;

[0025] If the initial driving behavior prediction model meets the preset verification conditions, then determine the initial driving behavior prediction model as the driving behavior prediction model;

[0026] Wherein, the preset verification conditions include: the accuracy rate of the prediction results is greater than the preset accuracy, the continuity of the accurate prediction results is greater than the preset continuity, and the recognition delay is less than the third preset duration.

[0027] In a possible implementation manner, the initial model further includes a feature extraction module,

[0028] For each training sample, the feature extraction module is used to extract features from the sample environmental time series information and sample vehicle time series information in the training sample, and obtain the first feature map for each time frame;

[0029] The feature extraction module is further used to extract features from the sample adjacent vehicle time series information in the training sample, and obtain the second feature map for each time frame;

[0030] For each time frame, the feature extraction module is further used to splice the first feature map and the second feature map of the time frame to generate the feature map of the time frame.

[0031] In a second aspect, an embodiment of the present application provides a driving behavior prediction device, including:

[0032] An acquisition module, configured to acquire environmental time series information, own vehicle time series information, and adjacent vehicle time series information during the driving process of the vehicle;

[0033] An input module, configured to input the environmental time series information, the own vehicle time series information, and the adjacent vehicle time series information into a driving behavior prediction model, and obtain the driving behaviors of at least one adjacent vehicle output by the driving behavior prediction model.

[0034] In a possible implementation manner, before collecting the environmental time series information, own vehicle time series information, and adjacent vehicle time series information during the driving process of the vehicle, the driving behavior prediction device further includes a training module, configured to:

[0035] During the driving of the sample vehicle, a plurality of training samples are collected, and each training sample includes sample environment time series information, sample vehicle time series information, and sample adjacent vehicle time series information;

[0036] Obtain the annotation information corresponding to each sample adjacent vehicle time series information.

[0037] Train the initial model according to the plurality of training samples and the annotation information corresponding to each training sample to generate the driving behavior prediction model.

[0038] In a possible implementation manner, the loss function of the initial model includes a cross-entropy loss sub-function and an anti-misdetection sub-function, and the anti-misdetection sub-function is the product of the cross-entropy loss sub-function and a first weight;

[0039] Wherein, the first weight corresponding to the sample vehicle when changing lanes or driving on a curve is greater than the first weight corresponding to the sample vehicle in other driving scenarios.

[0040] In a possible implementation manner,

[0041] The loss function of the initial model further includes an improvement continuity sub-function, and the improvement continuity sub-function is the product of the cross-entropy loss sub-function and a second weight; the annotation information further includes the start time period, the middle time period, and the end time period of the sample driving behavior;

[0042] If the prediction result of the target time period is consistent with the sample driving behavior corresponding in the annotation information, or the prediction result at the target moment is undetected, the second weight of the target time period is the preset weight of the target time period;

[0043] If the prediction result of the target time period is inconsistent with the sample driving behavior corresponding in the annotation information, the second weight of the target time period is the sum of the preset weight of the target time period and the weight of the second weight corresponding to the middle time period;

[0044] Wherein, the target time period is the start time period or the end time period, and the preset weight of the target time period is less than the second weight corresponding to the middle time period.

[0045] In a possible implementation manner, the preset weight of the start time period is positively correlated with time, and the preset weight of the end time period is negatively correlated with time.

[0046] In a possible implementation manner, the initial model includes a single-layer encoder and a Top-K structure, and the Top-K structure is used to determine K target feature maps from a plurality of feature maps in the time dimension according to the attention scores of each feature map determined by the single-layer encoder, and K is a positive integer.

[0047] In a possible implementation, the driving behavior prediction device further includes a verification module, configured to:

[0048] Train an initial model according to the multiple training samples and the annotation information corresponding to each training sample to generate an initial driving behavior prediction model;

[0049] Verify the initial driving behavior prediction model within a first time period, and obtain the prediction results of each time frame of the initial driving behavior prediction model within a second time period, where the second time period belongs to the first time period;

[0050] Judge whether the initial driving behavior prediction model meets a preset verification condition according to the prediction results of each time frame within the second time period and a preset prediction result;

[0051] If the initial driving behavior prediction model meets the preset verification condition, determine the initial driving behavior prediction model as the driving behavior prediction model;

[0052] Wherein, the preset verification condition includes: the accuracy rate of the prediction result is greater than a preset accuracy, the continuity of the accurate prediction results is greater than a preset continuity, and the recognition delay is less than a third preset duration.

[0053] In a possible implementation, the initial model further includes a feature extraction module,

[0054] For each training sample, the feature extraction module is configured to extract features from the sample environment time series information and the sample vehicle time series information in the training sample to obtain a first feature map of each time frame;

[0055] The feature extraction module is further configured to extract features from the sample adjacent vehicle time series information in the training sample to obtain a second feature map of each time frame;

[0056] For each time frame, the feature extraction module is further configured to splice the first feature map and the second feature map of the time frame to generate a feature map of the time frame.

[0057] In a third aspect, an embodiment of the present application provides a vehicle, including: a vehicle body, a memory, and a processor;

[0058] The memory stores computer execution instructions;

[0059] The processor executes the computer execution instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementation manners of the first aspect.

[0060] Fourthly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementation manners of the first aspect.

[0061] Fifthly, an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the above first aspect and / or various possible implementation manners of the first aspect.

[0062] The driving behavior prediction method, device, vehicle, medium and product provided by the embodiments of the present application collect environmental time series information, own vehicle time series information and adjacent vehicle time series information during the driving process of the vehicle. Then, the environmental time series information, the own vehicle time series information and the adjacent vehicle time series information are input into a driving behavior prediction model, and at least one driving behavior of an adjacent vehicle output by the driving behavior prediction model is obtained. In this technical solution, a driving behavior prediction model is used to replace the preset conditions for judging driving behavior, so as to improve the accuracy and timeliness of driving behavior prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with the present application, and are used together with the description to explain the principles of the present application.

[0064] Figure 1 It is a schematic flowchart of the training stage in the driving behavior prediction method provided by the present application;

[0065] Figure 2 It is a schematic diagram of the output result provided by an embodiment of the present application;

[0066] Figure 3 It is a schematic structural diagram of the annotation information provided by an embodiment of the present application;

[0067] Figure 4 It is a schematic structural diagram of the initial model provided by an embodiment of the present application;

[0068] Figure 5 It is a schematic structural diagram of the feature extraction module provided by an embodiment of the present application;

[0069] Figure 6 It is a schematic structural diagram of a single-layer encoder provided by an embodiment of the present application;

[0070] Figure 7 It is a schematic structural diagram of a task head provided by an embodiment of the present application;

[0071] Figure 8 It is a schematic flowchart of the verification stage of the driving behavior prediction method provided by the present application;

[0072] Figure 9 Schematic diagram of verification results provided for this application;

[0073] Figure 10 Schematic flow diagram of the model inference stage of the driving behavior prediction method provided for this application;

[0074] Figure 11 Schematic diagram of the driving behavior prediction method provided for the embodiments of this application in a bus scenario;

[0075] Figure 12 Schematic diagram of the scenario when an adjacent vehicle cuts in provided for the embodiments of this application;

[0076] Figure 13 Schematic diagram of the scenario when an adjacent vehicle continuously changes lanes provided for the embodiments of this application;

[0077] Figure 14 Schematic structural diagram of the driving behavior prediction device provided for this application;

[0078] Figure 15 Schematic structural diagram of the vehicle provided for this application.

[0079] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed Description of the Invention

[0080] Here, exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0081] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0082] First, the background involved in this application will be explained:

[0083] In the field of modern transportation, autonomous driving systems are gradually becoming an important development direction for future travel. During autonomous driving, vehicles need to detect and predict the driving behaviors of adjacent vehicles in real time in order to make corresponding response strategies in advance to ensure the safety and efficient driving of the vehicles.

[0084] For example, assume that an adjacent vehicle suddenly changes lanes, decelerates, or accelerates. If the vehicle can predict these driving behaviors in advance, it can take corresponding braking, steering, or accelerating measures in advance, providing more reliable safety protection for the vehicle's passengers and other traffic participants on the road. It can also better cooperate with other vehicles to optimize traffic flow, reduce traffic congestion, and improve the overall traffic efficiency.

[0085] Currently, to predict the driving behaviors of adjacent vehicles, a Gaussian mixture model can be used to construct a vehicle driving behavior classifier, solve the probabilities of adjacent vehicles belonging to each driving behavior category, and the driving behavior with the highest probability is the predicted driving behavior of the adjacent vehicle.

[0086] However, the existing technologies have the following technical problems:

[0087] 1. Low accuracy: The Gaussian mixture model mainly predicts the driving behaviors of adjacent vehicles through probability density functions based on the speed of the host vehicle, the relative speed and relative position between the host vehicle and its preceding vehicle. However, there are many other factors that can affect the driving behaviors of adjacent vehicles. For example, when the traffic signal is about to turn red, an adjacent vehicle may decelerate and brake, that is, environmental factors can affect the driving behaviors of adjacent vehicles; when the host vehicle turns, the driving behaviors of adjacent vehicles will also change to adapt to the change of the host vehicle. That is to say, the existing technologies only consider factors related to speed and position, with fewer dimensions, and cannot guarantee the accuracy of the prediction results.

[0088] 2. Poor real-time performance: During the calculation process, complex probability solving, construction, and solving processes are required, with a large amount of calculation, and the real-time performance is poor during vehicle driving. When the traffic conditions change rapidly, accurate optimization results may not be output in time, affecting the driver's decision-making.

[0089] In summary, the existing technologies have technical problems of low accuracy and poor real-time performance when predicting the driving behaviors of adjacent vehicles.

[0090] Based on the above technical problems, the technical concept of this application is as follows: The driving behavior prediction model can be pre-trained by using sample environmental time series information, sample vehicle time series information, sample adjacent vehicle time series information, and annotation information. In actual applications, the environmental time series information, the vehicle's own time series information, and the adjacent vehicle time series information can be input into the driving behavior prediction model to predict the driving behavior of adjacent vehicles. The entire process combines data from multiple aspects including the environment, the vehicle itself, and adjacent vehicles. Moreover, the vehicle's own time series information and the adjacent vehicle time series information also include multi-dimensional data other than distance and speed, enabling the driving behavior prediction model to more accurately predict the driving behavior of adjacent vehicles by fusing the above data during the processing. Additionally, the driving behavior prediction model is trained with a large amount of data, and the online inference process is relatively efficient, capable of quickly outputting the prediction results of the driving behavior of adjacent vehicles, thereby ensuring the driving safety of the vehicle itself.

[0091] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0092] The driving behavior prediction method provided by this application mainly includes two parts. One part is the model training process, that is, model training is performed according to sample environmental time series information, sample vehicle time series information, sample adjacent vehicle time series information, and annotation information to obtain a driving behavior prediction model; the other part is the model application process, that is, the process of using the driving behavior prediction model to predict the driving behavior of adjacent vehicles.

[0093] It should be understood that the execution entity of the model training process and the execution entity of the model usage process can be the same or different. The embodiments of this application do not limit this and can be determined according to actual situations.

[0094] First, an explanation of the model training process will be given.

[0095] Figure 1 is a schematic flowchart of the training stage in the driving behavior prediction method provided by this application. As Figure 1 shown, this method can be implemented through the following steps:

[0096] S11. During the driving process of the sample vehicle, collect multiple training samples.

[0097] In practical applications, driving data of a sample vehicle within a certain period of time or a certain driving distance can be obtained. For example, the driving data of the sample vehicle within 14,000 kilometers can be obtained. The driving data includes the sequential information of the sample vehicle obtained through the sensors of the sample vehicle and the sample images collected by the image acquisition device deployed in the sample vehicle. Among them, the sample images include the driving environment of the sample vehicle and the sample adjacent vehicles adjacent to the sample vehicle. The driving data can be sliced into multiple sub-driving data according to the input format of the initial model. For each sub-driving data, according to the sample images included in the sub-driving data, the sample environment sequential information and the sample adjacent vehicle sequential information are determined, and combined with the sample vehicle sequential information included in the sub-driving data, a training sample corresponding to the sub-driving data is constructed.

[0098] That is to say, each training sample includes sample environment sequential information, sample vehicle sequential information, and sample adjacent vehicle sequential information.

[0099] It should be understood that the image acquisition device can be deployed inside the sample vehicle or outside the sample vehicle. The embodiments of the present application do not specifically limit the installation position of the image acquisition device.

[0100] It should be understood that the sample adjacent vehicle can be a vehicle in another lane that is relatively close to the sample vehicle, or a vehicle in front of the sample vehicle in the lane where the sample vehicle is located.

[0101] Among them, the sample environment sequential information in the training sample may include the following elements: lane lines, road boundaries, traffic lights. The sample adjacent vehicle sequential information in the training sample includes the following elements: the unique identifier (Identification, ID) of the sample adjacent vehicle, vehicle type, heading angle, speed, acceleration, steering indication status, length, width, the lateral distance and longitudinal distance between the sample adjacent vehicle and the sample vehicle. The sample vehicle sequential information includes the following elements: timestamp, speed, yaw rate, steering indication status, and trajectory.

[0102] In a possible implementation, before training the initial model based on the training samples, the training samples can also be preprocessed to obtain the processed training samples to increase the element dimension of the processed training samples.

[0103] Exemplarily, the training samples can be represented by the following formula:

[0104]

[0105] Among them, X t is used to represent the training sample, t is used to represent the time dimension, N is used to represent the data corresponding to the time frame N in the training samples, F is used to represent the element dimension of the training samples, and R is used to represent the set of natural numbers.

[0106] In the actual implementation solution, N can be 16, indicating that the training samples include data of 16 consecutive time frames. It should be understood that in actual applications, N can also be other values, and the embodiments of the present application do not specifically limit this.

[0107] Furthermore, the training samples can be preprocessed through the following formula:

[0108] p 0x = latDst - cos(psi) * width * 0.5

[0109] p 0y = lgtDst + sin(psi) * width * 0.5

[0110] p 1x = latDst + cos(psi) * width * 0.5

[0111] p 1y = lgtDst + sin(psi) * width * 0.5

[0112] p 2x = latDst - cos(psi) * width * 0.5 + sin(psi) * length

[0113] p 2y = lgtDst + sin(psi) * width * 0.5 + cos(psi) * length

[0114] p 3x = latDst + cos(psi) * width * 0.5 + sin(psi) * length

[0115] p 3y = lgtDst - sin(psi) * width * 0.5 + cos(psi) * length

[0116] Among them, p 0x is the lateral distance between the lower left corner of the adjacent vehicle of the sample and the sample vehicle, p 0y is the longitudinal distance between the lower left corner of the adjacent vehicle of the sample and the sample vehicle, p 1x is the lateral distance between the lower right corner of the adjacent vehicle of the sample and the sample vehicle, p 1y is the longitudinal distance between the lower right corner of the adjacent vehicle of the sample and the sample vehicle, p 2x is the lateral distance between the upper right corner of the adjacent vehicle of the sample and the sample vehicle, p2y is the longitudinal distance between the upper right corner of the sample adjacent vehicle and the sample vehicle, p 3x is the lateral distance between the upper left corner of the sample adjacent vehicle and the sample vehicle, p 3y is the longitudinal distance between the upper left corner of the sample adjacent vehicle and the sample vehicle, latDst is the lateral distance between the sample adjacent vehicle and the sample vehicle, psi is the heading angle of the adjacent sample vehicle, width is the width of the adjacent sample vehicle, lgtDst is the longitudinal distance between the sample adjacent vehicle and the sample vehicle, and length is the length of the adjacent sample vehicle.

[0117] Furthermore, the training samples can also be preprocessed through the following formula:

[0118] v x = vel * cos(psi)

[0119] v y = vel * sin(psi)

[0120] a x = acc * cos(psi)

[0121] a y = acc * sin(psi)

[0122] where v x is the lateral velocity of the sample adjacent vehicle, v y is the longitudinal velocity of the sample adjacent vehicle, a x is the lateral acceleration of the sample adjacent vehicle, a y is the longitudinal acceleration of the sample adjacent vehicle, vel is the velocity of the sample adjacent vehicle, and acc is the acceleration of the sample adjacent vehicle.

[0123] During the preprocessing process, using mathematical functions (such as trigonometric functions), the velocity of the sample adjacent vehicle is split into two components, the lateral v x and the longitudinal v y The acceleration of the sample adjacent vehicle is also split into two components, the lateral a x and the longitudinal a y and the distance between the sample adjacent vehicle and the sample vehicle (lateral distance and longitudinal distance) is split into the distances between the four corners of the sample adjacent vehicle and the sample vehicle, effectively increasing the element dimension of the processed training samples and enabling the initial model to better capture the complex relationships among the sample vehicle, the sample adjacent vehicle, and the lane lines.

[0124] Furthermore, the training samples can also be preprocessed through the following formula:

[0125]

[0126] Among them, z is the normalized element value, X is the element value, X min is the minimum value of the element, X max is the maximum value of the element.

[0127] By normalizing the elements in the training samples through the above formula, different elements are unified to the same scale, so as to adjust all data dimensions to the same order of magnitude, enabling the initial model to converge better during training and improving the efficiency and accuracy of subsequent model training.

[0128] In the actual implementation plan, the sequential information of adjacent sample vehicles can include the information of 6 adjacent sample vehicles. The sequential information of adjacent sample vehicles is then data with a dimension of 102 (6×17), the sequential information of the sample environment is data with a dimension of 30, and the sequential information of the sample vehicle is data with a dimension of 10. After preprocessing the sequential information of adjacent sample vehicles, more comprehensive sequential information of adjacent sample vehicles with a dimension of 142 can be obtained.

[0129] It should be understood that the number of adjacent sample vehicles included in the sequential information of adjacent sample vehicles can be determined according to the actual situation, and can be other numbers other than 6. The embodiments of the present application do not specifically limit this.

[0130] S12. Obtain the annotation information corresponding to each sequential information of adjacent sample vehicles.

[0131] Among them, the annotation information includes the sample driving behaviors of adjacent sample vehicles.

[0132] Exemplarily, the sample driving behaviors can include cutting in, cutting out, turning left, turning right, etc. It should be understood that cutting in means that the adjacent sample vehicle changes lanes from the current lane to the lane where the sample vehicle is located, and cutting out means that the adjacent sample vehicle changes lanes from the lane where the sample vehicle is located to other lanes. The sample driving behaviors also include other driving behaviors, such as accelerating, etc., which can be determined according to the actual situation. The embodiments of the present application do not specifically limit this.

[0133] In practical applications, the annotation information can be obtained by relevant staff manually marking the sequential information of adjacent sample vehicles. Exemplarily, the annotation information can include the following attributes:

[0134] 1. The ID of the adjacent sample vehicle with sample driving behaviors;

[0135] 2. The sample driving behaviors.

[0136] 3. Start time: The moment when the adjacent sample vehicle starts the sample driving behavior;

[0137] 4. Intermediate time: The time when the rear axle center of the adjacent sample vehicle overlaps with the lane boundary;

[0138] 5. End time: The moment when the adjacent vehicle of the sample completes the sample driving behavior.

[0139] S13. Train the initial model according to multiple training samples and the annotation information corresponding to each training sample to generate a driving behavior prediction model.

[0140] In a possible implementation, the initial model can be trained according to multiple processed training samples and the annotation information corresponding to each processed training sample to generate a driving behavior prediction model.

[0141] The loss function of the initial model includes a cross-entropy loss sub-function and an anti-misdetection sub-function.

[0142] Exemplarily, the cross-entropy loss sub-function can be represented by the following formula:

[0143]

[0144] Where, is the value of the cross-entropy loss sub-function, y is the label value, is the predicted value, and k is used to represent the sample vehicle.

[0145] Where, the anti-misdetection sub-function is the product of the cross-entropy loss sub-function and the first weight. The first weight corresponding to the sample vehicle when changing lanes or driving on a curve is greater than the first weight corresponding to the sample vehicle in other driving scenarios.

[0146] Exemplarily, the anti-misdetection sub-function can be represented by the following formula:

[0147]

[0148] Where, is the value of the anti-misdetection sub-function, and weights is the first weight.

[0149] It can be understood that in the above example, the first weight corresponding to other driving scenarios is 1, and the first weight corresponding to the sample vehicle when changing lanes or driving on a curve is a positive number greater than 1. In practical applications, the first weight corresponding to other driving scenarios can also be other values, and only 1 is taken as an example here for illustration.

[0150] Since it is found that during a large number of validations of the trained driving behavior prediction model, the false alarms of the driving behavior prediction model mainly occur in the scenarios where the sample vehicle changes lanes or drives on a curve. Therefore, in order to make the initial model pay more attention to these scenarios, during the training process, a higher first weight is given to the scenarios where the sample vehicle changes lanes or drives on a curve, so as to improve the accuracy of the driving behavior prediction model and reduce the false alarm rate of the driving behavior prediction model.

[0151] It should be understood that the false alarms of the driving behavior prediction model mainly refer to the prediction of incorrect driving behaviors by the driving behavior prediction model or the failure to predict driving behaviors.

[0152] Furthermore, in practical applications, the driving behavior prediction model determines the driving behaviors of adjacent vehicles at the current moment based on data at multiple moments (environmental time series information, own vehicle time series information, and adjacent vehicle time series information). Exemplarily, the driving behavior prediction model determines the driving behaviors of adjacent vehicles at time frame 16 based on data from time frames 1 - 16, and then determines the driving behaviors of adjacent vehicles at time frame 17 based on data from time frames 2 - 17. That is to say, when in use, the driving behavior prediction model outputs the predicted driving behaviors of adjacent vehicles at each time frame.

[0153] Figure 2 This is a schematic diagram of the output result provided by the embodiment of the present application. As Figure 2 shown, each square containing "1" is the prediction result corresponding to each time frame. Among them, the green squares are used to indicate that the prediction results are consistent with the sample driving behaviors in the label information, that is, detected, which is a True Positive (TP); the gray squares are used to indicate that the prediction results are not detected, which is a False Negative (FN). It can Figure 2 be seen that the recognition continuity of the driving behavior prediction model 1 is the best, that of the driving behavior prediction model 2 is the second best, and that of the driving behavior prediction model 3 is the worst.

[0154] In practical applications, the autonomous driving system controls the vehicle's driving according to the prediction results of the driving behavior prediction model. Because the higher the continuity of accurate prediction results, the higher the stability of vehicle control, and the smoother the vehicle will drive. Therefore, during the model training process, it is necessary to improve the continuity of accurate prediction results output by the model as much as possible.

[0155] However, since in the initial stage of the occurrence of driving behaviors, the changes in adjacent vehicles are relatively small and it is relatively easy to be confused with other driving behaviors. For example, a slight adjustment of the direction of an adjacent vehicle may be due to the unconscious small actions of the driver or may be a precursor to a large - scale turn, which is difficult to determine in the initial stage. Therefore, if the driving behavior prediction model overly focuses on the initial stage of the occurrence of driving behaviors, it is very likely to output incorrect results, thereby resulting in the discontinuity of the accurate prediction results output.

[0156] At the same time, it is also considered that there is no unified judgment standard for manual labeling, which is subjective. This subjective bias will be transmitted to the initial model, resulting in the problem that the trained driving behavior prediction model is prone to incorrect judgments in the initial and final stages of driving behaviors when the changes in adjacent vehicles are not obvious.

[0157] Based on the above two considerations, when calibrating the sequential information of adjacent vehicles in the sample, the sample driving behavior can be divided into a start period, an intermediate period, and an end period, that is, the annotation information also includes the start period, the intermediate period, and the end period of the sample driving behavior.

[0158] Referring to the example in S12, it can be known that the annotation information may include a start time and an end time. The start period can be determined as the first preset duration starting from the start time, the end period can be determined as the second preset duration counting backward from the end time, and the period between the start period and the end period is determined as the intermediate period.

[0159] Moreover, the loss function of the initial model further includes an improvement continuous sub-function, and the improvement continuous sub-function is the product of the cross-entropy loss sub-function and the second weight.

[0160] In a possible implementation manner, if the prediction result of the target period is consistent with the corresponding sample driving behavior in the annotation information (i.e., positive detection), or the prediction result at the target moment is undetected (i.e., missed detection), then the second weight of the target period is the preset weight of the target period;

[0161] If the prediction result of the target period is inconsistent with the corresponding sample driving behavior in the annotation information (i.e., false detection), then the second weight of the target period is the sum of the preset weight of the target period and the weight sum of the second weight corresponding to the intermediate period;

[0162] Wherein, the target period is the start period or the end period, and the preset weight of the target period is less than the second weight corresponding to the intermediate period.

[0163] Exemplarily, the improvement continuous sub-function can be represented by the following formula:

[0164]

[0165] Wherein, ignoreweights is the initial weight, 1 is the second weight of the intermediate period, is the improvement continuous sub-function. It should be understood that the second weight corresponding to the intermediate period can also be other values, and only 1 is taken as an example for illustration here.

[0166] In the above implementation, when positive detection or missed detection occurs during the target time period, a second weight lower than that of the intermediate time period is assigned to the target time period, so that during the training process of the initial model, the start time period and the end time period are not overly concerned, thereby ensuring the accuracy and continuity of the prediction results output by the driving behavior prediction model. At the same time, when false detection occurs during the target time period, it indicates that the initial model does not yet have the ability to accurately predict driving behavior during the target time period. Therefore, the second weight of the target time period is increased to be greater than the second weight of the intermediate time period, ensuring that the initial model can pay more attention to the target time period during the training process, thereby improving the accuracy of model inference.

[0167] Further, in order to ensure that the driving behavior prediction model can predict the driving behavior of adjacent vehicles as early as possible, the preset weight of the start time period is positively correlated with time, and the preset weight of the end time period is negatively correlated with time. That is to say, in the start time period, the preset weight increases with the increase of time; in the end time period, the preset weight decreases with the increase of time. That is, the smaller the preset weight, the closer it is to the start or end of the driving behavior.

[0168] Figure 3 It is a schematic structural diagram of the annotation information provided by the embodiment of the present application. As Figure 3 shown, the annotation information includes a start time period, an intermediate time period, and an end time period, and the second weight of the intermediate time period is 1. The prediction results of the start time period and the end time period are both one of positive detection or missed detection. The second weight of the start time period and the second weight corresponding to the end time period are both the preset weights corresponding to their respective time periods. The second weight shows an upward trend in the start time period, a stable state in the intermediate time period, and a downward trend in the end time period.

[0169] By judging whether driving is detected in the target time period and whether the prediction result of the target time period is consistent with the sample driving behavior corresponding to the annotation information, the second weight corresponding to the target time period is determined. In the case of false detection in the target time period, the second weight of the target time period is set to be greater than the second weight of the intermediate time period, so that the model pays more attention to the target time period during the training process; in the case of missed detection or positive detection in the target time period, the second weight of the target time period is set to be less than the second weight of the intermediate time period. On the premise of ensuring the accuracy of the model training process, the initial model can not overly focus on the start time period and the end time period, thereby improving the continuity of the output results.

[0170] Further, the total loss function of the initial model can be expressed by the following formula:

[0171]

[0172] where L is the total loss function of the initial model, and J is the total number of sample adjacent vehicles.

[0173] The driving behavior prediction method provided by the embodiments of this application, in the model training stage, collects multiple training samples during the driving process of the sample vehicle. Each training sample includes sample environment time series information, sample vehicle time series information, and sample adjacent vehicle time series information. Then, the annotation information corresponding to each sample adjacent vehicle time series information is obtained. Finally, the initial model is trained according to the multiple training samples and the annotation information corresponding to each training sample to generate the driving behavior prediction model. In this technical solution, since the annotation information includes the sample driving behaviors of the sample adjacent vehicles, training the initial model with the training samples and the annotation information corresponding to the training samples can enable the driving behavior prediction model to have the ability to identify the driving behaviors of adjacent vehicles. In practical applications, the driving behavior prediction model can be used to replace the preset conditions for judging driving behaviors, improving the accuracy and timeliness of driving behavior prediction.

[0174] Next, the structure of the initial model will be explained:

[0175] Figure 4 It is a schematic structural diagram of the initial model provided by the embodiments of this application. As Figure 4 shown, the initial model includes a feature extraction module, a single-layer encoder, a Top-K structure, and a task head.

[0176] For each training sample, the feature extraction module is used to extract features from the sample environment time series information and the sample vehicle time series information in the training sample to obtain the first feature map for each time frame.

[0177] The feature extraction module is also used to extract features from the sample adjacent vehicle time series information in the training sample to obtain the second feature map for each time frame.

[0178] For each time frame, the feature extraction module is also used to splice the first feature map and the second feature map of the time frame to generate the feature map of the time frame.

[0179] Figure 5 It is a schematic structural diagram of the feature extraction module provided by the embodiments of this application. As Figure 5 shown, the feature extraction module includes two separate linear layers, namely linear layer 1 and linear layer 2. Each linear layer includes two linear rectification units. Linear layer 1 is used to extract features from the sample environment time series information and the sample vehicle time series information in the training sample to obtain the first feature map for each time frame. Linear layer 2 is used to extract features from the sample adjacent vehicle time series information in the training sample to obtain the second feature map for each time frame. Then, the first feature map and the second feature map of each time frame are spliced to generate the feature map of this time frame.

[0180] By setting up two independent linear layers in the feature extraction module, while maintaining the magnitude network architecture, it promotes efficient feature extraction and improves the model processing efficiency.

[0181] In practical applications, a single-layer encoder can contain only 15,000 weights, effectively reducing the model structure and facilitating deployment on the vehicle side. The single-layer encoder includes an attention module, which mainly captures the dependencies of time-series data and the non-linear features of spatial data to meet the requirements of considering the overall trend and parameter correlations.

[0182] Among them, the attention module consists of a self-attention mechanism and a multi-head attention mechanism. The self-attention mechanism uses a query matrix key matrix value matrix (where B is the batch size, T d is the downsampling of the time series, and F d is the downsampling of the feature map). The self-attention mechanism calculates the dot product of Q and K and uses as the scaling factor. Subsequently, the normalized exponential function (SoftMax function) is applied to obtain the weights of the values, that is, the attention scores of each feature map.

[0183] Figure 6 This is the structural schematic diagram of the single-layer encoder provided by the embodiment of the present application. As Figure 6 shown, the single-layer encoder includes: a multi-head attention mechanism, addition and normalization, and a feed-forward network.

[0184] Furthermore, the Top-K structure is used to determine K target feature maps from multiple feature maps in the time dimension according to the attention scores of each feature map determined by the single-layer encoder, where K is a positive integer. Utilizing the ability of the attention scores to abstract relevant time-series information, K target feature maps with the top K attention scores (in descending order) are determined from the feature maps corresponding to all time frames in the time dimension and input into the subsequent task head.

[0185] Among them, K is less than the total number of time frames. Exemplarily, K can be 3, 4, or 5, which can be set according to the actual situation, and the embodiments of the present application do not specifically limit this.

[0186] Exemplarily, the single-layer encoder and the Top-K structure can be represented by the following formula:

[0187]

[0188]

[0189] Top_K_indices=sort(SUM(i),K)

[0190] FM=(Attention(Q,K)*V)[Top_K_indices]

[0191] Among them, i is used to represent the time dimension, j is used to represent the feature dimension, and FM is used to represent the feature map.

[0192] Reference Figure 4 The gray blocks are the target feature maps with the top K attention scores, and the white blocks are other feature maps. Through the Top-K structure, the target feature map can be determined from all feature maps and input into the subsequent task head.

[0193] Figure 7 This is a schematic diagram of the structure of the task head provided in the embodiment of the present application. Figure 7 As shown in the figure, the task head is a stacked hourglass structure based on the DeLighT model, which includes 4 linear transformation layers. From bottom to top, the parameters are: [B, 4, 10], [B, 4, 20], [B, 4, 12], [B, 1, 48], [B, 1, 32], [B, 1, 12].

[0194] The difference from the existing technology is that the task head uses a shallower and narrower linear transformation layer near the input to expand the features and perform the first dimension reduction on the initial input [B, 4, 10]. Through the first feature expansion, the feature information of the input data is preserved to a greater extent, avoiding the problem of direct one-dimensionalization causing feature information loss or unclearness. After that, a deeper and wider linear transformation layer is used to expand the features and then reduce the dimension to [B, 4, 12] again, avoiding the problem of excessive parameters and consumption of computing resources during subsequent one-dimensionalization.

[0195] In practical applications, when the initial model meets the training cutoff conditions, the current initial driving behavior prediction model can be obtained, and the initial driving behavior prediction model can be verified to help timely discover potential problems that may not be exposed during the training process of the model, and take corresponding measures to optimize and improve it, so as to ensure the prediction accuracy of the model application process and thus ensure the safe driving of the vehicle.

[0196] Next, the model validation process is explained.

[0197] Figure 8 A flow chart of the verification phase of the driving behavior prediction method provided in this application, such as Figure 8 As shown, the method can be implemented by the following steps:

[0198] S81. Train an initial model based on multiple training samples and the annotation information corresponding to each training sample to generate an initial driving behavior prediction model.

[0199] It should be understood that the implementation process and principle of this step can refer to the explanation of the above embodiments and will not be elaborated here.

[0200] S82. Validate the initial driving behavior prediction model within a first time period to obtain the prediction results of the initial driving behavior prediction model for each time frame within a second time period.

[0201] Wherein, the second time period belongs to the first time period.

[0202] In practical applications, the initial driving behavior prediction model can be deployed in a validation vehicle. Relevant staff drive the validation vehicle within the first time period, and continuously predict the driving behavior of adjacent vehicles through the initial driving behavior prediction model.

[0203] Exemplarily, the first time period includes 1 - 300 time frames. The initial driving behavior prediction model determines the driving behavior of adjacent vehicles at time frame 16 based on the data of time frames 1 - 16, and then determines the driving behavior of adjacent vehicles at time frame 17 based on the data of time frames 2 - 17... Finally, it determines the driving behavior of adjacent vehicles at time frame 300 based on the data of time frames 285 - 300. At this time, the second time period includes time frames 16 - 300.

[0204] S83. Determine whether the initial driving behavior prediction model meets the preset validation conditions according to the prediction results of each time frame within the second time period and the preset prediction results.

[0205] Wherein, the preset prediction result is the correct driving behavior.

[0206] Wherein, the preset validation conditions include: the accuracy rate of the prediction results is greater than the preset accuracy, the continuity of accurate prediction results is greater than the preset continuity, and the recognition delay is less than the third preset duration.

[0207] Specifically, the accuracy rate of the prediction results refers to the proportion of the time frame data where the prediction results are consistent with the preset prediction results in the total number of time frames. The preset accuracy can be 85%, 90%, 95%, etc., and it can be specifically set according to the actual situation. The embodiments of this application do not specifically limit this.

[0208] The continuity of accurate prediction results can refer to the continuous degree of accurate prediction results corresponding to the same driving behavior.

[0209] The recognition delay refers to the difference between the start time of the recognized driving behavior and the preset start time. For example, the driver actually starts to turn at time frame 16, but the initial driving behavior prediction model recognizes that the vehicle starts to turn at time frame 20, then the recognition delay is 4 time frames.

[0210] Figure 9 This is the schematic diagram of the verification result provided by this application. As Figure 9 shown, it includes the prediction results of 3 initial driving behavior prediction models, namely the initial driving behavior prediction model A, the initial driving behavior prediction model B, and the initial driving behavior prediction model C. Each square containing "1" and each square containing "2" represents the prediction result corresponding to each time frame. Among them, each dark green square containing "1" is used to indicate that the prediction result is consistent with the sample driving behavior (driving behavior 1) in the label information, that is, detected, and it is a TP; each dark green square containing "2" is used to indicate that the prediction result is consistent with the sample driving behavior (driving behavior 2) in the label information, that is, detected, and it is a TP; each red square containing "1" is used to indicate that the prediction result (driving behavior 1) is inconsistent with the sample driving behavior in the label information, that is, misdetected, and it is a false positive (FP); each red square containing "2" is used to indicate that the prediction result (driving behavior 2) is inconsistent with the sample driving behavior in the label information, that is, misdetected, and it is an FP; each gray square containing "1" is used to indicate that the sample driving behavior in the label information is driving behavior 1, but the prediction result is not detected, and it is an FN; each gray square containing "2" is used to indicate that the sample driving behavior in the label information is driving behavior 2, but the prediction result is not detected, and it is an FN.

[0211] Each light green square containing "1" is a driving behavior 1 event composed of at least two dark green squares containing "1", and is used to indicate detection, which is a TP; each light green square containing "2" is a driving behavior 2 event composed of at least two dark green squares containing "2", and is used to indicate detection, which is a TP. Each orange square containing "1" is a driving behavior 1 event composed of at least one red square containing "1", and is used to indicate misdetection, which is an FP, and each orange square containing "2" is a driving behavior 2 event composed of at least one red square containing "2", and is used to indicate misdetection, which is an FP.

[0212] It should be understood that label represents the annotation information.

[0213] Exemplarily, the verification results shown in Figure 9 are statistically described through Table 1 below.

[0214] Table 1

[0215]

[0216] S84. If the initial driving behavior prediction model meets the preset verification conditions, the initial driving behavior prediction model is determined as the driving behavior prediction model.

[0217] When the initial driving behavior prediction model meets the preset verification conditions, it indicates that the initial driving behavior prediction model has passed the verification. The initial driving behavior prediction model can be determined as the driving behavior prediction model and deployed in the vehicle so as to cooperate with the automatic driving system in actual application to accurately control the vehicle driving. When the initial driving behavior prediction model does not meet the preset verification conditions, it indicates that the initial driving behavior prediction model has not passed the verification and needs to be retrained or continue to be trained.

[0218] In this implementation manner, after the model training is completed, by verifying the initial driving behavior prediction model, potential problems that may not have been exposed during the training process of the model can be timely discovered, and corresponding measures can be taken for optimization and improvement to ensure the prediction accuracy during the model application process, thereby ensuring the safe driving of the vehicle.

[0219] Finally, an explanation is given for the model application process.

[0220] Figure 10 As shown in the flowchart of the model inference stage of the driving behavior prediction method provided by this application, Figure 10 as shown, this method can be implemented through the following steps:

[0221] S101. During the vehicle driving process, collect environmental time-series information, the vehicle's own time-series information, and the adjacent vehicle's time-series information.

[0222] In actual application, an image acquisition device is deployed in the vehicle to collect images. Among them, the image includes the driving environment of the vehicle itself and the adjacent vehicles adjacent to the vehicle itself. The image can be analyzed to determine the environmental time-series information and the adjacent vehicle's time-series information. At the same time, the vehicle's own time-series information can also be obtained according to the sensors of the vehicle itself.

[0223] It should be understood that the image acquisition device can be deployed inside the vehicle itself or outside the vehicle itself. The embodiments of this application do not specifically limit the installation position of the image acquisition device.

[0224] Among them, the environmental time-series information may include the following elements: lane lines, road boundaries, and traffic lights; the adjacent vehicle's time-series information includes the following elements: the ID of the adjacent vehicle, vehicle type, heading angle, speed, acceleration, steering indication state, length, width, the lateral and longitudinal distances between the adjacent vehicle and the vehicle itself, and the vehicle's own time-series information includes the following elements: timestamp, speed, yaw rate, steering indication state, and trajectory.

[0225] Optionally, the environmental time series information and the adjacent vehicle time series information can also be preprocessed respectively to increase the element dimensions of the processed environmental time series information and the processed adjacent vehicle time series information.

[0226] During the preprocessing process, using mathematical functions (such as trigonometric functions), the speed of the adjacent vehicle is split into two components, namely the lateral and longitudinal components, the acceleration of the adjacent vehicle is also split into two components, namely the lateral and longitudinal components, and the distance between the adjacent vehicle and the vehicle (lateral distance and longitudinal distance) is split into the distances between the four corners of the adjacent vehicle and the vehicle, effectively increasing the element dimensions of the processed environmental time series information and the processed adjacent vehicle time series information, and being able to better help the driving behavior prediction model capture the complex relationship among the host vehicle, adjacent vehicles, and lane lines.

[0227] It should be understood that the specific implementation manner and principle of this step can refer to the relevant content in S12, which will not be elaborated here.

[0228] S102: Input the environmental time series information, the host vehicle time series information, and the adjacent vehicle time series information into the driving behavior prediction model to obtain the driving behaviors of at least one adjacent vehicle output by the driving behavior prediction model.

[0229] In a possible implementation manner, the processed environmental time series information, the host vehicle time series information, and the processed adjacent vehicle time series information can also be input into the driving behavior prediction model to obtain the driving behaviors of at least one adjacent vehicle output by the driving behavior prediction model.

[0230] Among them, the driving behavior prediction model is a model trained and verified through the above embodiments. The relevant results and explanations can refer to the above embodiments, which will not be elaborated here.

[0231] In practical applications, the driving behavior prediction model can determine the probability of each adjacent vehicle being in each driving behavior according to the environmental time series information, the host vehicle time series information, and the adjacent vehicle time series information. For each adjacent vehicle, the driving behavior with the highest probability and greater than the preset probability is determined as the driving behavior of the adjacent vehicle for output.

[0232] Taking the driving behavior prediction model outputting the driving behaviors of 6 adjacent vehicles as an example, the output of the driving behavior prediction model can be Y = [Y0, Y1, Y2, Y3, Y4, Y5], where Y i is the driving behavior of the i-th adjacent vehicle. Among them, Y i ∈R u , and u is the total number of driving behaviors.

[0233] The driving behavior prediction method provided by the embodiment of the present application collects environmental time-series information, the vehicle's own time-series information, and the time-series information of adjacent vehicles during the driving process of the vehicle. Then, the environmental time-series information, the vehicle's own time-series information, and the time-series information of adjacent vehicles are input into a driving behavior prediction model to obtain the driving behaviors of at least one adjacent vehicle output by the driving behavior prediction model. In this technical solution, a driving behavior prediction model is used to replace the preset conditions for judging driving behaviors, improving the accuracy and timeliness of driving behavior prediction.

[0234] It has been verified that by predicting the driving behaviors of adjacent vehicles through this technical solution, the accuracy rate can be as high as 99.32%, and compared with the existing algorithms, the performance improvement rate is 67.43%

[0235] For the cut-in scenario, next, the effects of this technical solution and the existing algorithms will be compared and described.

[0236] Figure 11 It is a schematic diagram of the driving behavior prediction method provided by the embodiment of the present application in a bus scenario. As Figure 11 shown, a is the image corresponding to the driving behavior of the adjacent vehicle detected for the first time through this technical solution, and b is the image corresponding to the driving behavior of the adjacent vehicle detected for the first time through the existing algorithm.

[0237] Figure 12 It is a schematic diagram of the scenario of an adjacent vehicle during cut-in provided by the embodiment of the present application. As Figure 12 shown, a is the image corresponding to the driving behavior of the adjacent vehicle detected for the first time through this technical solution, and b is the image corresponding to the driving behavior of the adjacent vehicle detected for the first time through the existing algorithm.

[0238] Figure 13 It is a schematic diagram of the scenario of an adjacent vehicle's continuous lane change provided by the embodiment of the present application. As Figure 13 shown, a is the image corresponding to the driving behavior of the adjacent vehicle detected for the first time through this technical solution, and b is the image corresponding to the driving behavior of the adjacent vehicle detected for the first time through the existing algorithm.

[0239] Through Figure 11 、 Figure 12 and Figure 13 it can be clearly seen that through this technical solution, the driving behaviors of adjacent vehicles can be detected earlier.

[0240] The following is an embodiment of the device of the present application, which can be used to execute the method embodiment of the present application. For the details not disclosed in the embodiment of the device of the present application, please refer to the method embodiment of the present application.

[0241] Figure 14 It is a schematic diagram of the structure of the driving behavior prediction device provided by the present application. AsFigure 14 As shown in Figure 14 , the driving behavior prediction device 140 provided in this embodiment includes:

[0242] An acquisition module 141, configured to acquire environmental time series information, own vehicle time series information, and adjacent vehicle time series information during vehicle driving;

[0243] An input module 142, configured to input the environmental time series information, own vehicle time series information, and adjacent vehicle time series information into a driving behavior prediction model, and obtain at least one driving behavior of an adjacent vehicle output by the driving behavior prediction model.

[0244] In a possible implementation manner, before acquiring the environmental time series information, own vehicle time series information, and adjacent vehicle time series information during vehicle driving, the driving behavior prediction device 140 further includes a training module, configured to:

[0245] During the driving of a sample vehicle, acquire a plurality of training samples, each training sample including sample environmental time series information, sample vehicle time series information, and sample adjacent vehicle time series information;

[0246] Obtain annotation information corresponding to each sample adjacent vehicle time series information.

[0247] Train an initial model according to the plurality of training samples and the annotation information corresponding to each training sample to generate a driving behavior prediction model.

[0248] In a possible implementation manner, the loss function of the initial model includes a cross-entropy loss sub-function and a suppression of false detection sub-function, and the suppression of false detection sub-function is the product of the cross-entropy loss sub-function and a first weight;

[0249] Wherein, the first weight corresponding to when the sample vehicle changes lanes or drives on a curve is greater than the first weight corresponding to when the sample vehicle is in other driving scenarios.

[0250] In a possible implementation manner, the loss function of the initial model further includes an improvement continuity sub-function, and the improvement continuity sub-function is the product of the cross-entropy loss sub-function and a second weight; the annotation information further includes the start period, middle period, and end period of the sample driving behavior;

[0251] If the prediction result of the target period is consistent with the corresponding sample driving behavior in the annotation information, or the prediction result at the target moment is undetected, the second weight of the target period is the preset weight of the target period;

[0252] If the prediction result of the target period is inconsistent with the corresponding sample driving behavior in the annotation information, the second weight of the target period is the sum of the preset weight of the target period and the weight sum of the second weight corresponding to the middle period;

[0253] Among them, the target time period is the start time period or the end time period, and the preset weight of the target time period is less than the second weight corresponding to the intermediate time period.

[0254] In a possible implementation manner, the preset weight of the start time period is positively correlated with time, and the preset weight of the end time period is negatively correlated with time.

[0255] In a possible implementation manner, the initial model includes a single-layer encoder and a Top-K structure. The Top-K structure is used to determine K target feature maps from multiple feature maps in the time dimension according to the attention scores of each feature map determined by the single-layer encoder, where K is a positive integer.

[0256] In a possible implementation manner, the driving behavior prediction device 140 further includes a verification module, which is used for:

[0257] Training the initial model according to multiple training samples and the annotation information corresponding to each training sample to generate an initial driving behavior prediction model;

[0258] Verifying the initial driving behavior prediction model within a first time period, obtaining the prediction results of each time frame of the initial driving behavior prediction model within a second time period, where the second time period belongs to the first time period;

[0259] Judging whether the initial driving behavior prediction model meets a preset verification condition according to the prediction results of each time frame within the second time period and the preset prediction results;

[0260] If the initial driving behavior prediction model meets the preset verification condition, then determine the initial driving behavior prediction model as the driving behavior prediction model;

[0261] Among them, the preset verification conditions include: the accuracy rate of the prediction results is greater than the preset accuracy, the continuity of the accurate prediction results is greater than the preset continuity, and the recognition delay is less than a third preset duration.

[0262] In a possible implementation manner, the initial model further includes a feature extraction module,

[0263] For each training sample, the feature extraction module is used to extract features from the sample environment time series information and the sample vehicle time series information in the training sample to obtain the first feature map of each time frame;

[0264] The feature extraction module is further used to extract features from the sample adjacent vehicle time series information in the training sample to obtain the second feature map of each time frame;

[0265] For each time frame, the feature extraction module is further used to splice the first feature map and the second feature map of the time frame to generate the feature map of the time frame.

[0266] The driving behavior prediction module provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effects are similar. Details are not described herein in this embodiment.

[0267] Figure 15 It is a schematic structural diagram of a vehicle provided in this application. As Figure 15 shown, the vehicle 150 provided in this embodiment includes: a vehicle body 151, at least one processor 152, and a memory 153. Optionally, the vehicle 150 further includes a communication component 154. Among them, the processor 152, the memory 153, and the communication component 154 are connected through a bus 155.

[0268] In a specific implementation process, at least one processor 152 executes computer-executable instructions stored in the memory 153, so that at least one processor 152 executes the above method.

[0269] For the specific implementation process of the processor 152, reference can be made to the above method embodiment, and its implementation principle and technical effects are similar. Details are not described herein in this embodiment.

[0270] In the above embodiment, it should be understood that the processor may be a central processing unit (English: Central Processing Unit, abbreviated: CPU), or other general-purpose processors, digital signal processors (English: Digital Signal Processor, abbreviated: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, abbreviated: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.

[0271] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0272] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0273] This application also provides a computer program product, including a computer program which, when executed by a processor, implements the above method.

[0274] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above method.

[0275] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0276] An exemplary readable storage medium is coupled to the processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0277] The division of units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of devices or units can be in electrical, mechanical or other forms.

[0278] The unit described as a separate component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0279] In addition, in each embodiment of the present invention, each functional unit may be integrated in a processing unit, may exist physically separately for each unit, or two or more units may be integrated in one unit.

[0280] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0281] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above method embodiments. The aforementioned storage medium includes: various media that can store program codes such as ROM, RAM, magnetic disks, or optical discs.

[0282] Finally, it should be noted that: After considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other implementation schemes of the present invention. The present invention aims to cover any variations, uses, or adaptive changes of the present invention. These variations, uses, or adaptive changes follow the general principles of the present invention and include common general knowledge or conventional technical means in the technical field not disclosed in the present invention. It is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. A driving behavior prediction method, characterized in that: include: During the driving process of the vehicle, the environment time sequence information, the vehicle time sequence information and the adjacent vehicle time sequence information are collected; The environment timing information, the vehicle timing information of the vehicle itself and the adjacent vehicle timing information are input into a driving behavior prediction model to obtain the driving behavior of at least one adjacent vehicle output by the driving behavior prediction model, wherein the driving behavior prediction model is pre-trained through sample environment timing information, sample vehicle timing information, sample adjacent vehicle timing information and labeled information, and the labeled information includes sample driving behaviors of sample adjacent vehicles.

2. The method according to claim 1, characterized in that Before collecting the environment time sequence information, the vehicle time sequence information and the adjacent vehicle time sequence information during the vehicle driving process, the method further includes: During the driving process of the sample vehicle, a plurality of training samples are collected, each training sample including sample environment time series information, sample vehicle time series information and sample adjacent vehicle time series information; Obtain the labeling information corresponding to the time series information of adjacent vehicles of each sample; The initial model is trained according to the multiple training samples and the annotation information corresponding to each training sample to generate the driving behavior prediction model.

3. The method according to claim 2, characterized in that The loss function of the initial model includes a cross entropy loss sub-function and a false-pick suppression sub-function, and the false-pick suppression sub-function is the product of the cross entropy loss sub-function and the first weight; Among them, the first weight corresponding to the sample vehicle when changing lanes or driving on a curve is greater than the first weight corresponding to the sample vehicle when it is in other driving scenarios.

4. The method according to claim 3, characterized in that The loss function of the initial model also includes an improved continuous sub-function, which is the product of the cross entropy loss sub-function and a second weight; the annotation information also includes the start period, middle period and end period of the sample driving behavior; If the prediction result of the target time period is consistent with the sample driving behavior corresponding to the annotation information, or the prediction result of the target time is not detected, the second weight of the target time period is the preset weight of the target time period; If the prediction result of the target period is inconsistent with the sample driving behavior corresponding to the annotation information, the second weight of the target period is the sum of the preset weight of the target period plus the second weight corresponding to the intermediate period; The target time period is the start time period or the end time period, and the preset weight of the target time period is less than the second weight corresponding to the middle time period.

5. The method according to claim 4, characterized in that The preset weight of the start time period is positively correlated with time, and the preset weight of the end time period is negatively correlated with time.

6. The method according to any one of claims 2 to 5, characterized in that: The initial model includes a single-layer encoder and a Top-K structure, wherein the Top-K structure is used to determine K target feature maps from multiple feature maps in the time dimension according to the attention score of each feature map determined by the single-layer encoder, where K is a positive integer.

7. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: Training the initial model according to the multiple training samples and the annotation information corresponding to each training sample to generate an initial driving behavior prediction model; Verifying the initial driving behavior prediction model in a first time period, and obtaining prediction results of the initial driving behavior prediction model in each time frame in a second time period, where the second time period belongs to the first time period; Determining whether the initial driving behavior prediction model meets a preset verification condition according to the prediction result of each time frame in the second time period and the preset prediction result; If the initial driving behavior prediction model satisfies the preset verification condition, determining the initial driving behavior prediction model as the driving behavior prediction model; The preset verification conditions include: the accuracy of the prediction result is greater than the preset accuracy, the continuity of the accurate prediction result is greater than the preset continuity, and the recognition delay is less than the third preset time length.

8. The method according to claim 6, characterized in that The initial model also includes a feature extraction module, For each training sample, the feature extraction module is used to extract features from the sample environment time series information and the sample vehicle time series information in the training sample to obtain a first feature map of each time frame; The feature extraction module is also used to extract features from the time sequence information of adjacent vehicles in the training samples to obtain a second feature map for each time frame; For each time frame, the feature extraction module is further used to splice the first feature map and the second feature map of the time frame to generate a feature map of the time frame.

9. A driving behavior prediction device, characterized in that: include: The acquisition module is used to collect the environment timing information, the vehicle timing information and the adjacent vehicle timing information during the vehicle driving process; The input module is used to input the environmental time series information, the vehicle time series information of the vehicle and the adjacent vehicle time series information into the driving behavior prediction model to obtain the driving behavior of at least one adjacent vehicle output by the driving behavior prediction model.

10. A vehicle, characterized in that: include: Vehicle body, memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it is used to implement the method according to any one of claims 1 to 8.

Images