Boundary type driving space prediction method for automatic driving BEV

Abstract

The invention relates to a boundary type driving space prediction method for an automatic driving BEV, and the method comprises the steps: S1, predicting a prediction boundary after a set time interval through a pre-constructed boundary type driving space prediction model according to the boundary of a boundary type driving space of an automobile under the view angle of the current automatic driving BEV, and enabling the prediction boundary to serve as a boundary type prediction result; s2, acquiring a current visual traffic scene graph, inputting the large language model, and outputting a large language model result of the current driving intention; and S3, carrying out adaptive fusion on the boundary type prediction result and the large language model result to obtain a prediction result of the boundary type driving space of the automatic driving BEV.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile automatic driving, and in particular to an automatic driving BEV (Bird-Eye View, BEV) boundary-type driving space prediction method. Background Art

[0002] In order to improve the safety of autonomous driving, it is crucial to accurately predict the future trajectories of all traffic participants. However, accurate trajectory prediction is a very complex and difficult task because the interactions between traffic participants in complex traffic conditions are difficult to judge.

[0003] In the traditional prediction model, the environmental information detected by the perception system and the location information of traffic participants are input into the neural network, and key features are extracted through certain methods to finally output the corresponding predicted trajectory. Although the prediction accuracy has been improved to a certain extent with the increase in computing power, the introduction of methods such as graph neural networks, and the development of data sets, it still cannot meet the current high safety requirements for autonomous driving, especially for traffic participants in long-tail scenarios. There are large deviations in the prediction, which seriously affects subsequent decision-making and poses certain safety risks. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to provide a method for predicting the boundary driving space of an autonomous driving BEV, in which a boundary prediction model is used, which realizes the overall expression of different environmental targets based on the boundary of the drivable space from the perspective of the BEV, and then predicts the boundary coordinates of the future drivable space, replacing the traditional target trajectory prediction method, which can achieve an overall prediction of the environmental state and improve accuracy and reliability; on the other hand, an image large language model fine-tuned by the traffic scene is used to predict the driving intention according to the visualized traffic scene of the current frame, and then the output driving intention is integrated with the boundary prediction model to further improve the overall prediction accuracy and improve the driving reliability of the autonomous driving vehicle.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present application provides a method for predicting a boundary driving space of an autonomous driving BEV, the method comprising:

[0007] S1, according to the boundary of the boundary driving space of the vehicle under the perspective of the current autonomous driving BEV, using the pre-built boundary driving space prediction model to predict the prediction boundary after the set time interval as the boundary prediction result;

[0008] S2, obtain the current visualized traffic scene graph, input the big language model, and output the big language model result of the current driving intention;

[0009] S3, adaptively fuses the boundary prediction results and the large language model results to obtain the prediction results of the boundary driving space of the autonomous driving BEV.

[0010] In one implementation, in S1, the environmental target perception information includes the spatial position, semantic category and motion state of the dynamic and static environmental targets detected by the environmental perception sensor of the automobile.

[0011] In one implementation, in S1, the boundary prediction result is related to the time interval, the boundary of the boundary driving space from the perspective of the current autonomous driving BEV, and the vehicle state of the vehicle; the formula is:

[0012] B t+Δt =f(Δt,X ego ,B t )

[0013] Set B t+Δt represents the boundary driving space after a fixed time interval, Δt represents a fixed time interval, and the current boundary is B t , the vehicle state is X ego .

[0014] In one implementation, in the boundary-type driving space, the boundary is represented by a unified combination of multi-dimensional set physical quantities of each point.

[0015] In one implementation, the boundary is recorded as:

[0016] B={(x1,y1,V x1 ,V y1 ,η1),(x2,y2,V x2 ,V y2 ,η2),…,(x n ,y n ,V xn ,V yn ,η n )}n=1-360

[0017] Each point is assigned five labels: {x n ,y n} represents the spatial position, {V xn ,V yn} represents the longitudinal velocity and the lateral velocity, {η n} indicates its semantic attributes.

[0018] In one implementation, in S2, the traffic elements in the visualized traffic scene graph include: position P cur 、Current speed V cur, the current yaw angle θ, the trajectory of the past set time and the road centerline M lane ;

[0019] The driving intention corresponding to each frame of the visualized traffic scene graph includes lateral intention and longitudinal intention. Among them, lateral intention can be divided into left turn, left lane change, right turn, right lane change, straight going, and U-turn according to the amplitude, and longitudinal intention can be divided into acceleration, deceleration, braking, and constant speed.

[0020] In one implementation, S3 includes: comparing the boundary driving space prediction result and the large language model intention prediction accuracy.

[0021] In one implementation, in S3, it also includes: using the lane centerline as a reference for the prediction result.

[0022] In one implementation, the final fused boundary-type driving space and the original boundary-type prediction result ζ predicted , driving intention I, accuracy of large language model τ LLM ,,The accuracy of the boundary prediction results τ NN and the adjacent center line M of the road lane Related, recorded as:

[0023] ζ modified =f(ζ predicted ,I,τ LLM ,τ NN ,M lane ).

[0024] In a second aspect, a computer-readable storage medium is provided, storing a computer program, wherein the computer program is executed by a processor to implement the autonomous driving BEV boundary driving space prediction method described in the first aspect.

[0025] Since the present invention adopts the above technical scheme, it has the following advantages: it provides an autonomous driving BEV boundary driving space prediction model, firstly, the perception results under the current traffic scene are abstracted into a boundary expression of the drivable space from the perspective of BEV, and input into the pre-trained boundary prediction neural network model to predict the future boundaries and generate the boundary driving space at the future moment. At the same time, the visual big language model is quoted, and through certain traffic scene fine-tuning, the big language model can generate future driving intention predictions for the current traffic scene. Finally, the adjustable neural network results and the big language model results are fused, and the results of the two methods are judged and fused, and finally a more accurate boundary driving space prediction result is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a BEV boundary driving space prediction model based on a large language model of the present invention;

[0027] Figure 2 is an example diagram of a boundary-type driving space of the present invention;

[0028] Figure 3 It is a visualization diagram of a traffic scene applicable to a large language model of the present invention;

[0029] Figure 4 It is a schematic diagram of the transformation of the boundary-based prediction to trajectory prediction of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0031] In one embodiment of the present application, Figure 1 , a method for predicting the boundary driving space of an autonomous driving BEV is provided, the method comprising:

[0032] S1, according to the boundary of the boundary driving space of the vehicle under the perspective of the current autonomous driving BEV, using the pre-built boundary driving space prediction model to predict the prediction boundary after the set time interval as the boundary prediction result;

[0033] Specifically, the drivable area for autonomous driving can be defined by a unified boundary that combines information from map boundaries, object detection, and occupancy grids. In addition, each part of the boundary is given semantic and dynamic state-related attributes (such as speed, acceleration, etc.). Figure 2 The specific expression of the above-mentioned boundary-type driving space is demonstrated, which can form a boundary-type expression of the traffic scene around the vehicle from the perspective of BEV.

[0034] For the prediction of boundary-based driving space, it is crucial to consider the composition of the boundary. The boundary is composed of discrete points, each of which has its own attributes, including static attributes such as position and semantic attributes, and dynamic attributes such as speed.

[0035] 1) The prediction of boundaries fundamentally involves the motion changes of each point. Due to its inherent physical properties, the motion of each point is related to its physical properties. In addition, semantic attributes mean the classification of points, and different semantic attribute values ​​represent static elements, dynamic elements, map boundaries, and unidentified objects. Therefore, position, speed, and semantic attributes play an important role in predicting the boundary-like driving space at the next moment. This method extracts the above physical quantities:

[0036] B={(x1,y1,V x1 ,x y1 ,η1),(x2,y2,V x2 ,V y2 ,η2),…,(x n ,y n ,V xn ,V yn ,η n )}η=1-360(1)

[0037] Formula 1 expresses the physical quantities in the boundary driving space. Set B represents the boundary, which consists of 360 points. Each point is assigned five labels: {x n ,y n} represents the spatial position, {V xn ,V yn} represents the longitudinal velocity and the lateral velocity, {η n} indicates its semantic attributes.

[0038] 2) In addition to the current state of the boundary, the state of the ego vehicle also affects the future boundary, such as considering the interaction between the ego vehicle and the dynamic elements. When the ego vehicle is moving too fast, other vehicles at the intersection may slow down as a reaction. Therefore, the state of the ego vehicle is also considered for prediction.

[0039] X egi =(x ego ,y ego ,V xego ,V yego ,ω ego ) (2)

[0040] Formula 2 expresses the state of the vehicle, the set X ego Represents the state of the vehicle: {x ego ,y ego} represents the spatial position of the vehicle, {V xn ,V yn} represents the longitudinal velocity and lateral velocity of the vehicle, {ω ego} represents the angular velocity of the vehicle.

[0041] 3) Select a fixed time interval and build a mapping model from the current boundary to the future boundary:

[0042] B t+Δt =f(Δt,X ego ,B t ) (3)

[0043] Set B t+Δtrepresents the boundary driving space after a fixed time interval, and Δt represents a fixed time interval. As shown in function (3), the future boundary is affected by the time interval Δt and the current boundary B t and the vehicle state X ego impact.

[0044] 4) Collect data and use the corresponding neural network for data training

[0045] Data can be collected in two ways. One is to collect boundary driving space data through continuous simulation using an autonomous driving simulator such as Carla. The other is to process data from public data sets such as nuScenes to convert relevant information such as the current target coordinates into boundary driving space data.

[0046] The processed data is input into a neural network, such as LSTM, for parameter learning and verification, thereby obtaining a special prediction model that is only for boundary driving spaces.

[0047] S2, obtain the current visualized traffic scene graph, input the big language model, and output the big language model result of the current driving intention;

[0048] Specifically, first, the data collected from driving simulation software or public datasets need to be converted into a visual scene top view picture. The picture should contain all relevant traffic elements, as described in Formula 4, including the current position P cur 、Current speed V cur , current yaw angle θ, trajectory ζ over the past 2 seconds 2seconds and the road centerline lane All elements are described by specific geometric shapes. Specific visualization effects, such as Figure 3 shown.

[0049] Θ per frame =(X{P cur ,V cur ,θ,ζ2 seconds},Μ lane ) (4)

[0050] Use appropriate prompt words to explain the above scenarios reasonably, such as what kind of traffic elements the various graphics represent, etc., to explain the task of judging driving intention. Because the above data comes from simulation software data collection or public data sets, there are true values ​​for future trajectories and positions. This method annotates the true values ​​and preset prompt words in the corresponding JSON file to support fine-tuning on the visual language pre-training model.

[0051] The driving intention output is the output result of the large language model: according to the actual driving behavior, it can be divided into lateral intention and longitudinal intention. The lateral intention can be divided into left turn, left lane change, right turn, right lane change, straight driving, and U-turn according to the amplitude, and the longitudinal intention can be divided into acceleration, deceleration, braking, and constant speed.

[0052] After fine-tuning the data on driving intention discrimination based on a certain amount of data, the original large language model can further deepen its understanding of the current traffic scenario based on its own excellent understanding ability, and conduct special training on the judgment of driving intention, thereby also improving the corresponding judgment accuracy.

[0053] S3, adaptively fuses the boundary prediction results and the large language model results to obtain the prediction results of the boundary driving space of the autonomous driving BEV.

[0054] When merging the driving intention results of the large model and the boundary-based driving space prediction results, we must first consider which result we trust more, and then fuse the two pieces of information. According to the principle of Kalman filter fusion, the prediction accuracy of both can be used as prior information for fusion, and corresponding result judgments can be made when the results diverge. In addition, the center line of the lane can be used as an important reference for the results, because the vehicle's driving trajectory is along the center line of the lane in most cases.

[0055] (1) Boundary driving space prediction results and large language model intention prediction accuracy statistics

[0056] The evaluation of the prediction results of the boundary driving space is derived from the mean square error MSE of the statistical data distance deviation, and a certain threshold ζ is set MSE , when the MSE of the predicted boundary <ζ MSE , the prediction is considered correct, otherwise it is considered wrong, and the validation data set is used to calculate the accuracy.

[0057] For the accuracy statistics of intent prediction of large language models, the horizontal and vertical intent true values ​​provided by the simulator or dataset are compared with the current prediction results to perform accuracy statistics.

[0058] (2) Lane centerline reference

[0059] The lane centerline coordinates of the current map are used for relevant fitting calculations to determine whether the current prediction results comply with the traffic rules of the current lane. If the car is in the left turn lane and the car is predicted to go straight, the prediction is considered inconsistent with the lane centerline.

[0060] (3) Adaptive fusion of boundary prediction results and large language model results

[0061] 1) Processing of boundary prediction results

[0062] Because the boundary prediction result is the future boundary of the drivable space, there is a certain difference in form compared with the driving intention and the lane centerline. Therefore, the first step is to convert the boundary prediction result into a trajectory to merge with the driving intention.

[0063] The predicted boundary of the future moment can be compared with the boundary-type driving space of the current moment. For example, if the boundary expands in one direction, it means that the vehicle is predicted to move in this direction. By comparing the boundaries, the predicted trajectory can be generated in reverse, and then compared and fused with the driving intention and lane line accordingly to generate the result as follows: Figure 4 As shown, the black boundary is the future boundary. By comparing it with the boundary at the current moment (blue boundary), the corresponding trajectory can be generated from the difference. Formula 5 represents the calculation process, ζ predicted represents the predicted trajectory, B predicted represents the predicted boundary, B current Represents the current boundary.

[0064] ζ predicted =f(B predicted -B current ) (5)

[0065] 2) Adaptive fusion with driving intention

[0066] The final fused boundary driving space and the original boundary prediction result ζ predicted , driving intention I, accuracy of large language model τ LLM ,,The accuracy of the boundary prediction results τ NN and the adjacent center line M of the road lane As shown in formula (6):

[0067] ζ modified =f(ζ predicted ,I,τ LLM ,τ NN ,M lane ) (6)

[0068] The detailed fusion process is divided into four cases:

[0069] Case 1: When the driving intention is consistent with the adjacent centerline of the road, the modified trajectory should follow the direction of the centerline.

[0070] Case 2: When the predicted trajectory coincides with the adjacent centerline of the road, the modified trajectory should follow the direction between the centerline and the predicted trajectory.

[0071] Case 3: When both the driving intention and the predicted trajectory are inconsistent with the adjacent center line of the road, the fused boundary prediction result should follow the weighted average of the driving intention and the predicted trajectory, and the weight is related to the prior information. Equation (7) shows the modification process of Case 3.

[0072]

[0073] Case 4: When the driving intention is consistent with the predicted trajectory, the boundary prediction result will remain unchanged.

[0074] 3) Convert to boundary prediction results

[0075] Finally, the fused trajectory is converted into the original prediction boundary. On the current driving space boundary, a certain contraction and expansion is performed according to the trajectory to form a fused boundary-type driving space prediction result.

[0076] B modified =B current +f(ζ modified ) (8)

[0077] Formula 8 represents the calculation process, ζ modified represents the trajectory predicted after fusion, B modified represents the fused boundary, B current Represents the current boundary.

[0078] The embodiment of the present application also provides a computer-readable storage medium, the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the above method. The specific implementation process will not be repeated here.

[0079] The present application also provides a computer device. The computer device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned method in the embodiment is implemented. To avoid repetition, it is not described one by one here. Alternatively, when the computer program is executed by the processor, the functions of each model / unit in the device in the embodiment are implemented. To avoid repetition, it is not described one by one here.

[0080] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a server, a cloud server, etc. The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the computer device may include more or fewer components than shown in the figure, or may combine certain components, or may include different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0081] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0082] The memory may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. The memory may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Furthermore, the memory may include both an internal storage unit of the computer device and an external storage device. The memory is used to store computer programs and other programs and data required by the computer device. The memory may also be used to temporarily store data that has been output or is to be output.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0084] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0085] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some steps of the above-mentioned method of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0086] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for predicting the boundary driving space of an autonomous driving BEV, characterized in that: The method comprises: S1, according to the boundary of the boundary driving space of the vehicle under the perspective of the current autonomous driving BEV, using the pre-built boundary driving space prediction model to predict the prediction boundary after the set time interval as the boundary prediction result; S2, obtain the current visualized traffic scene graph, input the big language model, and output the big language model result of the current driving intention; S3, adaptively fuses the boundary prediction results and the large language model results to obtain the prediction results of the boundary driving space of the autonomous driving BEV.

2. The method for predicting the boundary driving space of an autonomous driving BEV according to claim 1, characterized in that: In S1, the environmental target perception information includes the spatial position, semantic category and motion state of the dynamic and static environmental targets detected by the environmental perception sensor of the automobile.

3. The method for predicting the boundary driving space of an autonomous driving BEV according to claim 2, characterized in that: In said S1, the boundary prediction result is related to the time interval, the boundary of the boundary driving space from the perspective of the current autonomous driving BEV, and the ego vehicle state of the vehicle; The formula is: B t+Δt =f(Δt,X ego ,B t ) Set B t+Δt represents the boundary driving space after a fixed time interval, Δt represents a fixed time interval, and the current boundary is B t , the vehicle state is X ego .

4. The method for predicting the boundary driving space of an autonomous driving BEV according to claim 3, characterized in that: In the boundary driving space, the boundary is represented by a unified combination of multi-dimensional set physical quantities of each point.

5. The method for predicting the boundary travel space of an autonomous driving BEV according to claim 4, characterized in that: The boundary is recorded as: B={(x1,y1,V x1 ,V y1 ,η1),(x2,y2,V x2 ,V y2 ,η2),…,(x n ,y n ,V xn ,V yn ,η n )}n=1-360 Each point is assigned five labels: {x n ,y n } represents the spatial position, {V xn ,V yn } represents the longitudinal velocity and the lateral velocity, {η n } indicates its semantic attributes.

6. The method for predicting the boundary travel space of an autonomous driving BEV according to claim 1, characterized in that: In S2, traffic elements in the visualized traffic scene graph are visualized, including: cur 、Current speed V cur , the current yaw angle θ, the trajectory of the past set time and the road centerline M lane ; The driving intention corresponding to each frame of the visualized traffic scene graph includes lateral intention and longitudinal intention. Among them, lateral intention can be divided into left turn, left lane change, right turn, right lane change, straight going, and U-turn according to the amplitude, and longitudinal intention can be divided into acceleration, deceleration, braking, and constant speed.

7. The method for predicting the boundary travel space of an autonomous driving BEV according to claim 6, characterized in that: In S3, it includes: comparing the boundary driving space prediction result and the large language model intention prediction accuracy.

8. The method for predicting the boundary driving space of an autonomous driving BEV according to claim 7, characterized in that: In the S3, it also includes: using the lane centerline as a reference for the prediction result.

9. The method for predicting the boundary driving space of an autonomous driving BEV according to claim 8, characterized in that: The final fused boundary driving space and the original boundary prediction result ζ predicted , driving intention I, accuracy of large language model τ LLM ,,The accuracy of the boundary prediction results τ NN and the adjacent center line M of the road lane Related, recorded as: g modified =f(ζ predicted ,I,t LLM ,t NN ,M lane )。 10. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is executed by a processor to implement the autonomous driving BEV boundary driving space prediction method as described in any one of claims 1 to 9.

Images