A method for assisting driving on a highway

Abstract

This application provides a highway assisted driving method that uses video frame images captured by a camera as the images to be analyzed. Scene analysis is performed on the images to obtain vehicle detection results, lane segmentation results, and drivable area segmentation results. Depth estimation is then performed on the images to obtain depth estimation results. Based on the depth estimation results, the depth information of all vehicles detected in the images to be analyzed is calculated, thus obtaining the precise depth of each vehicle. A homography matrix is ​​calculated based on the depth estimation results. The images to be analyzed are then converted into a bird's-eye view based on the homography matrix. The distance information between vehicles is calculated using the depth information in the BEV coordinate system. The tracking trajectory of each vehicle is calculated in conjunction with the bird's-eye view, and then the vehicle tracking trajectory is used for subsequent warning calculations.

Description

Technical Field

[0001] This invention relates to the field of intelligent assisted driving technology, specifically a highway assisted driving method. Background Technology

[0002] Current advanced driver assistance systems (ADAS) typically integrate multiple sensors working in tandem, with cameras as the primary sensor and radar and GPS units as secondary sensors. For example, in highway driving scenarios, radar is used to detect the distance between vehicles. With the increasing prevalence of intelligent vehicles, some low-cost vehicles need to eliminate the radar module to reduce overall costs. Therefore, how to achieve driver assistance without using a radar module while ensuring vehicle safety has become a problem that needs to be solved. Summary of the Invention

[0003] To address the issue that most existing driver assistance methods rely on radar data, leading to excessively high vehicle costs, this invention provides a highway driver assistance method that can achieve driver assistance without using radar data and can provide accurate driver assistance data.

[0004] The technical solution of the present invention is as follows: a highway assisted driving method, characterized in that it includes the following steps: S1: Obtain the data to be analyzed based on the onboard cameras and sensors installed on the vehicle; The vehicle-mounted camera involved in the calculation is denoted as: the camera to be analyzed; the image obtained after preprocessing the video frames collected in real time by the camera to be analyzed is denoted as: the image to be analyzed; the acquisition time of the image to be analyzed is denoted as: time frame t; The data to be analyzed includes: the image to be analyzed, the vehicle speed acquired via the CAN bus, the camera intrinsic parameter matrix K of the camera to be analyzed, and the installation height H of the camera to be analyzed. cam ; S2: Obtain any image to be analyzed, perform multi-task reasoning based on the image to be analyzed, and output the scene analysis results of the highway environment included in the image to be analyzed in real time; the scene analysis results include: vehicle detection results, lane line segmentation results, and drivable area segmentation results; Each vehicle detection result corresponds to a vehicle to be analyzed; S3: Perform depth estimation on the image to be analyzed and output the depth estimation result corresponding to the image to be analyzed; The depth estimation results include: relative depth map D mono and absolute depth map D metric ; S4: Combining the scene analysis results and the depth estimation results, calculate the depth of each vehicle to be analyzed in the image to be analyzed, and output the precise depth D of each vehicle to be analyzed. vehicle ; S5: Combining the depth estimation results and the scene analysis results, calculate the homography matrix H. Based on the homography matrix H, convert the image to be analyzed into a bird's-eye view. Based on the precise depth D of each vehicle... vehicle The bird's-eye view information of the vehicle to be analyzed is calculated; the bird's-eye view information includes: the BEV coordinates of each vehicle and the lane affiliation of each vehicle; Acquire all video frame images captured by the camera to be analyzed; for each vehicle to be analyzed, combine its corresponding bird's-eye view information according to time frames to obtain the vehicle's tracking trajectory; output the vehicle tracking trajectory for each vehicle to be analyzed. S6: Perform time-series fusion of the vehicle tracking trajectory and vehicle speed of the vehicle to be analyzed; Calculate the collision probability between the vehicle and other vehicles to be analyzed based on the distance between the vehicles, and perform uncertainty TTC warning; For vehicles retrieved from the current image to be analyzed, trajectory prediction is performed, and forward-looking TTC is calculated based on the predicted trajectory, and early warning is issued.

[0005] Its further features are: Step S2 includes the following steps: s21: Constructing a task detection model based on YOLOPv2; The task detection model is used to perform task inference on the image to be analyzed. The tasks include vehicle detection, lane line detection, and drivable area detection. s22: The image to be analyzed is fed into the task detection model; s23: Construct the scene-aware module; The P3 layer feature map output by the Neck network of the YOLOPv2 model is obtained and sent to the scene perception module; wherein, the P3 layer feature map output by the Neck network is a high-resolution small target feature map; The scene perception module extracts scene features from the P3 layer feature map according to pre-specified scenes that may affect the image acquisition effect: f scene =MLP(GAP(P3 features ))∈R ds ; Among them, f scene P3 is the scene feature vector; featuresThe P3 layer feature map is output by the Neck network of the YOLOPv2 model; GAP is global average pooling; MLP is a multilayer perceptron, which uses two fully connected layers to map to ds dimensions, corresponding to ds kinds of scenes; s24: Add a dynamic task weight analysis module after the scene perception module, and send the P3 layer feature map processed by the scene perception module into the dynamic task weight analysis module. The dynamic task weight analysis module calculates the uncertainty weight λ(t) corresponding to the t-th time frame for each task detected by the task detection model based on the scene characteristics. λ(t)=W T ·c(t); Among them, W T It is a fixed weight matrix (3×ds) after training, with each column corresponding to a scenario for the weight allocation of the three tasks; c(t) represents the scene confidence score, and c(t) = softmax(f scene ); s25: Perform time-series smoothing on the weights of each task to obtain the smoothed uncertain weights; λ smooth (t)=JU1·λ(t)+JU2·λ smooth (t-1); In the formula, JU1 and JU2 are preset jump adjustment coefficients; s26: Combine the P3 layer feature map processed by the dynamic task weight analysis module with the P4 and P5 layer feature maps output by the YOLOPv2 model Neck network. Figure 1 The scene is fed into the multi-task decoder head; the multi-task decoder head outputs the scene parsing result; the multi-task decoder head includes: a detection head, a lane line segmentation head, and a drivable area segmentation head; Step S2 also includes the following steps: s27: Calculate the uncertainty-weighted total loss L total ; L total =∑ i (L i / 2σ i 2 )+logσ i ; Among them, L i Let σ represent the loss of the i-th task. i This represents the uncertainty weight of the i-th task; Step S3 specifically includes the following operations: s31: Based on the monocular depth estimation Monodepth2, the image to be analyzed is parsed to obtain the inverse depth map corresponding to the image to be analyzed; s32: Obtain the relative depth map D corresponding to the image to be analyzed based on the inverse depth map. mono ; s33: Acquire each lane line and measure the pixel width d between two lane lines in the image to be analyzed. pixel ; s34: For each lane line, obtain the inverse depth value (disparity) of the lane line marking pixels corresponding to each lane line in the image to be analyzed based on the inverse depth map. lane Based on the inverse depth value disparity lane The relative depth corresponding to the lane marking pixels is calculated, and then the average relative depth D for each lane marking is obtained by averaging. lane_avg ; s35: The principal dimensions of the lane lines are calculated; s1 ~ =(W lane ·f y ) / (d pixel ·D lane_avg ); In the formula, s1 ~ W is the lane line scale factor. lane The standard width of a highway lane is d. pixel D represents the pixel width between the two lane lines in the image to be analyzed. lane_avg f is the average relative depth of the lane line area. y The vertical focal length of the camera to be analyzed; s36: Calculate the auxiliary scale for the horizon; s2 ~ =H cam / (f y ·tan(arctan((v horizon -c y ) / f y ) ) ); In the formula, s2 ~ v is the horizon scale factor; horizon c represents the vertical pixel coordinates of the horizon in the image to be analyzed. y H represents the ordinate of the principal point in the image from the camera to be analyzed, in pixels. cam The installation height of the camera to be analyzed; ; In the formula, To calculate the partial derivative of depth D with respect to the vertical pixel coordinate v of a pixel, we take the gradient of depth D along the vertical direction v; argmax represents the pixel position where the maximum value is obtained. s37: Perform Bayesian scale fusion calculation on the main scale of lane lines and the auxiliary scale of the horizon to obtain the mean scale μ after fusion. fused and the scale variance σ after fusion fused ; μ fused =(μ1 / σ1 2 +μ2 / σ2 2 ) / (1 / σ1 2 +1 / σ2 2 ); σ fused 2 =1 / (1 / σ1 2 +1 / σ2 2 ); In the formula, μ1 is the lane line scale factor s1 ~ The mean value, μ2 is the horizon scale factor s2. ~ The mean; σ1 is the lane line scale factor s1 ~ The variance, σ² is the horizon scale factor s². ~ The variance; s38: Calculate the absolute depth and depth confidence; D metric =μ fused ·D mono ; conf scale =1- σ fused / μ fused ; In the formula, D mono D is the relative depth map corresponding to the image to be analyzed. metric This is the absolute depth map corresponding to the image to be analyzed; conf scale D metric The corresponding depth confidence level; Step S4 includes the following operations: s41: Obtain the vehicle detection result in the scene parsing result, and directly regard all pixels within the rectangular vehicle detection box in the vehicle detection result as the vehicle region of the vehicle to be analyzed, and obtain the vehicle segmentation mask corresponding to the vehicle region; the image within the vehicle region is the vehicle image of the vehicle to be analyzed. s42: Calculate the mask gradient of the vehicle segmentation mask using the Sobel operator: G mask 2 =(G x 2 +G y2 ); In the formula, G mask G represents the magnitude of the mask gradient. x For the horizontal Sobel gradient, G y The vertical Sobel gradient; G x =Sobel x (mask), G y =Sobel y (mask); s43: For each vehicle to be analyzed, calculate the core mask of the vehicle segmentation mask. core ; mask core ={(u,v)∈mask|G mask <0.1∩erode(mask,3)=1}; In the formula, (u,v) are the image pixel coordinates in the vehicle detection box, and erode(mask,3) indicates that the mask is eroded using a 3×3 structuring element; s44: Calculate the gradient weights of pixels in the vehicle image of the vehicle to be analyzed; w(u,v)=exp(-β·|▽D(u,v)|); In the formula, w(u,v) is the depth weight corresponding to pixel (u,v), β is the gradient weight adjustment coefficient; ▽D(u,v) represents the gradient of pixel (u,v) in the absolute depth map Dmetric; s45: Divide the vehicle detection bounding box in the vehicle detection result into three equal regions in the vertical direction: top, mid, and bottom. The height of each region is 1 / 3 of the height of the vehicle detection bounding box. Calculate the depth D corresponding to the vehicle image in the k-th region. k ; D k =(∑w(u,v)D(u,v) ) / (∑w(u,v)); In the formula, k∈{top,mid,bot}; s46: Calculate the depth corresponding to the vehicle image of each vehicle to be analyzed; D vehicle =d top ·D top + d mid ·D mid + d bot ·D bot ; Among them, D vehicle For the precise depth of each vehicle; d top d mid and dbot Adjust the depth weights for the three regions; Step S5 includes the following operations in detail: s51: Calculate the theoretical road surface depth corresponding to the image to be analyzed; D theory (v)=H cam / sin(arctan((vc y ) / f y )+θ pitch ); In the formula, D theory (v) represents the theoretical road surface depth corresponding to the v-th row pixel of the image to be analyzed at angle θ; c y For the y-value of the principal point coordinate in the camera image to be analyzed, f y To determine the vertical focal length of the camera to be analyzed, θ pitch H is the preset initial value of the camera's pitch angle to be analyzed; cam The installation height of the camera to be analyzed; s52: The pitch angle of the camera to be analyzed is fitted based on the road surface depth; θ pitch (t) '=argmin θ Σ v [D road (v) - D theory (v) 2 ; In the formula, θ pitch (t)' represents the estimated pitch angle of the camera to be analyzed corresponding to time frame t, D road (v) represents the actual road surface depth corresponding to the v-th row pixel at angle θ; argmin θ : The pitch angle that minimizes the error; s53: Divide the road region in the image to be analyzed into left and right sides, and estimate the roll angle by comparing the road surface depth of the left and right roads in the image to be analyzed; θ roll (t) '=arctan((D left - D right ) / ( α roll ·W / 2)); In the formula, θ roll (t) represents the estimated roll angle of the camera to be analyzed; D left D represents the average actual road surface depth of the road on the left. right α represents the average actual road surface depth of the road on the right; W is the image width; α roll This is the roll angle scaling factor; α roll = Hcam / f x ;f x The horizontal focal length of the camera to be analyzed; s54: Perform Kalman filtering on the angle estimate; θ filt (t)=Kalman update (θ filt (t-1),θ(t)' ); In the formula, Kalman update () represents the Kalman filter, θ filt (t) represents the filtered angle corresponding to time frame t; θ(t)' represents the pitch angle estimate or roll angle estimate corresponding to time frame t; s55: For each frame of the image to be analyzed, recalculate the rotation matrix R based on the real-time angle, and then synthesize the homography matrix H; R(t) = R z (θ yaw )R x (θ pitch ')R y (θ roll '); H(t') = K·[R(t) [:,0:2] |t']; In the formula, R(t) is the rotation matrix corresponding to time frame t, R z Rotate the camera coordinate system around the z-axis by an angle θ yaw R x Rotate the camera coordinate system by an angle θ around the x-axis pitch ', R y Rotate the camera coordinate system around the y-axis by an angle θ roll ';θ yaw yaw angle of camera, collected from vehicle navigation system; [:,0:2] indicates taking the first two columns of rotation matrix, t' represents translation vector, and | represents matrix concatenation operation; Translation vector t'=[0,0,H cam ] T ; s56: Based on the homography matrix H(t'), the pixel coordinates of the image to be analyzed are transformed into the BEV coordinate system of the ground BEV bird's-eye view to obtain the bird's-eye view to be analyzed, and the vehicle tracking trajectory of the vehicle to be analyzed is obtained based on the bird's-eye view to be analyzed. The method for determining the vehicle tracking trajectory of the vehicle to be analyzed includes the following operations: a1: Based on the BEV coordinate system in the bird's-eye view to be analyzed, utilize the precise depth D of the vehicle to be analyzed. vehicle The BEV coordinates (Xw, Yw) corresponding to the vehicle to be processed are calculated. a2: Obtain the lane line segmentation result from the scene parsing result corresponding to the image to be analyzed, and use the lane line pixel mask to obtain the pixel coordinates (u) of the lane line. lane ,v lane Based on the homography matrix H(t'), the lane line pixel coordinates (u) are... lane ,v lane Convert the coordinates to BEV coordinates to obtain the lane line BEV coordinates; a3: Based on the BEV coordinates (Xw, Yw) of the vehicle to be processed, determine which set of lane lines the vehicle is in, and then obtain the lane affiliation of the vehicle to be analyzed corresponding to time frame t; a4: BEV coordinates and lane affiliation of the vehicle to be analyzed, stored as the vehicle bird's-eye view information [BEV coordinates, lane affiliation] corresponding to time frame t; a5: Sort the vehicle bird's-eye view information of the vehicle to be analyzed according to time frames to obtain the vehicle tracking trajectory corresponding to the vehicle to be analyzed; In step S6, the uncertainty TTC early warning process specifically includes the following steps: s61: Modeling the distance uncertainty between the analyzed vehicle and the self-vehicle; The distance variable d between the two vehicles follows a mean of μ d The variance is σ d 2 Gaussian distribution: d ~ N(μ d ,σ d 2 ); σ d = μ d ·σ fused / μ fused ; In the formula, μ fused The mean scale is the fused value of the lane line main scale and the horizon auxiliary scale; σ fused The scale variance after fusing the lane line main scale and the horizon auxiliary scale; The relative velocities of the two vehicles along the direction of collision follow a mean of μ. v The variance is σ v 2 Gaussian distribution: v rel ~N(μ v ,σ v 2 ); In the formula, v rel ‎ is the relative speed of the two vehicles along the direction of collision, measured in meters per second; μ d σ is the mean distance. d2 μ represents the distance variance. v σ represents the mean relative velocity. v 2 The variance of relative velocity; s62: Calculate the mean and variance of TTC; μ TTC ≈μ d / μ v ; σ TTC ≈μ TTC· sqrt((σ d / μ d ) 2 +(σ v / μ v ) 2 ); In the formula, μ TTC σ is the average collision time. TTC The variance of the collision time is sqrt(), which is the square root. s63: Calculate the collision probability between the vehicle to be analyzed and the vehicle itself; P(TTC<τ)=Φ((τ-μ TTC ) / σ TTC ); In the formula, Φ is the Gaussian cumulative distribution function; τ is the preset collision time threshold; P(TTC<τ) represents the probability that the collision time TTC is less than τ; s64: Set warning levels and set different warning conditions for each warning level; Based on the value of P(TTC<τ), when P(TTC<τ) meets the preset warning conditions, TTC warnings of different levels are issued. Step S6, the process of calculating the forward TTC based on the predicted trajectory and providing early warning, includes the following operations: ss61: For the image to be analyzed, construct a dynamic graph attention network G(t)=(V(t),E(t)); Where V(t) represents the set of all detected vehicle nodes included in the image to be processed; E(t) is the set of edges, whose elements are edges e. ij It holds true if and only if the following conditions are met: |Y j -Y i | <IN th ∩lane rel ∈{same lane, left adjacent lane, right adjacent lane}; Among them, Y i and Y j This represents the Y-coordinate values ​​of vehicle node i and vehicle node j in the world coordinate system; lanerel Indicates the lane relationship between vehicle node i and vehicle node j; IN th This is a preset threshold for interaction impact, in meters (m). Features h of vehicle node i in set V(t) i For: h i =[Xw,Yw,v x ,v y [TTC, vehicle category class, detection confidence level conf]; Where (Xw, Yw) are the coordinates of vehicle node j in the world coordinate system, (v x , v y ) represents the velocity of vehicle node i in the x and y directions of the world coordinate system; TTC represents the TTC value between vehicle node i and its own vehicle at time frame t; the values ​​of vehicle class and detection confidence conf are derived from the scene parsing results; ss62: Based on vehicle node i, find all its neighbor nodes, denoted as vehicle node j, and calculate the graph attention feature vector: α ij =softmax j (LeakyReLU(W a ·[h i ||h j ||f ij ])); In the formula, α ij Let f represent the attention weight of vehicle node i to its neighbor vehicle node j, || be the vector concatenation operation, and f be the attention weight of vehicle node i to its neighbor vehicle node j. ij Represents the interaction features between vehicle node i and vehicle node j; LeakyReLU is the activation function; softmax... j This means normalizing all neighbors j of vehicle node i so that the sum of the attention weights of all neighboring vehicles to vehicle node i is 1; W a This is the attention matrix that is fixed after training. f ij =[△Xw, △Yw, △v x , △v y , d ij TTC ij lane rel ]; Where (△Xw, △Yw) represent the relative positions between vehicle node i and vehicle node j; (△v x , △v y ) represents the relative speed between vehicle node i and vehicle node j; d ij Indicates the relative distance between two vehicles; TTC ijThe lane represents the TTC between vehicle node i and vehicle node j. rel Indicates the lane relationship between the two vehicles; ss63: Introduces a gated recurrent unit (GRU) to update the features of node vehicle i; m i =∑ j∈N(i) α ij W m h j ; h i ~ =GRU(h i ,m i ); In the formula, m i The aggregation feature of vehicle node i; W m The mapping matrix is ​​trained to obtain the optimal value; GRU is a gated recurrent unit; N(i) is the set of neighboring vehicle nodes of node i. ss64: Predicting the driving trajectory of node vehicle i using MLP; τ i =MLP dec (h i ~ ={(△X1,△Y1),…,(△X n ,△Y n )}; In the formula, τ i For the predicted trajectory of vehicle node i, MLP dec For MLP decoder; (△X) n ,△Y n ) represents the coordinate offset of vehicle node i in the BEV coordinate system at the nth future step, starting from time frame t, as output by the MLP decoder; ss65: Calculates the TTC of vehicle node i and the vehicle itself based on the predicted driving trajectory; TTC predict =min t {t∣d(τ ego (t),τ i (t)) <D th}; In the formula, TTC predict Let τ be the predicted collision time between vehicle node i and its own vehicle. ego (t) represents the predicted trajectory of the vehicle, D th min is the preset distance threshold. t d() represents the minimum collision time; d() is a function to calculate the straight-line distance between the predicted positions of the two vehicles.

[0006] This application provides a highway assisted driving method that uses video frame images captured by a camera as the images to be analyzed. Scene analysis is performed on the images to obtain vehicle detection results, lane segmentation results, and drivable area segmentation results. Then, depth estimation is performed on the images to obtain depth estimation results. Based on the depth estimation results, the depth information of all vehicles detected in the images to be analyzed is calculated, thereby obtaining the precise depth D of each vehicle. vehicle The homography matrix is ​​calculated based on the depth estimation results. The image to be analyzed is then converted into a bird's-eye view based on the homography matrix. In the BEV coordinate system, depth information is used to calculate the distance between vehicles. Combined with the bird's-eye view, the tracking trajectory of each vehicle is calculated, and subsequent warning calculations are performed using these vehicle tracking trajectories. The tracking process for vehicles other than the vehicle itself is calculated based on video frame images. This method can handle not only single-vehicle tracking but also tracking multiple vehicles, ensuring applicability to highway driving scenarios. In this application, the depth information of the road surface itself is used to infer the real-time camera pose of the vehicle-mounted camera in each frame of the image to be analyzed, obtaining an accurate homography matrix. This ensures the accuracy of subsequent calculations of lane and vehicle positions based on the BEV coordinate system, thereby ensuring accurate calculation of vehicle distances and accurate warnings without the need for a radar module. This method eliminates the need for a radar module, utilizing video frame images collected by the vehicle-mounted camera combined with vehicle speed and other data to achieve assisted driving on highways, effectively reducing the cost of highway assisted driving. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the highway assisted driving method in this application; Figure 2 This is a schematic diagram of the task detection model. Figure 3 This is a schematic diagram of the absolute depth map calculation process; Figure 4 This is a schematic diagram of the vehicle tracking trajectory calculation process; Figure 5 Examples of BEV bird's-eye view and lane trajectory prediction results; Figure 6 This is an example of the TTC warning effect. Detailed Implementation

[0008] like Figure 1 As shown, this application includes a highway assisted driving method, which includes the following steps.

[0009] S1: Obtain the data to be analyzed based on the onboard cameras and sensors installed on the vehicle.

[0010] When intelligent vehicles travel on highways, they are affected by vehicles coming from the front and rear directions within the same lane, as well as from adjacent lanes. Therefore, intelligent vehicles using this method should have at least four cameras positioned in the front, rear, left, and right directions, or surround-view cameras, some of which may be able to observe the ground. To improve computational accuracy, the number of cameras can be increased; the specific number depends on cost control requirements. If the vehicle already has many cameras, data from only a subset of them can be used for computation.

[0011] The vehicle-mounted camera involved in the calculation is denoted as: the camera to be analyzed; the image obtained after preprocessing the video frame images captured in real time by the camera to be analyzed is denoted as: the image to be analyzed; the acquisition time of the image to be analyzed is denoted as: time frame t; the conversion between the specific time frame and the actual time is implemented based on existing technology. For example: 30FPS video: the interval between each frame is 1 / 30 ≈ 0.033 seconds, which is equivalent to 0.033 seconds for 1 time frame. The specific format of the image to be analyzed adopts existing image formats; in this embodiment, RGB format images are used.

[0012] The first step in assisted driving methods is to collect data to be analyzed, which serves as the basis for computation. This data includes: the image to be analyzed, the vehicle speed acquired via the CAN bus, the camera intrinsic parameter matrix K of the camera to be analyzed, and the installation height H of the camera to be analyzed. cam The unit is meters (m). H cam The value is set adaptively according to different vehicle models.

[0013] The camera intrinsic parameter matrix K is a 3×3 matrix, and the elements in the first row of the matrix are [f x 0 c y The second line element is [0f] y c y The third row contains elements [0 0 1]. Where f x and f y The horizontal and vertical focal lengths of the camera to be analyzed are given in pixels; (c x ,c y () represents the coordinates of the principal point in the camera image, in pixels.

[0014] The vehicle-mounted camera captures video frames in real time and sends them to the real-time operating system (such as QNX) responsible for driving safety. This method is integrated into the real-time operating system, which converts the video into video frame images, performs image preprocessing, and then performs subsequent calculations. The specific video frame image conversion method and image preprocessing are implemented based on existing technologies. In this embodiment, the preprocessing includes: using the Brown-Conrady model combined with ISP enhancement and ROI cropping methods to perform distortion correction on the video frames, eliminating geometric distortion, making the processed image to be analyzed closer to the real environment, thereby ensuring the accuracy of subsequent calculations.

[0015] S2: Acquire any image to be analyzed, perform multi-task inference based on the image, and output the scene analysis results of the highway environment included in the image in real time. The scene analysis results include: vehicle detection results, lane line segmentation results, and drivable area segmentation results; each vehicle detection result corresponds to one vehicle to be analyzed.

[0016] Multi-task inference of the image to be analyzed can be implemented based on existing algorithms and models. In this embodiment, YOLOPv2 is used to implement multi-task detection. In the scene parsing results output by YOLOPv2, the vehicle detection results are represented by vehicle detection boxes (BBoxes), and the confidence level and vehicle category are also output; the lane line segmentation results are represented by lane line segmentation masks; and the drivable region segmentation results are represented by drivable region masks (DA Masks).

[0017] Specifically, step S2 includes the following operations.

[0018] s21: Constructing a task detection model based on YOLOPv2; The task detection model is used to perform task inference on the image to be analyzed. The tasks include vehicle detection, lane line detection, and drivable area detection.

[0019] like Figure 2As shown, the task detection model includes a backbone network, a neck, and a head. The backbone network is a main network built on the YOLOv7 architecture, with an E-ELAN (Extended Efficient Layer Aggregation Network) module to enhance feature extraction capabilities while maintaining computational efficiency. The neck consists of a bidirectional pyramid constructed from FPN (Feature Pyramid Network) and PAN (Path Aggregation Network). The head includes multi-task decoder heads: a detection head, a lane line segmentation head, and a drivable area segmentation head.

[0020] The data flow in the task detection model is as follows: the backbone network extracts multi-layer features from the input image and sends them to the neck network. The neck network fuses and processes the received multi-layer feature maps to generate a multi-scale feature map sequence, which corresponds to the P3 layer feature map, the P4 layer feature map and the P5 layer feature map. P3 layer feature map: high resolution (80×80), strong semantics, specializing in small targets.

[0021] P4 layer feature map: medium resolution (40×40), medium semantics, specializing in medium-sized targets.

[0022] P5 layer feature map: low resolution (20×20), strong semantics, specializing in large targets.

[0023] The calculation of the target detection head is based on three layers of features: P3, P4, and P5; the calculation of the lane line segment head and the drivable area segment head is based on two layers of features: P3 and P4.

[0024] This application constructs a scene perception module and a dynamic task weight analysis module in the task detection model to improve the feature representation of small targets in the P3 layer feature map.

[0025] s22: Input the image to be analyzed into the task detection model.

[0026] s23: Construct the scene perception module; obtain the P3 layer feature map output by the Neck network of the YOLOPv2 model and send it into the scene perception module.

[0027] The scene perception module defines ds common highway driving scenarios, with specific scenarios pre-set according to the region where the highway is located. In this embodiment, ds is set to 6, corresponding to 6 common driving scenarios, including: normal, nighttime, foggy, tunnel, rainy, and congested.

[0028] The scene awareness module extracts scene features from the P3 layer feature map according to pre-specified scenes that may affect image acquisition quality: f scene =MLP(GAP(P3 features ))∈R ds ; Among them, f scene The extracted scene feature vector represents the P3-based feature vector. feature The determination result of the scene to which the image to be analyzed belongs. (P3) features This is the P3 layer feature map output by the Neck network of the YOLOPv2 model. GAP is global average pooling, which compresses the entire feature map into a single vector, representing "global scene" information. MLP is a multilayer perceptron, which uses two fully connected layers to map to ds dimensions, corresponding to ds kinds of scenes.

[0029] s24: Add a dynamic task weight analysis module after the scene perception module, and send the P3 layer feature map processed by the scene perception module into the dynamic task weight analysis module. The dynamic task weight analysis module assigns an uncertainty weight λ(t) to each task detected by the task detection model at the t-th time frame based on the scene features extracted by the scene perception module. λ(t)=W T ·c(t); c(t) = softmax(f scene ); Where c(t) is the scene confidence score, and softmax normalizes the preset ds original scores into a probability distribution so that the sum of the probabilities corresponding to the ds scenes is 1. For example, in this embodiment, ds=6, for 6 scenes: normal, night, fog, tunnel, rain, and congestion, the calculated result is: c(t)= [0.9, 0.02, 0.02, 0.02, 0.02, 0.02]; it can be seen that the confidence score of 0.9 is the highest for the normal scene, so the scene corresponding to the image to be analyzed is the normal scene, that is, the high-speed driving scene on a sunny day.

[0030] W T It is a fixed weight matrix (3×6) after training. Each column corresponds to the weight allocation for the three tasks in a specific scenario. This is used to adjust the model's emphasis on the three detection tasks in different scenarios, making the detection results more consistent with the environment of highway assisted driving and thus more practical. Training WT The model is trained using historical data on the driving performance of vehicles with the lowest accident probability in different scenarios, constructing training samples. In this embodiment, the weight matrix obtained after training is as follows: The weights for the standard scenarios are [0.4, 0.3, 0.3], while the weight for the target vehicle detection task is slightly higher at 0.4. The weights for nighttime scenes are [0.6.0.2.0.2], with the target vehicle detection task having the highest weight of 0.6, indicating that the focus is on detecting target vehicles in the driving environment at night. The weights for foggy scenes are [0.3, 0.2, 0.5]. The weight for the drivable area detection task is the highest at 0.5, indicating that there are difficulties in recognizing various tasks in foggy weather, vehicle targets are blurred, and forcibly using high weights for detection will result in false detections; therefore, the focus is on detecting the drivable area of ​​the vehicle to ensure that the vehicle is driving in the correct lane. The weights for the tunnel scenario are [0.3, 0.5, 0.2], while the weight for the lane detection task is the highest at 0.5. The weights for rainy scenes are [0.5, 0.1, 0.4], and the weight for lane detection tasks is the lowest at 0.1. The weights for congestion scenarios are [0.6, 0.2, 0.2], while the target vehicle detection task has the highest weight of 0.6.

[0031] s25: Perform time-series smoothing on the weights of each task to obtain the smoothed uncertain weights; λ smooth (t)=JU1·λ(t)+JU2·λ smooth (t-1); In the formula, JU1 and JU2 are preset jump adjustment coefficients; the values ​​of JU1 and JU2 are engineering experience values ​​based on historical data collection; in this embodiment, JU1=0.1, JU2=0.9, λ smooth (t) = 0.1·λ(t) + 0.9·λ smooth (t-1).

[0032] By using temporal smoothing, we can prevent weight jitter caused by scene changes between adjacent frames. For example, a frame may show a momentary shadow but be misjudged as nighttime.

[0033] s26: Combine the P3 layer feature map processed by the dynamic task weight analysis module with the P4 and P5 layer feature maps output by the YOLOPv2 model's Neck network. Figure 1 The data is fed into the multi-task decoder head of the header; the multi-task decoder head outputs the scene parsing results.

[0034] In existing technologies, YOLOPv2 uses fixed weights for its three tasks, but high-speed scenarios are extremely diverse. This method, through a scene-aware module and a dynamic task weight analysis module, enables the model to adaptively and dynamically adjust the weights of each detection task according to the scene, effectively improving the accuracy of the detection results. This method, by designing dynamic weights, allows a single task detection model to adapt to all weather and lighting conditions without needing to train multiple dedicated models, effectively reducing the model's resource consumption and making it more suitable for resource-constrained applications such as in-vehicle systems.

[0035] s27: Calculate the uncertainty-weighted total loss L total The model automatically adjusts the proportion of each sub-loss for the three detection tasks in dynamic scenarios, avoiding manual parameter tuning, achieving collaborative optimization and performance balance during training, and improving the model's generalization ability.

[0036] L total =∑ i (L i / 2σ i 2 )+logσ i ; Among them, L i Let σ represent the loss of the i-th task. i This represents the uncertainty weight of the i-th task.

[0037] S3: Perform depth estimation on the image to be analyzed and output the depth estimation result corresponding to the image to be analyzed; The depth estimation results include: relative depth map D mono and absolute depth map D metric .

[0038] This method utilizes Monodepth2, a monocular depth estimation model based on self-supervised learning, to estimate the depth of the image to be analyzed. Existing Monodepth2 techniques reconstruct adjacent frames by predicting pose transformations and depth maps between them, unifying two training methods into a pose transformation-based framework. The Monodepth2 model comprises two networks: a depth prediction network (a depth encoder ResNet18 and a multi-scale depth decoder U-Net) and a pose transformation prediction network (ResNet18).

[0039] Monodepth2's input is the current frame l t and adjacent frames, which include: the previous frame l t-1 or the next frame l t+1The depth encoder, serving as the feature extraction backbone, employs a pre-trained ResNet18 architecture to progressively downsample the input image, generating feature maps at five different scales. The multi-scale depth decoder, based on the U-net structure, progressively fuses features from each encoder layer and restores them to their original resolution through upsampling and skip connections. The depth prediction network outputs a predicted depth map for the current input frame; the final output inverse depth value is constrained to the range of 0-1, representing a relative depth ratio rather than an absolute distance. Monodepth2 is a dense depth estimation network that outputs a relative depth value for each pixel of the input image, ultimately producing a complete depth map of the same size as the original image. The pose transformation prediction network, based on the ResNet18 architecture, takes the current frame and adjacent frames as input and predicts the pose transformation between them.

[0040] This application calculates the absolute depth map D based on the output of the Monodepth2 model. metric Specifically, as follows Figure 3 As shown, step S3 specifically includes the following operations: s31: Based on Monodepth2, the image to be analyzed is parsed to obtain the inverse depth map corresponding to the image to be analyzed.

[0041] s32: Obtain the relative depth map D corresponding to the image to be analyzed based on the inverse depth map. mono The method for converting between inverse depth maps and relative depth maps is implemented based on existing technology. The resulting relative depth map D is... mono After that, the restoration scale factor s needs to be calculated in order to obtain the absolute depth D. metric :D metric = s × D mono .

[0042] s33: Using the lane line segmentation mask from the scene parsing results, obtain each lane line, and measure the pixel width d between two lane lines in each lane in the image to be analyzed. pixel This gives the pixel width of the lane.

[0043] s34: For each lane line, obtain the inverse depth value (disparity) of the lane line marking pixels corresponding to each lane line in the image to be analyzed based on the inverse depth map. lane Based on the inverse depth value disparity lane The relative depths corresponding to the lane marking pixels are calculated, and then the average relative depth D for each lane marking in the image to be analyzed is obtained by averaging. lane_avg .

[0044] s35: For each lane line, calculate the corresponding lane line principal scale; s1~ =(W lane ·f y ) / (d pixel ·D lane_avg ); In the formula, s1 ~ W is the lane line scale factor. lane The standard width of a highway lane is 3.75m (as per national standards). pixel D represents the pixel width between the two lane lines in the image to be analyzed. lane_avg f is the average relative depth of the lane line area. y The vertical focal length of the camera to be analyzed.

[0045] For images captured by vehicle-mounted cameras, the following relationship exists: True width = Pixel width × True length represented by each pixel. Assuming the camera is square, fx ≈ fy, according to the imaging principle of pinhole cameras, the following relationship holds: The absolute physical distance between lane lines and cameras (Dis) ab =Standard lane width W lane ×(vertical focal length f) y / lane pixel width d pixel ).

[0046] The relative depth map D can be calculated from the image output by Monodepth2. mono Relative depth map D mono The relative depth value in the image reflects the relative distance between that point and the camera. Extracting the average relative depth of this lane line region from the relative depth map is equivalent to calculating the relative distance between the lane line and the camera in the image being analyzed. rel And the absolute physical distance Dis ab Equivalent to the absolute depth D of the lane line metric Then the relation Dis is satisfied. ab =s×Dis rel .

[0047] Dis ab After substituting the formula, the lane line scale factor s1 is obtained. ~ The calculation formula.

[0048] s36: Calculate the auxiliary scale for the horizon.

[0049] In the pinhole camera model, when the camera is horizontally mounted and the ground is an ideal plane, the depth d of any point on the road surface below the horizon (i.e., the vertical distance from that point to the ground directly below the camera) is compared with the vertical coordinate v of that point in the image's pixel coordinate system and the vertical coordinate v of the horizon. horizon Camera installation height H cam And the camera focal length f, satisfying the following geometric relationship: d=H cam ·f / (v horizon -v); Using the vertical coordinate v of the horizon in the image horizon and camera installation height H cam Based on this geometric relationship, the auxiliary scale of the horizon can be deduced.

[0050] ; In the formula, To calculate the partial derivative of depth D with respect to the vertical pixel coordinate v of a pixel, we take the gradient of depth D along the vertical direction v; argmax represents the pixel position where the absolute value of the gradient is the largest.

[0051] s2 ~ =H cam / (f y ·tan(arctan((v horizon - c y ) / f y ) ) ); In the formula, s2 ~ v is the horizon scale factor; horizon c represents the vertical pixel coordinates of the horizon in the image to be analyzed. y Let be the ordinate of the principal point of the image from the camera to be analyzed in the pixel coordinate system.

[0052] s37: Perform Bayesian scale fusion calculation on the main scale of lane lines and the auxiliary scale of the horizon to obtain the mean scale μ after fusion. fused and the scale variance σ after fusion fused ; μ fused =(μ1 / σ1 2 +μ2 / σ2 2 ) / (1 / σ1 2 +1 / σ2 2 ); σ fused 2 =1 / (1 / σ1 2 +1 / σ2 2 ); In the formula, μ1 is the lane line scale factor s1 ~ The mean value, μ2 is the horizon scale factor s2. ~ The mean; σ1 is the lane line scale factor s1 ~ The variance, σ² is the horizon scale factor s². ~ The variance.

[0053] After weighted average calculation, the road with smaller variance has a larger weight and is more reliable. If lane markings are clear, σ1 is smaller, and s1 is also smaller. ~When lane lines are blurred, σ² is smaller, and the horizon dominates. fused The smaller the value, the more reliable the fusion result.

[0054] s38: Calculate the absolute depth and depth confidence; D metric =μ fused ·D mono ; conf scale =1- σ fused / μ fused ; In the formula, D mono D represents the relative depth value in the relative depth map corresponding to the image to be analyzed. metric Here are the absolute depth values ​​in the absolute depth map corresponding to the image to be analyzed; the unit of absolute depth is meters. scale D metric The corresponding depth confidence level. fscale A value close to 1 indicates that the two estimates are highly consistent, and the output is reliable.

[0055] Monocular cameras inherently cannot determine absolute distance (scale ambiguity), which is a physical limitation. This method introduces geometric constraints using two known physical quantities: the standard lane width W. lane Camera installation height H cam This anchors relative depth to the real world. The application designs s1... ~ For lane line scale factor and horizon scale factor s2 ~ The method uses two-way scale factor estimation, two independent estimates, and then performs Bayesian fusion, which is more robust than the method that relies on a single-way scale factor. At the same time, it provides a confidence score so that downstream users know "whether this depth estimate is reliable" and ensures that the method can calculate accurate distance values ​​without radar ranging.

[0056] S4: Combining the scene analysis results and depth estimation results, calculate the depth of each vehicle to be analyzed in the image, and output the precise depth D of each vehicle to be analyzed. vehicle .

[0057] The scene analysis results include vehicle detection bounding boxes (BBoxes), but directly taking the average depth of all pixels within the vehicle's BBox is inaccurate: edge pixel depths span the foreground and background, "contaminating" the depth of distant background pixels; the depth meanings of the upper part (roof / glass) and the lower part (bumper / ground) of the vehicle are different. Therefore, the average depth of all pixels within the vehicle's BBox cannot be directly used as the accurate depth of the vehicle being analyzed. This method designs a calculation method to improve the accuracy of the results. Step S4 includes the following operations.

[0058] s41: Obtain the vehicle detection results from the scene parsing results. Treat all pixels within the rectangular vehicle detection boxes in the vehicle detection results as the vehicle region to be analyzed, and obtain the binary form of the vehicle segmentation mask corresponding to the vehicle region: vehicle image pixels = 1, background = 0. The image within the vehicle region is considered the vehicle image to be analyzed.

[0059] s42: Calculate the mask gradient of the vehicle segmentation mask using the Sobel operator: G mask 2 =(G x 2 +G y 2 ); In the formula, G mask G represents the magnitude of the mask gradient. x For the horizontal Sobel gradient, G y The vertical Sobel gradient; G x =Sobel x (mask), G y =Sobel y (mask)

[0060] The edges of vehicle images are found by mask gradient. The gradient is large at the mask boundary, and the corresponding pixel is a mixed area of ​​"half vehicle and half background", so the depth is unreliable; the area with small gradient is pure vehicle interior, so the depth is reliable.

[0061] s43: For each vehicle to be analyzed, calculate the core mask of the vehicle segmentation mask. core ; mask core ={(u,v)∈mask|G mask <0.1∩erode(mask,3)=1}; In the formula, (u,v) are the image pixel coordinates in the vehicle detection box, erode(mask,3) indicates that the mask is eroded using a 3×3 structuring element; erode(mask,3)=1 indicates that the pixel is retained after erosion and its value is still 1.

[0062] Using a 3x3 structuring element erosion operation, remove a few pixels from the edge of the mask (set them to 0), keeping only the central area of ​​the vehicle (set to 1), resulting in the mask. core It is the "clean vehicle area" after removing edge noise.

[0063] After eroding the vehicle segmentation mask using the erosion operation, the output is still a binary image. The vehicle boundary pixels are removed, and only the stable area inside the vehicle is retained. The purpose is to avoid the depth value at the vehicle boundary being disturbed by the background, and to use only the depth of the vehicle center area to estimate the vehicle distance, thereby improving accuracy.

[0064] s44: Calculate the gradient weights of pixels in the vehicle image of the vehicle to be analyzed; w(u,v)=exp(-β·|▽D(u,v)|); In the formula, w(u,v) is the depth weight corresponding to pixel (u,v), β is the gradient weight adjustment coefficient, and the specific value of β is obtained based on historical data. In this embodiment, β is set to 0.5 based on engineering experience. ▽D(u,v) represents the depth weight of pixel (u,v) in the absolute depth map D. metric The corresponding gradient.

[0065] Areas with large gradients in the depth map (abrupt depth changes) indicate unstable depth estimation, so the gradient weights should be small. More stable pixels should have larger gradient weights.

[0066] s45: Divide the vehicle detection bounding box in the vehicle detection result into three equal regions in the vertical direction: top, mid, and bottom. The height of each region is 1 / 3 of the height of the vehicle detection bounding box. Calculate the depth D corresponding to the vehicle image in the k-th region. k ; D k =(∑w(u,v)D(u,v) ) / (∑w(u,v)); In the formula, k∈{top,mid,bot}.

[0067] s46: Calculate the depth corresponding to the vehicle image of each vehicle to be analyzed; D vehicle =d top ·D top + d mid ·D mid + d bot ·D bot ; Among them, D vehicle For the precise depth of each vehicle, in meters; d top d mid and d bot Adjust the depth weights for the three regions. top d mid and d bot These are fixed values ​​based on engineering experience. In this embodiment, the upper area of ​​the vehicle (roof) has the following characteristics: reflections, glass, and unstable depth. The weight d is... top = 0.15; Vehicle mid-zone (body): Relatively stable, weight d mid=0.35; Vehicle lower zone (bumper / rear wheel): closest to the road surface, most relevant to the actual vehicle distance, weight d bot = 0.50.

[0068] According to experimental data, the average depth calculated by simply taking the mean value of all pixels in the vehicle detection bounding box can have an error of 2-3 meters compared to the true value, which can lead to a huge discrepancy in the calculated distance between vehicles in high-speed scenarios. In this method, a semantic boundary awareness method based on mask gradients is used to exclude contaminated pixels in the "half foreground, half background" region, and then weights are assigned according to the depth reliability of different parts of the vehicle, which greatly improves the accuracy of the final vehicle distance estimation.

[0069] S5: Combining depth estimation results and scene analysis results, the homography matrix H is calculated. Based on the homography matrix H, the image to be analyzed is converted into a bird's-eye view, and the precise depth D of each vehicle is calculated. vehicle The system calculates the vehicle bird's-eye view information of the vehicle to be analyzed, including the BEV coordinates and lane affiliation of each vehicle; it acquires all video frame images captured by the camera to be analyzed; for each vehicle to be analyzed, its corresponding vehicle bird's-eye view information is combined according to time frames to obtain the vehicle's corresponding tracking trajectory; and it outputs the vehicle tracking trajectory of each vehicle to be analyzed.

[0070] Images captured by vehicle-mounted cameras are perspective projections, where objects appear larger than they are, making it impossible to directly measure vehicle distance and lateral offset. This method transforms the image coordinates to a top-down 2D plane with the vehicle as the origin based on BEV projection, directly reading the data for calculation. For example... Figure 4 As shown, step S5 includes the following operations in detail.

[0071] s51: Calculate the theoretical road surface depth corresponding to the image to be analyzed; D theory (v)=H cam / sin(arctan((vc y ) / f y )+θ pitch ); In the formula, D theory (v) represents the theoretical road surface depth corresponding to the v-th row pixel of the image to be analyzed at angle θ; c y f represents the y-coordinate of the principal point in the camera image to be analyzed in the pixel coordinate system. y To determine the vertical focal length of the camera to be analyzed, θ pitch In this embodiment, θ is the preset initial value of the camera's pitch angle to be analyzed. pitch Set to 0; H cam The installation height of the camera to be analyzed.

[0072] s52: Based on the road surface depth, fit the pitch angle of the camera to be analyzed to correct the projection error caused by the camera pitch. θ pitch (t) '=argmin θ Σ v [D road (v) - D theory (v) 2 ; In the formula, θ pitch (t)' represents the estimated pitch angle of the camera to be analyzed corresponding to time frame t, D road (v) represents the actual road surface depth corresponding to the v-th row pixel at angle θ, with specific values ​​derived from the absolute depth map D. metric Get from; argmin θ : The pitch angle that minimizes the error.

[0073] s53: Divide the road region in the image to be analyzed into left and right sides. Estimate the roll angle by comparing the road surface depth of the left and right roads in the image; and estimate the roll angle using θ. roll (t) 'Corrects errors caused by camera tilt; θ roll (t) '=arctan((D left - D right ) / (α roll ·W / 2)); In the formula, θ roll (t) represents the estimated roll angle of the camera to be analyzed; D left D represents the average actual road surface depth of the road on the left. right α represents the average actual road surface depth of the road on the right; W is the image width; α roll This is the roll angle scaling factor; α roll = H cam / f x ;f x The horizontal focal length of the camera to be analyzed is expressed in pixels.

[0074] s54: Perform Kalman filtering on the angle estimate to eliminate frame-by-frame jitter, maintain the temporal stability of the homography matrix, and avoid BEV coordinate jumps; θ filt (t)=Kalman update (θ filt (t-1),θ(t)' ); In the formula, Kalman update () represents the Kalman filter, θ filt (t) represents the filtered angle corresponding to time frame t; θ(t)' represents the pitch angle estimate or roll angle estimate corresponding to time frame t.

[0075] s55: For each frame of the image to be analyzed, the rotation matrix R is recalculated based on the real-time angle, and then the homography matrix H is synthesized. The homography matrix in this method is updated online frame by frame, so that the camera can adaptively adjust the angle deviation.

[0076] R(t) = R z (θ yaw )R x (θ pitch ')R y (θ roll '); H(t') = K·[R(t) [:,0:2] |t']; In the formula, R(t) is the rotation matrix corresponding to time frame t, R z Rotate the camera coordinate system around the z-axis by an angle θ yaw R x Rotate the camera coordinate system by an angle θ around the x-axis pitch ', R y Rotate the camera coordinate system around the y-axis by an angle θ roll '; The camera coordinate system is first rotated around the z-axis, then around the x-axis, and then around the y-axis, finally obtaining the rotation matrix R(t).

[0077] θ yaw The yaw angle is the camera angle, collected from the vehicle navigation system; [:,0:2] indicates taking the first two columns of the rotation matrix, t' represents the translation vector, and | represents the matrix concatenation operation; the translation vector t'=[0,0,H cam ] T .

[0078] s56: Based on the homography matrix H(t'), a ground-based BEV bird's-eye view is obtained using images from multi-view vehicle-mounted cameras. The pixel coordinates of the images to be analyzed are then transformed into the meter-level BEV coordinate system of the ground-based bird's-eye view to obtain the bird's-eye view itself. This yields the BEV coordinates (Xw, Yw) for each vehicle, and the vehicle tracking trajectory is obtained based on the bird's-eye view. For example... Figure 5 The image shown is an example of a ground-based BEV diagram in this embodiment. In the vehicle-mounted scenario, the world coordinate system (Xw, Yw) is the same as the ground-based BEV coordinate system, so in this application, the ground-based BEV coordinate system is represented by (Xw, Yw, Zw). The origin of the ground-based BEV coordinate system is the vehicle, the Xw axis is positive along the vehicle's forward direction, the Yw axis is positive along the vehicle's leftward lateral direction, and Zw=0 in the ground-based BEV coordinate system.

[0079] Based on existing BEV projection algorithms, the homography matrix H(t) is typically a fixed matrix based on offline calibration. However, without considering the changes in the pitch and roll angles of the onboard camera during actual driving, the BEV projection may shift by several meters due to inclines, declines, or bumps, leading to incorrect lane assignment. In this method, the theoretical road surface depth D corresponding to the v-th row of pixels in the image to be analyzed at angle θ is first calculated. theory (v) represents the actual road surface depth D corresponding to the v-th row pixel in the image to be analyzed at angle θ, obtained by measuring the road surface depth using Monodepth2. road (v); then through argmin θ Σ v [D road (v) - D theory (v) 2 Find the optimal value θ pitch (t)', determine the current θ angle, obtain the current angle of the camera, and then use θ pitch (t) 'Calculates the corrected homography matrix H(t)'. This method uses the road surface's own depth information to infer the camera pose in each frame, using the road surface as a level, eliminating the need for additional sensors and achieving zero-cost online calibration. This application performs real-time correction of the homography matrix H(t) for each video frame to obtain the homography matrix H(t'), ensuring the accuracy of the converted BEV coordinates, and thus ensuring the accuracy of lane assignment determination.

[0080] The method for determining the vehicle tracking trajectory of the vehicle to be analyzed includes the following operations: a1: Based on the BEV coordinate system in the bird's-eye view to be analyzed, utilize the precise depth D of the vehicle to be analyzed. vehicle The BEV coordinates (Xw, Yw) corresponding to the vehicle to be processed are calculated. a2: Obtain the lane line segmentation result from the scene parsing result corresponding to the image to be analyzed, and use the lane line pixel mask to obtain the pixel coordinates (u) of the lane line. lane ,v lane Based on the homography matrix H(t'), the lane line pixel coordinates (u) are... lane ,v lane Convert the coordinates to BEV coordinates to obtain the lane line BEV coordinates; a3: Based on the BEV coordinates (Xw, Yw) of the vehicle to be processed, determine which set of lane lines the vehicle is in, and then obtain the lane affiliation of the vehicle to be analyzed corresponding to time frame t; In actual calculations, because the standard width W of a national standard highway lane... lane = 3.75m, so when the lateral distance between a vehicle and your own vehicle is within ±1.875m, it can be judged as a vehicle in the same lane; a4: BEV coordinates and lane affiliation of the vehicle to be analyzed, stored as the vehicle bird's-eye view information [BEV coordinates, lane affiliation] corresponding to time frame t; a5: Sort the vehicle bird's-eye view information of the vehicle to be analyzed according to time frames to obtain the vehicle tracking trajectory corresponding to the vehicle to be analyzed.

[0081] Suppose that the vehicle's BEV coordinates in frame t are (Xw, Yw). t ; The vehicle's BEV coordinates in frame t-1 are: (Xw, Yw) t-1 ; The vehicle's BEV coordinates in frame t-2 are: (Xw, Yw) t-2 ; The vehicle's BEV coordinates in frame tn are: (Xw, Yw) t-n ; The vehicle trajectory is then: [(Xw, Yw)] t-n …(Xw, Yw) t-2 , (Xw, Yw) t-1 , (Xw, Yw) t ].

[0082] S6: Perform time-series fusion of the vehicle tracking trajectory and vehicle speed of the vehicle to be analyzed; Calculate the collision probability between the vehicle and other vehicles to be analyzed based on the distance between the vehicles, and perform uncertainty TTC warning; For vehicles retrieved from the current image to be analyzed, trajectory prediction is performed, and forward-looking TTC is calculated based on the predicted trajectory, and early warning is issued.

[0083] TTC (Time to Collision): This refers to the time it takes for a vehicle to reach the vehicle in front at its current speed, usually measured in seconds. The existing formula for calculating TTC is: TTC = Distance between vehicles d / Relative speed of the two vehicles along the direction of collision v. rel However, during real-time calculations during driving, d and v rel Both methods involve measurement errors and inherent uncertainties. To improve the accuracy of early warnings, this method designs a deterministic TTC early warning method. Specifically, in step S6, the uncertain TTC early warning process includes the following steps.

[0084] In step S6, the uncertainty TTC early warning process specifically includes the following steps: s61: Modeling the uncertainty of the distance between the vehicle and the other vehicle: Assume that the distance variable d between the two vehicles and the relative speed of the two vehicles along the collision direction both follow a Gaussian distribution; The distance variable d between the two vehicles follows a mean of μ d The variance is σ d 2 Gaussian distribution: d ~ N(μ d ,σ d 2 ); σ d = μ d ·σ fused / μ fused ; The relative velocities of the two vehicles along the direction of collision follow a mean of μ. v The variance is σ v 2 Gaussian distribution: μ fused The mean scale is the fused value of the lane line main scale and the horizon auxiliary scale; σ fused The scale variance after fusing the lane line main scale and the horizon auxiliary scale; v rel ~ N(μ v ,σ v 2 ); In the formula, v rel ‎ is the relative speed of the two vehicles along the direction of collision, measured in meters per second; μ d σ is the mean distance between vehicles. d 2 μ represents the variance of the distance between vehicles. v σ represents the mean relative velocity. v 2 Let μ be the relative velocity variance. In actual calculations, μ... v The difference from adjacent frame d, σ v The standard deviation of the relative velocity estimate is calculated from the statistical fluctuations of the velocity difference sequence over the past N frames, reflecting the stability of the velocity estimate.

[0085] s62: Calculate the mean and variance of TTC; Construct an error propagation formula to describe d and v rel How does uncertainty propagate to the TTC? μ TTC ≈ μ d / μ v ; σ TTC ≈ μ TTC· sqrt((σ d / μ d ) 2 +(σ v / μ v ) 2 ); In the formula, μ TTC σ is the average collision time. TTC Let be the variance of the collision time, and sqrt() be the square root.

[0086] s63: Calculate the collision probability between the vehicle to be analyzed and the vehicle itself; P(TTC<τ)=Φ((τ-μ TTC ) / σ TTC ); In the formula, Φ is the Gaussian cumulative distribution function; τ is the preset collision time threshold, the specific value of which is set according to the parameters of the intelligent vehicle itself, such as 4s, 2s, or 1s; P(TTC<τ) represents the probability that the collision time TTC is less than τ. For example: μ TTC = 3s,σ TTC = 0.335s, τ = 4s, calculate P=Φ(2.985)≈0.998.

[0087] s64: Set warning levels and set different warning conditions for each warning level; Based on the value of P(TTC<τ), different levels of TTC warnings are issued when P(TTC<τ) meets the preset warning conditions. The specific warning conditions and warning levels are set according to the actual needs of the vehicle. In this implementation, four warning levels are designed, with corresponding collision time thresholds and probability thresholds set as warning conditions for each TTC level. The warning levels include: Green, Yellow, Red, and Emergency; the specific corresponding warning conditions are as follows: Green∶P(TTC<4.0 s)<0.05; Yellow∶P(TTC<4.0 s)>0.30; Red∶P(TTC<2.0 s)>0.50; Emergency: P(TTC<1.0 s)>0.70.

[0088] Traditional TTC (Traffic Traction Control) directly uses the single-point value of depth / velocity (d / v), but depth estimation in high-speed scenarios has errors. If d=14m but the actual depth is 13m, the TTC timeout changes from 2.8s to 2.6s, resulting in a completely different warning timing. This method, based on uncertain TTC, explicitly models the error as a probability, giving the driver not "TTC=3s" but "98% probability of collision within 3s," which is more in line with actual risk. The TTC warning effect is referenced... Figure 6 .

[0089] Because the risks of high-speed driving come not only from vehicles in the same lane directly in front, but also from vehicles in adjacent lanes, long-distance fast lanes, and slow lanes, there are potential risks such as lane changes, cutting in, slowing down, and occupying lanes, which must be perceived throughout the entire journey; therefore, as a safety redundancy design for high-speed assisted driving, this method will combine multi-vehicle full-scenario tracking with full-domain TTC warning, and calculate the distance, trajectory, and collision probability of all detectable vehicles in front and in adjacent lanes at a long distance, so as to prevent sudden lane change risks in advance.

[0090] Figure 6 The vehicle in blue is traveling at 130 km / h in five lanes. All vehicles in front of the vehicle are referred to as "the preceding vehicle." The red dashed box represents the visual marker (BBox) of the vehicle detection results output by the YOLOPv2 model for the target vehicle ahead. Binding the red preceding vehicle to the dashed box indicates that this vehicle has been successfully identified, included in tracking, and its distance and TTC (Time to Collision) are calculated in real time; it is a valid traffic participant perceived by the algorithm. The 60m marked in red represents the estimated straight-line absolute distance between the vehicle and the red preceding vehicle, in meters, calculated using monocular depth estimation and BEV (Balance of Vehicle) bird's-eye view coordinates. TTC≈43.2s is the calculated TTC collision time between the vehicle and the red preceding vehicle. Theoretically, a collision will occur in 43.2 seconds, assuming both vehicles maintain their current speed and trajectory. Because the distance between the two vehicles is large and the relative speed difference is small, the TTC value is high, constituting a safe long-range warning for situational awareness only, without triggering a hazard warning.

[0091] The image labeled "Depth" in the lower right corner of the figure is a visualization of monocular depth estimation, corresponding to the depth estimation result of the image to be analyzed output in step S3 of this application. It visually displays the relative / absolute depth of each pixel using grayscale; the depth of color represents distance: generally, brighter / lighter colors indicate closer distances, and darker / deeper colors indicate farther distances. This visual demonstration of the algorithm's depth estimation effect on road surfaces, vehicles, and distant views is crucial for subsequent calculations of vehicle distance and accurate vehicle depth D. vehicle The underlying basis for scale integration.

[0092] This method constructs a forward-looking TTC warning by considering not only the vehicle's adjacent vehicles but also the adjacent vehicles' neighboring vehicles, which better reflects the complex driving environment on highways in real life.

[0093] Step S6, which involves calculating the forward-looking TTC based on the predicted trajectory and providing an early warning, includes the following operations.

[0094] ss61: For the image to be analyzed, construct a dynamic graph attention network G(t)=(V(t),E(t)); Where V(t) represents the set of all vehicle nodes detected in the image to be processed in the current frame; E(t) is the set of edges, whose elements are edges e.ij It holds true if and only if the following conditions are met: |Y j -Y i | <IN th ∩lane rel ∈{same lane, left adjacent lane, right adjacent lane}; Among them, Y i and Y j This represents the Y-coordinate values ​​of vehicle node i and vehicle node j in the world coordinate system; lane rel Indicates the lane relationship between vehicle node i and vehicle node j; IN th The preset interaction impact threshold is expressed in meters (m); in this embodiment, IN th =30m is a value based on the engineering configuration.

[0095] Features h of vehicle node i in set V(t) i For: h i =[Xw,Yw,v x ,v y [TTC, vehicle category class, detection confidence level conf]; Where (Xw, Yw) are the coordinates of vehicle node j in the world coordinate system, (v x , v y ) represents the velocity of vehicle node i in the x and y directions of the world coordinate system; TTC represents the TTC value between vehicle node i and its own vehicle at time frame t; the values ​​of vehicle class and detection confidence conf are derived from the scene parsing results.

[0096] v x and v y Calculated by the difference in BEV coordinates between adjacent frames: Frame t: Vehicle BEV coordinates (Xw(t), Yw(t)); Frame t-1: Vehicle BEV coordinates (Xw(t-1), Yw(t-1)); If the time difference between the two frames is Δt, then: Lateral relative velocity v x =[Xw(t)-Xw(t-1)] / △t; Longitudinal relative velocity v y =[Yw(t)-Yw(t-1)] / △t; v x and v y It is a relative speed. Since the BEV coordinate system takes the vehicle as its origin, the speed obtained by calculating the difference is relative to the vehicle.

[0097] ss62: Based on vehicle node i, find all its neighbor nodes, denoted as vehicle node j, and calculate the graph attention feature vector: α ij =softmax j (LeakyReLU(W a ·[h i ||h j ||f ij ])); In the formula, α ij Let f represent the attention weight of vehicle node i to its neighbor vehicle node j, || be the vector concatenation operation, and f be the attention weight of vehicle node i to its neighbor vehicle node j. ij Represents the interaction features between vehicle node i and vehicle node j; LeakyReLU is the activation function; softmax... j This means normalizing all neighbors j of vehicle node i so that the sum of the attention weights of all neighboring vehicles to vehicle node i is 1; W a This is the attention matrix that is fixed after training. f ij =[△Xw, △Yw, △v x , △v y , d ij TTC ij lane rel ]; Where (△Xw, △Yw) represent the relative positions between vehicle node i and vehicle node j; (△v x , △v y ) represents the relative speed between vehicle node i and vehicle node j; d ij Indicates the relative distance between two vehicles; TTC ij Represents the TTC between vehicle node i and vehicle node j; lane rel Indicates the lane relationship between two vehicles, lane rel ∈{same lane, left adjacent lane, right adjacent lane}.

[0098] ss63: Introduces a gated recurrent unit (GRU) to update the features of node vehicle i; m i =∑ j∈N(i) α ij W m h j ; h i ~ =GRU(h i ,m i ); In the formula, m i The aggregation feature of vehicle node i; W mFor the mapping matrix, this method is based on the Graph Attention Network (GAT) trained and implemented in PyTorch, W. m is the weight parameter of the nn.Linear layer, which is learned to obtain the optimal value through training; GRU is a gated recurrent unit; N(i) is the set of neighboring vehicle nodes of node i.

[0099] ss64: Predicting the driving trajectory of node vehicle i using MLP; τ i =MLP dec (h i ~ ={(△X1,△Y1),…,(△X n ,△Y n )}; In the formula, τ i For the predicted trajectory of vehicle node i, MLP dec For MLP decoder; (△X) n ,△Y n ) represents the coordinate offset of vehicle node i in the BEV coordinate system at the nth future step, starting from time frame t, as output by the MLP decoder. The specific value of n is set according to actual needs; in this embodiment, n is set to 5. The output is the (△X) coordinate offset in the BEV coordinate system for the next 5 steps. n ,△Y n Coordinate offset.

[0100] Simulation results of trajectory prediction (reference) Figure 5 . Figure 5 The parameter in the upper right corner is: T+0.5s / step × 5 steps total 2.5s, which means that the trajectory prediction is based on a step size of 0.5 seconds, predicts a total of 5 steps, and covers the motion trajectory for the next 2.5 seconds. Figure 5 The vehicles in the middle are marked in blue, including the red vehicle in the same lane and the vehicles in different lanes marked in yellow (L2), green (L3), and purple (L4). Figure 5 The concentric circles in the diagram represent the safety zone visualization. Each concentric circle around a vehicle indicates the range of prediction uncertainty or a safety buffer zone, with greater uncertainty towards the outermost circle. The diagram shows the L2 safety zone visualization results for the vehicle in the same lane and the vehicle in the adjacent lane.

[0101] Figure 5 The red dotted line trajectory corresponding to the vehicle in front in the same lane is the predicted trajectory of the vehicle in front in the same lane. Five red dots represent the predicted trajectory for the next five steps, with the red car moving straight forward. The red arrow pointing from the vehicle in front in the through lane to the car in front indicates the direction of collision risk. The red text marking "60m" indicates a distance of 60 meters from the vehicle in front, and "TTC=43s" indicates a collision time of 43 seconds, meaning there is ample safe distance.

[0102] Figure 5The yellow dotted line represents the predicted trajectory of the L2 vehicle. The five yellow dots indicate the predicted trajectory for the next five steps, showing a tendency to change lanes to the left, i.e., moving closer to the vehicle's lane. The yellow text marking "45m, TTC=16s" indicates a distance of 45 meters between the L2 vehicle and the vehicle, with a collision time of 16 seconds, a high risk requiring vigilance. The yellow line pointing from the L2 vehicle to the vehicle indicates that the algorithm predicts the L2 vehicle is performing a lane change maneuver, cutting from the L2 lane into the L1 lane where the vehicle is located. The L2 vehicle is attempting to change lanes to the left; if it continues to change lanes to the left, a collision will occur. The 45m TTC=16s scenario specifically addresses the situation where the L2 vehicle is moving towards the vehicle, indicating a collision will occur if no intervention is taken within 16 seconds.

[0103] ss65: Calculates the TTC of vehicle node i and the vehicle itself based on the predicted driving trajectory; TTC predict =min t {t∣d(τ ego (t),τ i (t)) <D th}; In the formula, TTC predict Let τ be the predicted collision time between vehicle node i and its own vehicle. ego (t) represents the predicted trajectory of the vehicle, D th min is the preset distance threshold. t d() represents the minimum collision time; d() is a function to calculate the straight-line distance between the predicted positions of the two vehicles.

[0104] This method is based on predictive trajectory calculation for forward-looking TTC: instead of waiting for the vehicle to get close before issuing a warning, it predicts the danger that may occur 3 seconds later.

[0105] When changing lanes at high speed, the acceleration / deceleration of adjacent vehicles can affect the safety of your vehicle. Single-vehicle TTC only considers the vehicle in front, while multi-vehicle interaction graphs take into account the mutual influence of all nearby vehicles; for example, if a vehicle accelerates and approaches from the left lane, the graph attention weight will increase, and forward-looking TTC will issue an early warning, which the single-vehicle model cannot see at all.

Claims

1. A highway assisted driving method, characterized in that, It includes the following steps: S1: Obtain the data to be analyzed based on the onboard cameras and sensors installed on the vehicle; The vehicle-mounted cameras involved in the calculation are denoted as: cameras to be analyzed; The image obtained after preprocessing the video frames captured in real time by the camera to be analyzed is denoted as the image to be analyzed. The acquisition time of the image to be analyzed is denoted as time frame t; The data to be analyzed includes: the image to be analyzed, the vehicle speed acquired via the CAN bus, the camera intrinsic parameter matrix K of the camera to be analyzed, and the installation height H of the camera to be analyzed. cam ; S2: Obtain any image to be analyzed, perform multi-task reasoning based on the image to be analyzed, and output the scene analysis results of the highway environment included in the image to be analyzed in real time; the scene analysis results include: vehicle detection results, lane line segmentation results, and drivable area segmentation results; Each vehicle detection result corresponds to a vehicle to be analyzed; S3: Perform depth estimation on the image to be analyzed and output the depth estimation result corresponding to the image to be analyzed; The depth estimation results include: relative depth map D mono and absolute depth map D metric ; S4: Combining the scene analysis results and the depth estimation results, calculate the depth of each vehicle to be analyzed in the image to be analyzed, and output the precise depth D of each vehicle to be analyzed. vehicle ; S5: Combining the depth estimation results and the scene analysis results, calculate the homography matrix H. Based on the homography matrix H, convert the image to be analyzed into a bird's-eye view. Based on the precise depth D of each vehicle... vehicle The bird's-eye view information of the vehicle to be analyzed is calculated; the bird's-eye view information includes: the BEV coordinates of each vehicle and the lane affiliation of each vehicle; Acquire all video frame images captured by the camera to be analyzed; for each vehicle to be analyzed, combine its corresponding bird's-eye view information according to time frames to obtain the vehicle's tracking trajectory; output the vehicle tracking trajectory for each vehicle to be analyzed. S6: Perform time-series fusion of the vehicle tracking trajectory and vehicle speed of the vehicle to be analyzed; Calculate the collision probability between the vehicle and other vehicles to be analyzed based on the distance between the vehicles, and perform uncertainty TTC warning; For vehicles retrieved from the current image to be analyzed, trajectory prediction is performed, and forward-looking TTC is calculated based on the predicted trajectory, and early warning is issued.

2. The highway assisted driving method according to claim 1, characterized in that: Step S2 includes the following steps: s21: Constructing a task detection model based on YOLOPv2; The task detection model is used to perform task inference on the image to be analyzed. The tasks include vehicle detection, lane line detection, and drivable area detection. s22: The image to be analyzed is fed into the task detection model; s23: Construct the scene-aware module; The P3 layer feature map output by the Neck network of the YOLOPv2 model is obtained and sent to the scene perception module; wherein, the P3 layer feature map output by the Neck network is a high-resolution small target feature map; The scene perception module extracts scene features from the P3 layer feature map according to pre-specified scenes that may affect the image acquisition effect: f scene =MLP(GAP(P3 features ))∈R ds ; Among them, f scene P3 is the scene feature vector; features The P3 layer feature map is output by the Neck network of the YOLOPv2 model; GAP is global average pooling; MLP is a multilayer perceptron, which uses two fully connected layers to map to ds dimensions, corresponding to ds kinds of scenes; s24: Add a dynamic task weight analysis module after the scene perception module, and send the P3 layer feature map processed by the scene perception module into the dynamic task weight analysis module. The dynamic task weight analysis module calculates the uncertainty weight λ(t) corresponding to the t-th time frame for each task detected by the task detection model based on the scene characteristics. λ(t)=W T ·c(t); Among them, W T It is a fixed weight matrix (3×ds) after training, with each column corresponding to a scenario for the weight allocation of the three tasks; c(t) represents the scene confidence score, and c(t) = softmax(f scene ); s25: Perform time-series smoothing on the weights of each task to obtain the smoothed uncertain weights; l smooth (t)=JU1·λ(t)+JU2·λ smooth (t-1); In the formula, JU1 and JU2 are preset jump adjustment coefficients; s26: The P3 layer feature map processed by the dynamic task weight analysis module and the P4 and P5 layer feature maps output by the YOLOPv2 model Neck network are fed into the multi-task decoder head; the multi-task decoder head outputs the scene parsing result; the multi-task decoder head includes: a detection head, a lane line segmentation head, and a drivable area segmentation head.

3. The highway assisted driving method according to claim 2, characterized in that: Step S2 also includes the following steps: s27: Calculate the uncertainty-weighted total loss L total ; L total =∑ i (L i / 2σ i 2 )+logσ i ? Among them, L i Let σ represent the loss of the i-th task. i This represents the uncertainty weight of the i-th task.

4. The highway assisted driving method according to claim 1, characterized in that: Step S3 specifically includes the following operations: s31: Based on the monocular depth estimation Monodepth2, the image to be analyzed is parsed to obtain the inverse depth map corresponding to the image to be analyzed; s32: Obtain the relative depth map D corresponding to the image to be analyzed based on the inverse depth map. mono ; s33: Acquire each lane line and measure the pixel width d between two lane lines in the image to be analyzed. pixel ; s34: For each lane line, obtain the inverse depth value (disparity) of the lane line marking pixels corresponding to each lane line in the image to be analyzed based on the inverse depth map. lane Based on the inverse depth value disparity lane The relative depth corresponding to the lane marking pixels is calculated, and then the average relative depth D for each lane marking is obtained by averaging. lane_avg ; s35: The principal dimensions of the lane lines are calculated; s1 ~ =(W lane ·f y ) / (d pixel ·D lane_avg ); In the formula, s1 ~ W is the lane line scale factor. lane The standard width of a highway lane is d (as per national standards). pixel D represents the pixel width between the two lane lines in the image to be analyzed. lane_avg f is the average relative depth of the lane line area. y The vertical focal length of the camera to be analyzed; s36: Calculate the auxiliary scale for the horizon; s2 ~ =H cam / (f y ·tan(arctane((v horizon -c y ) / f y ) ) ); In the formula, s2 ~ v is the horizon scale factor; horizon c represents the vertical pixel coordinates of the horizon in the image to be analyzed. y H represents the ordinate of the principal point in the image from the camera to be analyzed. cam The installation height of the camera to be analyzed; ; In the formula, To calculate the partial derivative of depth D with respect to the vertical pixel coordinate v of a pixel, we take the gradient of depth D along the vertical direction v; argmax represents the pixel position where the maximum value is obtained. s37: Perform Bayesian scale fusion calculation on the main scale of lane lines and the auxiliary scale of the horizon to obtain the mean scale μ after fusion. fused and the scale variance σ after fusion fused ; m fused =(μ1 / σ1 2 +μ2 / σ2 2 ) / (1 / σ1 2 +1 / σ2 2 ); s fused 2 =1 / (1 / σ1 2 +1 / σ2 2 ); In the formula, μ1 is the lane line scale factor s1 ~ The mean value, μ2 is the horizon scale factor s2. ~ The mean; σ1 is the lane line scale factor s1 ~ The variance, σ² is the horizon scale factor s². ~ The variance; s38: Calculate the absolute depth and depth confidence; D metric =m fused ·D mono ; conf scale =1- σ fused / m fused ; In the formula, D mono D is the relative depth map corresponding to the image to be analyzed. metric This is the absolute depth map corresponding to the image to be analyzed; conf scale D metric The corresponding depth confidence level.

5. The highway assisted driving method according to claim 1, characterized in that: In step S4, Includes the following operations: s41: Obtain the vehicle detection result in the scene parsing result, and directly regard all pixels in the vehicle detection box in the vehicle detection result as the vehicle region of the vehicle to be analyzed, and obtain the vehicle segmentation mask corresponding to the vehicle region; the image in the vehicle region is the vehicle image of the vehicle to be analyzed. s42: Calculate the mask gradient of the vehicle segmentation mask using the Sobel operator: G mask 2 =(G x 2 +G y 2 ); In the formula, G mask G represents the magnitude of the mask gradient. x For the horizontal Sobel gradient, G y The vertical Sobel gradient; G x =Sobel x (mask),G y =Sobel y (mask); s43: For each vehicle to be analyzed, calculate the core mask of the vehicle segmentation mask. core ; mask core ={(u,v)∈mask|G mask <0.1∩erode(mask,3)=1}; In the formula, (u,v) are the image pixel coordinates in the vehicle detection box, and erode(mask,3) indicates that the mask is eroded using a 3×3 structuring element; s44: Calculate the gradient weights of pixels in the vehicle image of the vehicle to be analyzed; w(u,v)=exp(-β·|▽D(u,v)|); In the formula, w(u,v) is the depth weight corresponding to pixel (u,v), β is the gradient weight adjustment coefficient; ▽D(u,v) represents the gradient of pixel (u,v) in the absolute depth map Dmetric; s45: Divide the vehicle detection bounding box in the vehicle detection result into three equal regions in the vertical direction: top, mid, and bottom. The height of each region is 1 / 3 of the height of the vehicle detection bounding box. Calculate the depth D corresponding to the vehicle image in the k-th region. k ; D k =(∑w(u,v)D(u,v) ) / (∑w(u,v)); In the formula, k∈{top,mid,bot}; s46: Calculate the depth corresponding to the vehicle image of each vehicle to be analyzed; D vehicle =d top ·D top + d mid ·D mid + d bot ·D bot ; Among them, D vehicle For the precise depth of each vehicle; d top d mid and d bot Adjust the depth weights for the three regions.

6. The highway assisted driving method according to claim 1, characterized in that: Step S5 includes the following detailed operations: s51: Calculate the theoretical road surface depth corresponding to the image to be analyzed; D theory (v)=H cam / sin(arctane((vc y ) / f y )+θ pitch ); In the formula, D theory (v) represents the theoretical road surface depth corresponding to the v-th row pixel of the image to be analyzed at angle θ; c y For the y-value of the principal point coordinate in the camera image to be analyzed, f y To determine the vertical focal length of the camera to be analyzed, θ pitch H is the preset initial value of the camera's pitch angle to be analyzed; cam The installation height of the camera to be analyzed; s52: The pitch angle of the camera to be analyzed is fitted based on the road surface depth; θ pitch (t) '=argmin θ Σ v [D road (v) - D theory (v) ] 2 ; In the formula, θ pitch (t)' represents the estimated pitch angle of the camera to be analyzed corresponding to time frame t, D road (v) represents the actual road surface depth corresponding to the v-th row pixel at angle θ; argmin θ : The pitch angle that minimizes the error; s53: Divide the road region in the image to be analyzed into left and right sides, and estimate the roll angle by comparing the road surface depth of the left and right roads in the image to be analyzed; θ roll (t) ’=arctan((D left - D right ) / ( α roll ·W / 2)); In the formula, θ roll (t) represents the estimated roll angle of the camera to be analyzed; D left D represents the average actual road surface depth of the road on the left. right α represents the average actual road surface depth of the road on the right; W is the image width; α roll This is the roll angle scaling factor; α roll = H cam / f x ;f x The horizontal focal length of the camera to be analyzed; s54: Perform Kalman filtering on the angle estimate; i filt (t)=Kalman update (i filt (t-1),θ(t)' ); In the formula, Kalman update () represents the Kalman filter, θ filt (t) represents the filtered angle corresponding to time frame t; θ(t)' represents the pitch angle estimate or roll angle estimate corresponding to time frame t; s55: For each frame of the image to be analyzed, the rotation matrix R is recalculated based on the real-time angle, and then the homography matrix H is synthesized. R(t)=R z (i yaw )R x (i pitch ')R y (i roll '); H(t’)=K·[R(t) [:,0:2] ∣t’]; In the formula, R(t) is the rotation matrix corresponding to time frame t, R z Rotate the camera coordinate system around the z-axis by an angle θ yaw R x Rotate the camera coordinate system by an angle θ around the x-axis pitch ', R y Rotate the camera coordinate system around the y-axis by an angle θ roll ';θ yaw yaw angle of camera, collected from vehicle navigation system; [:,0:2] indicates taking the first two columns of rotation matrix, t' represents translation vector, and | represents matrix concatenation operation; Translation vector t'=[0,0,H cam ] T ; s56: Based on the homography matrix H(t'), the pixel coordinates of the image to be analyzed are transformed into the BEV coordinate system of the ground BEV bird's-eye view to obtain the bird's-eye view to be analyzed, and the vehicle tracking trajectory of the vehicle to be analyzed is obtained based on the bird's-eye view to be analyzed.

7. The highway assisted driving method according to claim 6, characterized in that: The method for determining the vehicle tracking trajectory of the vehicle to be analyzed includes the following operations: a1: Based on the BEV coordinate system in the bird's-eye view to be analyzed, utilize the precise depth D of the vehicle to be analyzed. vehicle The BEV coordinates (Xw, Yw) corresponding to the vehicle to be processed are calculated. a2: Obtain the lane line segmentation result from the scene parsing result corresponding to the image to be analyzed, and use the lane line pixel mask to obtain the pixel coordinates (u) of the lane line. lane ,v lane Based on the homography matrix H(t'), the lane line pixel coordinates (u) are... lane ,v lane Convert the coordinates to BEV coordinates to obtain the lane line BEV coordinates; a3: Based on the BEV coordinates (Xw, Yw) of the vehicle to be processed, determine which set of lane lines the vehicle is in, and then obtain the lane affiliation of the vehicle to be analyzed corresponding to time frame t; a4: BEV coordinates and lane affiliation of the vehicle to be analyzed, stored as the vehicle bird's-eye view information [BEV coordinates, lane affiliation] corresponding to time frame t; a5: Sort the vehicle bird's-eye view information of the vehicle to be analyzed according to time frames to obtain the vehicle tracking trajectory corresponding to the vehicle to be analyzed.

8. The highway assisted driving method according to claim 1, characterized in that: In step S6, the uncertainty TTC early warning process specifically includes the following steps: s61: Modeling the distance uncertainty between the analyzed vehicle and the self-vehicle; The distance variable d between the two vehicles follows a mean of μ d The variance is σ d 2 Gaussian distribution: d ~ N(μ d ,σ d 2 ); s d = m d ·s fused / m fused ; In the formula, μ fused The mean scale is the fused value of the lane line main scale and the horizon auxiliary scale; σ fused The scale variance after fusing the lane line main scale and the horizon auxiliary scale; The relative velocities of the two vehicles along the direction of collision follow a mean of μ. v The variance is σ v 2 Gaussian distribution: v rel ~N(μ v ,s v 2 ); In the formula, v rel ‎ is the relative velocity of the two vehicles along the direction of collision; μ d σ is the mean distance. d 2 μ represents the distance variance. v σ represents the mean relative velocity. v 2 The variance of relative velocity; s62: Calculate the mean and variance of TTC; m TTC ≈μ d / m v ; s TTC ≈μ TTC· sqrt((σ d / m d ) 2 +(s v / m v ) 2 ); In the formula, μ TTC σ is the average collision time. TTC The variance of the collision time is sqrt(), which is the square root. s63: Calculate the collision probability between the vehicle to be analyzed and the vehicle itself; P(TTC<τ)=Φ((τ-μ TTC ) / s TTC ); In the formula, Φ is the Gaussian cumulative distribution function; τ is the preset collision time threshold; P(TTC<τ) represents the probability that the collision time TTC is less than τ; s64: Set warning levels and set different warning conditions for each warning level; Based on the value of P(TTC<τ), when P(TTC<τ) meets the preset warning conditions, different levels of TTC warnings are issued.

9. The highway assisted driving method according to claim 1, characterized in that: Step S6, the process of calculating the forward TTC based on the predicted trajectory and providing early warning, includes the following operations: ss61: For the image to be analyzed, construct a dynamic graph attention network G(t)=(V(t),E(t)); Where V(t) represents the set of all detected vehicle nodes included in the image to be processed; E(t) is the set of edges, whose elements are edges e. ij It holds true if and only if the following conditions are met: |Y j -Y i | <IN th ∩lane rel ∈{same lane, left adjacent lane, right adjacent lane}; Among them, Y i and Y j This represents the Y-coordinate values ​​of vehicle node i and vehicle node j in the world coordinate system; lane rel Indicates the lane relationship between vehicle node i and vehicle node j; IN th The preset interaction impact threshold; Features h of vehicle node i in set V(t) i For: h i =[Xw,Yw,v x ,v y [TTC, vehicle category class, detection confidence level conf]; Where (Xw, Yw) are the coordinates of vehicle node j in the world coordinate system, (v x , v y ) represents the velocity of vehicle node i in the x and y directions of the world coordinate system; TTC represents the TTC value between vehicle node i and its own vehicle at time frame t; the values ​​of vehicle class and detection confidence conf are derived from the scene parsing results; ss62: Based on vehicle node i, find all its neighbor nodes, denoted as vehicle node j, and calculate the graph attention feature vector: α ij =softmax j (LeakyReLU(W a ·[h i ||h j ||f ij ])); In the formula, α ij Let f represent the attention weight of vehicle node i to its neighbor vehicle node j, || be the vector concatenation operation, and f be the attention weight of vehicle node i to its neighbor vehicle node j. ij Represents the interaction features between vehicle node i and vehicle node j; LeakyReLU is the activation function; softmax... j This means normalizing all neighbors j of vehicle node i so that the sum of the attention weights of all neighboring vehicles to vehicle node i is 1; W a This is the attention matrix that is fixed after training. f ij =[△Xw, △Yw, △v x , △v y , d ij , TTC ij , lane rel ]; Where (△Xw, △Yw) represent the relative positions between vehicle node i and vehicle node j; (△v x , △v y ) represents the relative speed between vehicle node i and vehicle node j; d ij Indicates the relative distance between two vehicles; TTC ij The lane represents the TTC between vehicle node i and vehicle node j. rel Indicates the lane relationship between the two vehicles; ss63: Introduces a gated recurrent unit (GRU) to update the features of node vehicle i; m i =∑ j∈N(i) a ij W m h j ; h i ~ =GRU(h i ,m i ); In the formula, m i The aggregation feature of vehicle node i; W m The mapping matrix is ​​trained to obtain the optimal value; GRU is a gated recurrent unit; N(i) is the set of neighboring vehicle nodes of node i. ss64: Predicting the driving trajectory of node vehicle i using MLP; τ i =MLP dec (h i ~ ) ={(△X1,△Y1),…,(△X n ,△Y n )}; In the formula, τ i For the predicted trajectory of vehicle node i, MLP dec For MLP decoder; (△X) n ,△Y n ) represents the coordinate offset of vehicle node i in the BEV coordinate system at the nth future step, starting from time frame t, as output by the MLP decoder; ss65: Calculates the TTC of vehicle node i and the vehicle itself based on the predicted driving trajectory; TTC predict =min t {t∣d(τ ego (t),τ i (t)) <D th }; In the formula, TTC predict Let τ be the predicted collision time between vehicle node i and its own vehicle. ego (t) represents the predicted trajectory of the vehicle, D th min is the preset distance threshold. t d() represents the minimum collision time; d() is a function to calculate the straight-line distance between the predicted positions of the two vehicles.

Images