Automobile auxiliary driving monocular distance measurement method and device based on lightweight occupancy prediction network

Abstract

The invention discloses an automobile auxiliary driving monocular distance measurement method and device based on a lightweight occupation prediction network, and the method comprises the steps: firstly constructing a 3D occupation prediction data set, and then constructing a monocular occupation prediction teacher model based on multi-frame time sequence fusion and time sequence stereo depth estimation, and a lightweight monocular occupation prediction student model based on single-frame input. And training the occupation prediction teacher model to obtain the occupation prediction teacher model with good performance, and evaluating the performance of the teacher model. And performing distillation training of the student model by using the teacher network intermediate layer features and output to obtain a performance-improved lightweight occupation prediction student model, and evaluating the performance of the student model. And finally, performing forward reasoning on the student model, and performing connected domain analysis on an interested instance (such as a vehicle and a pedestrian) from a reasoning result to complete monocular distance measurement. The method provided by the invention overcomes the problem of inaccurate distance measurement caused by unstable detection frame in the traditional monocular distance measurement method, and improves the performance while lightening the model by using the distillation technology.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, specifically to the field of autonomous driving, and in particular to a monocular ranging method and device for automobile assisted driving based on a lightweight occupancy prediction network (Occupancy Network). Background Art

[0002] The progress of autonomous driving technology means a profound change in the field of transportation. In recent years, with the rapid development of artificial intelligence in the computer field, especially deep learning and sensor technology, autonomous driving systems have brought more efficient, safer and more environmentally friendly prospects to transportation through real-time perception and intelligent decision-making. The Advanced Driving Assistance System (ADAS) currently carried by cars on the market has the following main functions: front / rear vehicle collision warning, vehicle distance detection warning, blind spot detection, lane change assistance, adaptive cruise control, etc. These functions are inseparable from the perception of the 3D space around the car. The bird's-eye view (BEV) perspective is often used to perceive the 3D space: the information obtained by sensors installed at different positions of the vehicle is uniformly converted to a bird's-eye view from above the vehicle after feature extraction, as a unified view for subsequent modules (3D target detection, path planning, etc.). The disadvantage of BEV is that it compresses the height dimension in the entire three-dimensional space. Although this dimension has less information content in autonomous driving, it still contains some information. Therefore, in BEV mode, height or depth information cannot be perceived. The occupancy prediction network technology proposed in 2022 solves the problem of high spatial missing of BEV. The basic idea is to divide the 3D space around the car into voxels, and model the 3D environment around the car by predicting the semantic information of each voxel.

[0003] Distance information plays an important role in the application of intelligent driving. Compared with the use of expensive radar equipment, the pure visual ranging solution using cameras is widely used due to its relatively low price. Pure visual ranging solutions include binocular ranging and monocular ranging. Binocular ranging uses the parallax of the left and right views to estimate depth. Its advantage is that the depth estimation accuracy is high. Its disadvantage is that the binocular correction and matching process brings a huge amount of calculation, and the two cameras need to be strictly calibrated and registered. The effect of calibration and registration directly affects the accuracy of ranging. Traditional monocular ranging mainly combines the target detection rectangular frame to perform ranging tasks. The distance is estimated by the width of the rectangular frame of the target in the image and the camera matrix. Its advantages are small amount of calculation, fast speed, high cost performance, and relatively simple system structure. Its obvious disadvantage is that it needs to maintain a sample database for different objects and the distance measurement accuracy is low. The emergence of occupancy prediction network provides a new idea for ranging. It divides the 3D space into voxels of preset size, making the distance calculation extremely simple. At the same time, its predicted voxel occupancy information avoids explicitly predicting the object category: when encountering an object that has never appeared in the data set, the occupancy prediction method predicts whether the current voxel is occupied, avoiding the failure to identify abnormal objects and causing crashes. However, the current occupancy prediction network in the field of autonomous driving often contains complex depth estimation, context prediction, or Transformer-based 2D to 3D conversion modules, which require high computing power and limit its deployment on the vehicle chip side.

[0004] Knowledge distillation is a lightweight compression method for deep learning models. It aims to transfer the knowledge of a large model (teacher model) to a small model (student model) to improve the performance and generalization ability of the small model. The core idea of ​​knowledge distillation is to transform the knowledge of a complex model into a more concise and effective representation, so that it can reduce computational complexity and resource requirements while maintaining high performance. In order to address the problem that the occupancy prediction network has high computing power requirements and is difficult to deploy, knowledge distillation technology can be used: use a pre-trained teacher model with good performance but high computing power requirements to guide the learning of a student model with poor performance but low computing power requirements, while improving its performance without increasing additional computing power requirements. Summary of the invention

[0005] In view of the shortcomings of the existing traditional monocular ranging technology, the present invention proposes a monocular ranging method and device for automobile assisted driving based on a lightweight occupancy prediction network. The specific technical solution is as follows:

[0006] A monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network, the method specifically comprises the following steps:

[0007] S1: Constructing a 3D occupancy prediction dataset.

[0008] S2: Construct a monocular occupancy prediction teacher model based on multi-frame temporal fusion and temporal stereo depth estimation.

[0009] S3: Build a lightweight monocular occupancy prediction student model based on single-frame input.

[0010] S4: Train the occupancy prediction teacher model to obtain an occupancy prediction teacher model with good performance, and evaluate the performance of the teacher model.

[0011] S5: Use the intermediate layer features and outputs of the teacher network to perform distillation training on the student model to obtain a lightweight occupancy prediction student model with improved performance, and evaluate the performance of the student model.

[0012] S6: Perform forward reasoning on the student model and complete monocular ranging based on the reasoning results.

[0013] Furthermore, the step S1 is specifically implemented by the following steps:

[0014] S1.1: Collect sensor data: Use a collection vehicle equipped with multiple sensors (lidar, camera) to collect raw data on the road environment. This data should include point clouds, image data, etc. with multiple time stamps.

[0015] S1.2: Annotated point cloud data: In the collected data, the semantics of the 3D point cloud of the keyframes are manually annotated. The values ​​in the label are encoded as integers starting from 0, and each integer corresponds to a category.

[0016] S1.3: Generate ground-truth labels for occupancy prediction: Voxelize the annotated point cloud data to generate accurate 3D ground-truth labels.

[0017] Furthermore, the step S1.3 is specifically implemented by the following steps:

[0018] S1.3.1: Aggregate multi-frame point clouds to generate dense point cloud data.

[0019] S1.3.2: Use the K-nearest neighbor algorithm (KNN) to assign semantic labels to unlabeled frames. For each point in the unlabeled frame, find its nearest K labeled frames and assign a semantic label to the point based on the majority voting principle.

[0020] S1.3.3: For the aggregated dense point cloud, the VDBFusion method is used for mesh reconstruction.

[0021] S1.3.4: Densely sample the reconstructed grid and continue to use KNN to semantically label the sampled points to obtain accurate 3D truth values.

[0022] S1.4: Divide the set into training, validation and test sets.

[0023] S1.5: Save the data information in a format that is easy to read, such as .pkl format, to facilitate reading during training and verification.

[0024] The pkl file contains:

[0025] (1) Data identification: including the tokens of the current frame and its previous and next frames, whether it is a key frame, and the scene token where the frame data is located.

[0026] (2) Transformation matrix: including the external parameter matrix of the lidar, the internal and external parameter matrix of the camera, the transformation matrix from the camera coordinate system to the vehicle coordinate system, and the transformation matrix from the vehicle coordinate system to the global coordinate system.

[0027] (3) Storage path: including the point cloud data storage path, the camera image storage path, the 2D semantic pseudo-label storage path, and the occupancy (voxel) true value storage path.

[0028] Furthermore, the vehicle coordinate system, global coordinate system, 2D semantic pseudo-label, occupancy true value, etc. in step S1.5 are specifically as follows:

[0029] S1.5.1: The ego vehicle coordinate system is a local coordinate system with the vehicle (ego vehicle) as the reference point, used to describe sensor data and the position of objects relative to the vehicle. The origin is located at the center of the ego vehicle, that is, the geometric center of the vehicle chassis. The X axis points to the front of the vehicle. The Y axis points to the left of the vehicle. The Z axis points to the top of the vehicle.

[0030] S1.5.2: The global coordinate system is a fixed world coordinate system used to describe the absolute position of objects in the entire environment. It does not change as the vehicle moves. The origin is located at the point where the longitude and latitude in the GPS are both 0°. The X axis points to the east. The Y axis points to the north. The Z axis points to the sky.

[0031] S1.5.3: 2D semantic pseudo-labels are obtained by projecting the semantically annotated 3D point cloud in S1.2 onto a 2D plane and then performing morphological processing and smoothing operations.

[0032] S1.5.4: The ground truth of occupancy (voxel) contains all scene keyframes, and the shape is (D x ,D y ,D z ), indicating that the size is D x ×D y ×D zThe predicted semantic class index of each voxel in the three-dimensional space of D. Referring to step S1.3, compared with the semantic index of the point cloud in S1.2, an air (free) class is added, which represents the voxels that do not contain any objects in the three-dimensional space. x , D y , D z They are the number of voxels in the front-back, left-right, and up-down directions.

[0033] Furthermore, step S2 is specifically implemented by the following steps:

[0034] S2.1: Set the input camera, input image size, image enhancement matrix, BEV spatial grid size, depth prediction range, multi-frame input index, BEV feature enhancement matrix, optimizer, and learning rate.

[0035] S2.2: Construct 2D feature extraction module: Use ResNet50 network as the backbone network to extract multi-scale features. Send features at different levels into FPN (feature pyramid network) to integrate features at different scales.

[0036] S2.3: Construct a 2D to 3D feature conversion module: Depth estimation predicts depth and context features simultaneously. When predicting depth, monocular depth prediction and multi-view depth prediction based on MVS (temporal multi-view stereo) are combined. For multi-view depth prediction, first predict the depth center (μ) and depth range (σ), then use μ and σ to generate the depth distribution, and finally combine the monocular depth prediction and the weighted multi-view depth to obtain the final depth. Use the Bev Pooling operation to obtain BEV features. For the continuous n frames of time-series input, extract their respective BEV features, and uniformly project the BEV features of non-current key frames to the BEV coordinate system of the current key frame, and fuse the BEV features of multiple frames through the splicing operation along the channel dimension.

[0037] Furthermore, the Bev Pooling operation in step S2.3 projects the 2D features into the BEV space according to the camera intrinsic and extrinsic parameters and the pre-calculated frustum pseudo point cloud, which is specifically implemented by the following sub-steps:

[0038] S2.3.1: Based on the preset depth distribution and image size, construct a frustum pseudo point cloud, which means the pixel coordinates and possible depth distribution of each pixel in the image.

[0039] S2.3.2: Subtract the image enhancement translation matrix from the pixel coordinates of the pseudo point cloud and multiply it by the inverse of the image enhancement rotation matrix to compensate for the impact of the image enhancement operation. Multiply the compensated enhanced pixel coordinates by the inverse matrix of the camera to the ego vehicle and the camera internal parameters to convert them to the ego vehicle coordinates. Multiply the ego vehicle coordinates by the BEV feature enhancement matrix to synchronize the data enhancement in the BEV space. Subtract the three minimum values ​​of the preset three-dimensional direction of the BEV space from the ego vehicle coordinates and divide them by the length of each voxel to obtain the voxel coordinates of the frustum pseudo point cloud in the BEV space, exclude the frustum points that fall outside the preset range, and generate the index of the frustum pseudo point cloud in the BEV space.

[0040] S2.3.3: Filter the context features and depth value indexes that need to be converted according to the index value, sort the frustum pseudo point cloud according to the voxel index, and make the pseudo point clouds belonging to the same voxel adjacent. Use multi-threaded parallel processing, and for multiple pseudo point clouds falling into the same voxel, perform weighted summation of features according to the predicted depth, and finally obtain the BEV feature.

[0041] S2.4: Construct BEV feature processing module: ResNet50 network is used as the backbone network to send features at different levels into FPN (feature pyramid network) to integrate features of different scales.

[0042] S2.5: Construct occupancy prediction head. First, the input feature map is input into a convolutional layer for processing, and then a channel to height operation is performed to decode the height direction voxels for all categories from the channel dimension of the feature map.

[0043] Furthermore, the channel to height operation in step S2.5 is specifically implemented through the following sub-steps:

[0044] S2.5.1: For input channels C f The feature map is output after the full connection layer + activation function + full connection layer, and the number of channels is D z ×C. Where D z and C are the number of voxels in the height direction and the total number of voxel categories in S1.5.4, respectively.

[0045] S2.5.2: Split the output channel into a size of (D x ,D y ,D z ,C) feature map, which represents the model’s prediction scores for all voxels in the three-dimensional voxel space in C categories.

[0046] Furthermore, step S3 is specifically implemented by the following steps:

[0047] S3.1: Set the input camera, input image size, image enhancement matrix, BEV spatial grid size, depth prediction range, BEV feature enhancement matrix, optimizer, and learning rate.

[0048] S3.2: Construct an improved 2D feature extraction module: Use the ResNet50 network as the backbone network to extract multi-scale features. Send features at different levels to the FPN (Feature Pyramid Network) to integrate features at different scales. The output features of the FPN layer are further sent to two residual blocks and then upsampled, supervised by 2D semantic pseudo-labels. The features of the FPN layer and the features after passing through the two residual blocks are concatenated along the channel dimension and passed through a convolutional layer to enhance the 2D features.

[0049] S3.3: Construct a 2D to 3D feature conversion module: The features obtained in S3.2 are further fed into a convolutional layer to output the coupled features of the context features and the depth estimation. The context features and the depth estimation are separated from the coupled features. The same Bev Pooling operation as in S2.3 is used to obtain the BEV features.

[0050] S3.4: Construct an improved BEV feature processing module, with the ResNet50 network as the backbone network. A spatial attention (SA) module is added after each residual block of the ResNet50 network to spatially enhance the feature extraction of the BEV feature map. The calculation formula of spatial attention is as follows:

[0051] M spatial =σ(Conv 7×7 ([AvgPool(F);MaxPool(F)])) (3.1)

[0052] F out =F×M spatial (3.2)

[0053] Where F is the input feature map, AvgPool(·) and MaxPool(·) are the average pooling and maximum pooling operations along the channel dimension respectively; Conv 7×7 is a 7×7 convolution with 2 input channels and 1 output channel; σ is the Sigmoid function. spatial is the generated spatial attention map. Output feature map F out From the input feature map F and the spatial attention map M spatial Multiply them together to get .

[0054] S3.5: Construct occupancy prediction head. The input feature map is first processed by a convolutional layer, followed by a channel to height operation, which decodes the height-direction voxel predictions for all categories from the channel dimension of the feature map.

[0055] Furthermore, step S4 is specifically implemented by the following steps:

[0056] S4.1: Read multiple front camera images input in time series from the training set at once, the conversion matrix from the camera coordinate system to the vehicle coordinate system, the conversion matrix from the vehicle coordinate system to the global coordinate system, the camera intrinsic parameters, the rotation and translation matrices for image enhancement, and the enhancement matrix in the BEV space, which are used as data to be fed into the teacher model during training.

[0057] S4.2: Execute the forward process during training, returning the predicted value of the occupancy network and the depth predicted value predicted by the 2D to 3D feature conversion module.

[0058] S4.3: Read the true occupancy of the voxel according to the storage path of the true value of the voxel contained in the pkl file in S1.5.

[0059] S4.4: According to the storage path of the point cloud data contained in the pkl file in S1.5, the point cloud data is read, and the true value of the depth is generated through the point cloud data to supervise the depth value predicted by the 2D to 3D feature conversion module.

[0060] Furthermore, step S4.4 is implemented by the following sub-steps:

[0061] S4.4.1: Read the three-dimensional coordinates (x, y, z) of the point cloud data from the pkl file.

[0062] S4.4.2: Use the transformation matrix from the vehicle coordinate system to the camera coordinate system to convert the read point cloud data into the camera coordinate system.

[0063] S4.4.3: The point cloud data converted to the camera coordinate system is further projected onto the image plane: Through the camera intrinsic parameter matrix, the points in the three-dimensional space are projected onto the pixel coordinates (u, v) on the two-dimensional image plane while maintaining the depth information d (i.e., distance).

[0064] S4.4.4: Generate depth map: For each point projected onto the image plane, record its corresponding depth value. Fill in the corresponding depth value d at the pixel coordinate (u, v) to generate a depth map. Select the point closest to the camera as the depth value of the pixel in the depth map.

[0065] S4.5: Create a mask to exclude ground-truth occupancy values ​​outside the field of view of the front camera.

[0066] S4.6: Calculate the binary cross entropy loss between the predicted depth and the true depth:

[0067]

[0068] where d and They are the true depth label and the predicted depth respectively.

[0069] S4.7: Calculate the cross entropy loss between the predicted occupancy and the true occupancy:

[0070]

[0071] C is the total number of voxel categories in S1.5.4. i It is the one-hot true label of the i-th class obtained from the true voxel occupancy value in S4.3. is the probability of the i-th class predicted by the model. When calculating the loss, the mask is used to exclude the loss outside the field of view of the front camera.

[0072] S4.8: Final loss L total for:

[0073] L total =λ1L depth +λ2L occ (4.3)

[0074] Where λ1 and λ2 are the weights of depth loss and occupancy loss. The final loss is returned and the back-propagation process is performed to update the gradient.

[0075] S4.9: Put the trained teacher model in the evaluation (eval) mode, perform forward reasoning, calculate the mIoU (mean intersection over union) value between the reasoning result and the true value of the occupancy prediction, and evaluate the performance of the teacher model through the mIoU value. The calculation of the mIoU value is as follows:

[0076] For each category c, calculate its IoU (intersection over union) value. IoU is defined as the intersection of the predicted area and the true area divided by the union of the predicted area and the true area. The formula is as follows:

[0077]

[0078] Where: TP(c) is the true positive of category c. FP(c) is the false positive of category c. FN(c) is the false negative of category c.

[0079] mIoU is the average IoU of all categories. The formula is as follows:

[0080]

[0081] Where C is the number of categories.

[0082] Furthermore, step S5 is specifically implemented by the following steps:

[0083] S5.1: Read and fix the weights of the trained teacher model, and put the teacher model in evaluation (eval) mode and the student model in training (train) mode.

[0084] S5.2: In one training cycle, read three adjacent frames of front camera images from the training set at a time and feed them into the teacher model to perform forward reasoning to obtain the feature map T of the last convolutional layer in the occupancy prediction head. feat and the output of the occupancy prediction head T out .

[0085] S5.3: In a training cycle, the image of the current frame is sent to the student model to perform the forward process of training, and the feature map S of the last convolutional layer in the occupancy prediction head is obtained. feat and the output S of the occupancy prediction head out .

[0086] S5.4: For S feat After one MLP layer and T feat Calculate L2 loss:

[0087]

[0088] The MLP layer consists of 1×1 convolution + ReLU + 1×1 convolution, and N is the batch size during training. When calculating the L2 loss, a mask is used to calculate only the valid voxels within the field of view of the front camera.

[0089] S5.5: From T out and S out The occupancy prediction in the height direction is decoded, and the prediction scores y of the teacher model and the student model for all voxels in the three-dimensional voxel space for all categories of objects are obtained. T and S .

[0090] S5.6: For the predictions of the teacher model and the student model in a training batch, use the mask to find the valid voxels within the field of view of the front camera, and calculate the KL divergence of the prediction scores of the voxel space corresponding to all categories of the teacher model and the student model as the distillation loss. The distillation loss is calculated as follows:

[0091]

[0092] It is used to transform the eigenvalues ​​into a probability distribution as shown below:

[0093]

[0094] y T and S are the outputs of the teacher model and the student model, respectively, representing the prediction scores of the valid voxels for C categories in the three-dimensional voxel space of size W·H·Z. T is the distillation temperature, which is used to smooth the prediction distribution of the student model. The larger T is, the more attention is paid to a wider area in the three-dimensional space.

[0095] During the entire training process, the distillation temperature changes according to the following formula, following the distillation strategy of high temperature first and then low temperature. The temperature T in each round of training is calculated according to the following formula:

[0096]

[0097] Where T0 is the starting temperature, T end is the end temperature, E total is the total number of training rounds, and e is the current number of training rounds.

[0098] S5.7: The predicted categories of voxels in S1.5.4 are divided into three categories: foreground objects, background objects, and air according to actual needs. Foreground objects can include vehicles and pedestrians on the road, and other objects of focus; background objects can include trees, roads, buildings, etc. Different weights are given to these three types of objects when calculating the distillation loss.

[0099] S5.8: The total losses during the distillation process are as follows:

[0100] L totat =λ seg L seg +λ occ L occ +λ feat L feat +λ cw3d L cw3d (5.5)

[0101] Where L seg is the 2D semantic loss of the student model itself, that is, the cross entropy loss function between the 2D semantic map and the 2D semantic pseudo-label:

[0102]

[0103] in is the model's true category y at pixel (i, j) i,j The predicted probability of . H and W are the height and width of the feature map respectively.

[0104] L occ is the occupancy prediction loss of the student model itself, which is calculated the same as formula (4.2) in S4.7.

[0105] L featis the L2 distillation loss of the feature map.

[0106] L cw3d is the distillation loss of the final output.

[0107] λ seg , occ , feat and λ cw3d are the weights of the four. For the distillation loss, set the cosine annealing learning rate strategy. Return the final loss and perform the back propagation process to update the gradient.

[0108] S5.9: Put the student model after distillation training in the evaluation (eval) mode, perform forward reasoning, calculate the mIoU (mean intersection over union) value between the reasoning result and the true value of the occupancy prediction, and evaluate the performance of the student model by the mIoU value. The calculation of the mIoU value is the same as formula (4.5) in S4.9.

[0109] Furthermore, step S6 is specifically implemented by the following steps:

[0110] S6.1: The vehicle-mounted forward-view camera image, the conversion matrix from the camera coordinate system to the vehicle coordinate system, the conversion matrix from the vehicle coordinate system to the global coordinate system, and the camera internal parameters are input into the student model. The student model is in evaluation (eval) mode and performs forward reasoning to obtain a prediction of shape (Dx, Dy, Dz).

[0111] S6.2: Initialize the data structure, including: an array of the same shape as the prediction: labeled array, used to store the connected region identifier. A Boolean array: visited, used to record the visited voxels. A list: region props, used to store the properties of each connected region, such as the number of voxels, category, the voxel closest to the ego vehicle, and the distance.

[0112] S6.3: Use the breadth-first search strategy to find the connected domain of each object of interest and the voxel coordinates closest to the ego vehicle, and record the connected domains that meet the requirements. For each connected domain, traverse all its voxels and calculate the Euclidean distance from each voxel to the ego vehicle.

[0113] Furthermore, step S6.3 is implemented by the following sub-steps:

[0114] S6.3.1: Traverse all voxels, and the voxels that meet the requirements meet the following conditions: 1. The voxel is within the field of view of the front camera. 2. The voxel belongs to the category of interest. 3. The voxel has not been visited.

[0115] S6.3.2: For voxels that meet the requirements, initialize the breadth-first search queue and add the current voxel coordinates to the queue, and mark it as visited. Initialize the voxel counter of the current connected domain to 1, and record the current voxel as the voxel closest to the ego vehicle, and the distance from the current voxel to the ego vehicle is the minimum distance.

[0116] S6.3.3: Perform a breadth-first search until the queue is empty. Each time a voxel coordinate is taken from the queue, check whether its 6 adjacent voxels meet the requirements of S6.3.1. For voxels that meet the requirements, mark them as visited and add them to the queue. At the same time, update the voxel counter of the connected domain and calculate its distance to the ego vehicle. If it is less than the previously calculated result, update the minimum distance.

[0117] S6.3.4: After the search is completed, if the voxel counter of the current connected domain is greater than 1, the connected domain is recorded in the labeled array, and the properties of the connected domain, including the number of voxels, category, the voxel closest to the vehicle, and the distance, are recorded in the region props, and the traversal continues to the next voxel.

[0118] S6.4: For each connected domain, find the minimum value among all distances and use it as the minimum distance of the current connected domain and the distance from the instance of current interest to the vehicle.

[0119] S6.5: If the distance is less than a preset safety threshold, an alarm message is generated.

[0120] A second aspect of the present invention relates to a monocular ranging device for automobile assisted driving based on a lightweight occupancy prediction network, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network of the present invention.

[0121] A third aspect of the present invention relates to a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements a monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network of the present invention.

[0122] The innovation of the present invention is:

[0123] (1) Introducing occupancy prediction, which is currently popular in the field of autonomous driving, into the field of monocular ranging. The ability to model 3D space through occupancy prediction overcomes the inherent shortcomings of traditional monocular ranging.

[0124] (2) The model used is lightweight and easy to deploy. The improved 2D feature processing part adds 2D semantic supervision, and the improved BEV feature processing part adds a spatial attention mechanism. The added modules require very little computing power and improve the model's ability to extract 2D features and BEV features.

[0125] (3) The knowledge distillation technology that combines feature distillation and output distillation is used to improve model performance. Feature distillation uses the MLP layer to alleviate the feature differences between the teacher model and the student model, so that the student model can better learn features. Output distillation extends the channel distillation in 2D semantic segmentation to 3D space, which can significantly improve the performance of occupancy prediction, a dense prediction task.

[0126] Compared with the existing technology, the technical solution provided by the present invention has the following beneficial effects:

[0127] Improve ranging accuracy: Traditional monocular ranging methods mainly rely on the width of the detected object bounding box and the known camera intrinsic parameters to estimate the object distance through geometric calculation. This method is highly dependent on the accurate detection of the object. If the detection is inaccurate or the object is blocked, the ranging accuracy will be greatly reduced. And it is usually necessary to know the actual size or appearance characteristics of the target object in advance. Therefore, it is necessary to maintain a database containing information about various objects. This is a challenge for application scenarios with many types of objects or frequent changes. If an object that is not in the database appears, the traditional method may not be able to provide accurate ranging information. At the same time, the ranging accuracy of the traditional method is easily affected by the error of camera parameters and the shape of the object, especially when the object is irregular, the ranging error will increase significantly. When faced with unstable input image quality (such as light changes, motion blur), the size and position of the detection box of the traditional method will change (jitter) between frames, even if the position of the target object is relatively fixed. This jitter directly affects the distance calculated based on the width of the detection box, resulting in unstable ranging. In contrast, the monocular ranging method based on a lightweight occupancy prediction network of the present invention does not rely on prior information of the object, but directly obtains distance information and semantic information through the output of an occupancy prediction network that can perform panoramic perception of 3D space. This method does not require the maintenance of an object database, and can still provide reasonable distance estimates for newly emerging objects or objects of irregular shapes, which not only improves the versatility and adaptability of the system, but also reduces maintenance costs. At the same time, the present invention generates a stable three-dimensional space representation through occupancy prediction, which overcomes the ranging error caused by the jitter of the detection frame in traditional methods.

[0128] Real-time performance and computational efficiency: A lightweight student model with 2D semantic supervision and spatial attention enhances its perception of monocular input images and BEV features. At the same time, a complex teacher model is learned through knowledge distillation technology, which not only retains high-precision features but also significantly reduces model complexity and computational complexity. After model quantization, it can be applied to vehicle systems with high real-time requirements.

[0129] Wide applicability: The technical solution of the present invention is not only applicable to monocular ranging, but can also be extended to other application scenarios related to occupancy prediction, such as path planning and obstacle detection in autonomous driving.

[0130] In summary, the present invention not only ensures model performance, but also takes into account real-time performance and efficient use of computing resources, and provides an efficient and practical solution for monocular ranging and related applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] Figure 1 It is a flowchart of the method of the present invention.

[0132] Figure 2 It is a monocular occupancy prediction teacher model diagram based on multi-frame temporal fusion and temporal stereo depth estimation of the present invention.

[0133] Figure 3 It is a student model diagram of lightweight monocular occupancy prediction based on single frame input of the present invention.

[0134] Figure 4 It is the range of the field of view of the front camera of the present invention in the BEV space.

[0135] Figure 5 It is a schematic diagram of the present invention using a teacher model to perform knowledge distillation on a student model.

[0136] Figure 6 It is a flow chart of the monocular ranging method of the lightweight monocular occupancy prediction student model based on single-frame input of the present invention.

[0137] Figure 7 This is a rendering of the occupancy prediction inference results of the student model after distillation training of the present invention.

[0138] Figure 8 This is a rendering of the effect of the present invention using the occupancy prediction inference result of the student model to perform monocular ranging. DETAILED DESCRIPTION

[0139] The present invention will be described in detail below according to the accompanying drawings and preferred examples, and the purpose and effect of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0140] Example 1

[0141] The principle of the present invention is: first construct a 3D occupancy prediction dataset, and then construct a monocular occupancy prediction teacher model based on multi-frame temporal fusion and temporal stereo depth estimation and a lightweight monocular occupancy prediction student model based on single frame input. The lightweight monocular occupancy prediction student model adds 2D semantic pseudo-label supervision after the FPN layer of its 2D feature processing, and adds a spatial attention module after each residual block of the BEV feature backbone network, which enhances its ability to extract 2D features and BEV features respectively. The teacher model is trained using temporal input pictures to obtain an occupancy prediction teacher model with good performance, and the performance of the teacher model is evaluated. During distillation training, the teacher model is in evaluation (eval) mode, using temporal input, and the student model is in training mode. It only accepts key frames as input, and uses the features of the middle layer of the teacher model and the final output to guide the learning of the student model. After distillation training, a lightweight occupancy prediction student model with improved performance is obtained, and the performance of the student model is evaluated. Finally, the student model is used for reasoning. By performing a connected domain analysis based on breadth-first search on its output results, the connected domains of all instances of interest (such as vehicles, pedestrians, etc.) and the Euclidean distance from each voxel to the vehicle are found, and the smallest Euclidean distance is selected to complete the monocular ranging.

[0142] As one of the implementation methods, the automobile assisted driving monocular ranging method based on lightweight occupancy prediction network of the present invention is as follows: Figure 1 As shown, the specific steps include:

[0143] The step S1 of constructing a 3D occupancy prediction dataset specifically includes the following steps:

[0144] S1.1: Collect sensor data: Use a collection vehicle equipped with multiple sensors (lidar, camera) to collect raw data on the road environment. This data should include point clouds, image data, etc. with multiple time stamps.

[0145] S1.2: Annotated point cloud data: In the collected data, the semantics of the 3D point cloud of the key frame is manually annotated. As shown in Table 1:

[0146] Table 1 Point cloud semantic index

[0147]

[0148]

[0149] S1.3: Generate ground-truth labels for occupancy prediction: Voxelize the annotated point cloud data to generate accurate 3D ground-truth labels.

[0150] Furthermore, the step S1.3 is specifically implemented by the following steps:

[0151] S1.3.1: Aggregate multi-frame point clouds to generate dense point cloud data.

[0152] S1.3.2: Use the K-nearest neighbor algorithm (KNN) to assign semantic labels to unlabeled frames. For each point in the unlabeled frame, find its nearest K labeled frames and assign a semantic label to the point based on the majority voting principle.

[0153] S1.3.3: For the aggregated dense point cloud, the VDBFusion method is used for mesh reconstruction.

[0154] S1.3.4: Densely sample the reconstructed grid, and continue to use KNN to semantically label the sampled points to obtain accurate 3D true values ​​S1.4: Divide the training set, validation set, and test set. The training set and validation set contain 850 scenes and 34,149 frames of data. The test set contains 150 scenes and 6,000 frames of data.

[0155] S1.5: Save the data information in a format that is easy to read, such as .pkl format, to facilitate reading during training and verification.

[0156] The pkl file contains:

[0157] (1) Data identification: including the tokens of the current frame and its previous and next frames, whether it is a key frame, and the scene token where the frame data is located.

[0158] (2) Transformation matrix: including the external parameter matrix of the lidar, the internal and external parameter matrix of the camera, the transformation matrix from the camera coordinate system to the vehicle coordinate system, and the transformation matrix from the vehicle coordinate system to the global coordinate system.

[0159] (3) Storage path: including the point cloud data storage path, the camera image storage path, the 2D semantic pseudo-label storage path, and the occupancy (voxel) true value storage path.

[0160] Furthermore, the vehicle coordinate system, global coordinate system, and occupancy (voxel) true value in step S1.5 are specifically as follows:

[0161] S1.5.1: The ego vehicle coordinate system is a local coordinate system with the vehicle (ego vehicle) as the reference point, used to describe sensor data and the position of objects relative to the vehicle. The origin is located at the center of the ego vehicle, that is, the geometric center of the vehicle chassis. The X axis points to the front of the vehicle. The Y axis points to the left of the vehicle. The Z axis points to the top of the vehicle.

[0162] S1.5.2: The global coordinate system is a fixed world coordinate system used to describe the absolute position of objects in the entire environment. It does not change as the vehicle moves. The origin is located at the point where the longitude and latitude in the GPS are both 0°. The X axis points to the east. The Y axis points to the north. The Z axis points to the sky.

[0163] S1.5.3: 2D semantic pseudo-labels are obtained by projecting the semantically annotated 3D point cloud in S1.2 onto a 2D plane and then performing morphological processing and smoothing operations. Their shape is consistent with the size of the input image.

[0164] S1.5.4: Occupancy (voxel) ground truth contains all scene keyframes and is the voxel label of a 3D grid with shape (200, 200, 16). Each value is the predicted semantic class index of the corresponding voxel in 3D space, and a 17th class is added compared to Table 1: air (free).

[0165] Step S2 describes the construction of a monocular occupancy prediction teacher model based on multi-frame temporal fusion and temporal stereo depth estimation, such as Figure 2 As shown, the specific steps include:

[0166] S2.1: Set the input camera to the front camera, the input image size to 256×704 pixels, the image enhancement scaling ratio to (-0.06, 0.11), and the BEV space X, Y, and Z direction ranges to (-8m, 40m), (-32m, 32m), and (-1m, 5.4m), respectively. The depth prediction range is set to predict a set of distributions of length (45-1) / 0.5=88 for every 0.5m range from 1m to 45m from the vehicle. 2 frames + 1 additional reference frame are used as time series input. The BEV space enhancement flips the BEV features along the X-axis with a probability of 50%. The optimizer uses Adam and the learning rate is set to 1e-4. The batch size during training is 8.

[0167] S2.2: Construct 2D feature extraction module: Use ResNet50 network as the backbone network to extract the features of its 0th, 2nd and 3rd layers. Send the features of different levels into FPN (Feature Pyramid Network) to integrate features of different scales. The feature size of the FPN layer output is (8, 256, 16, 44).

[0168] S2.3: Construct a 2D to 3D feature conversion module: Depth estimation predicts depth and context features simultaneously. When predicting depth, monocular depth prediction and multi-view depth prediction based on MVS (temporal multi-view stereo) are combined. For multi-view depth prediction, first predict the depth center (μ) and depth range (σ), then use μ and σ to generate the depth distribution, and finally combine the monocular depth prediction and the weighted multi-view depth to obtain the final depth. Use the Bev Pooling operation to obtain the BEV feature. For the three consecutive frames of time-series input, extract the BEV features of the current key frame and the previous frame respectively, and project the BEV features of the previous frame to the BEV coordinate system of the current key frame, and fuse the BEV features of multiple frames through the splicing operation along the channel dimension.

[0169] Furthermore, the Bev Pooling operation in step S2.3 projects the 2D features into the BEV space according to the camera intrinsic and extrinsic parameters and the pre-calculated frustum pseudo point cloud, which is specifically implemented by the following sub-steps:

[0170] S2.3.1: Based on the preset depth distribution and image size, construct a frustum pseudo point cloud, which means the pixel coordinates and possible depth distribution of each pixel in the image.

[0171] S2.3.2: Subtract the image enhancement translation matrix from the pixel coordinates of the pseudo point cloud and multiply it by the inverse of the image enhancement rotation matrix to compensate for the impact of the image enhancement operation. Multiply the compensated enhanced pixel coordinates by the inverse matrix of the camera to the ego vehicle and the camera internal parameters to convert them to the ego vehicle coordinates. Multiply the ego vehicle coordinates by the BEV feature enhancement matrix to synchronize the data enhancement in the BEV space. Subtract the three minimum values ​​of the preset three-dimensional direction of the BEV space from the ego vehicle coordinates and divide them by the length of each voxel to obtain the voxel coordinates of the frustum pseudo point cloud in the BEV space, exclude the frustum points that fall outside the preset range, and generate the index of the frustum pseudo point cloud in the BEV space.

[0172] S2.3.3: Filter the context features and depth value indexes that need to be converted according to the index value, sort the frustum pseudo point cloud according to the voxel index, and make the pseudo point clouds belonging to the same voxel adjacent. Use multi-threaded parallel processing, and for multiple pseudo point clouds falling into the same voxel, perform weighted summation of features according to the predicted depth, and finally obtain a BEV feature with a size of (8, 64, 160, 120).

[0173] S2.4: Construct BEV feature processing module: ResNet50 network is used as the backbone network to send features of different levels into FPN (Feature Pyramid Network) to integrate features of different scales. Finally, the FPN layer outputs a feature map with a size of (8, 256, 160, 120).

[0174] S2.5: Construct occupancy prediction head. First, the input feature map of size (8, 256, 160, 120) is input into a convolutional layer for processing to obtain a feature map of size (8, 120, 160, 256), and then the channel to height operation is performed to output a size of (8, 120, 160, 16, 18), which represents the model's prediction scores for all voxels in the 120×160×16 three-dimensional voxel space on 18 categories.

[0175] Furthermore, the channel to height operation in step S2.5 is specifically implemented through the following sub-steps:

[0176] S2.5.1: For the feature map of size (8, 120, 160, 256), it passes through a fully connected layer with an input dimension of 256 and an output dimension of 512, and then passes through a fully connected layer with an input dimension of 512 and an output dimension of 288 after activation by the Softplus function, and obtains a feature map of size (8, 120, 160, 288).

[0177] S2.5.2: Split the last dimension 288 of the feature map into 16×18 to obtain a feature map of size (8, 120, 160, 16, 18), which represents the model's prediction scores for all voxels in the 120×160×16 three-dimensional voxel space on 18 categories.

[0178] Step S3 constructs a lightweight monocular occupancy prediction student model based on single frame input, such as Figure 3 As shown, the specific steps include:

[0179] S3.1: Set the input camera to the front camera, the input image size to 256×704, the image enhancement scaling ratio to (-0.06, 0.11), and the BEV space X, Y, and Z direction ranges to (-8m, 40m), (-32m, 32m), and (-1m, 5.4m), respectively. The depth prediction range is set to predict a set of distributions of length 88 for every 0.5m range from 1m to 45m from the vehicle. The BEV space enhancement flips the BEV features along the X-axis with a probability of 50%. The optimizer uses Adam and the learning rate is set to 1e-4. The batch size during training is 8.

[0180] S3.2: Construct an improved 2D feature extraction module: Use the ResNet50 network as the backbone network to extract the features of its 0th, 2nd and 3rd layers. Send the features of different levels into the FPN (feature pyramid network) to integrate features of different scales. The FPN layer features of size (8, 256, 16, 44) are sent to two residual blocks and then passed through a convolution layer to reduce the number of channels to 18 (corresponding to 18 categories). After 16 times upsampling, they are supervised by 2D semantic pseudo-labels. The FPN layer features and the features after passing through the two residual blocks are concatenated along the channel dimension and passed through a convolution layer to enhance the 2D features.

[0181] S3.3: Construct a 2D to 3D feature conversion module: The enhanced 2D features obtained in S3.2 are further fed into a convolutional layer with a size of (8, 152, 16, 44) to obtain the coupled features of context features and depth estimation. The first 88 channels and the last 64 channels of the coupled features are separated as context features and depth estimation values, respectively, and the Softmax function of the depth estimation value is calculated. The same Bev Pooling operation as in S2.3 is used to obtain the BEV feature.

[0182] S3.4: Construct an improved BEV feature processing module with the ResNet50 network as the backbone network. A spatial attention (SA) module is added after each residual block of the ResNet50 network to spatially enhance the feature extraction of the BEV feature map. The spatial attention module calculates the average and maximum values ​​of the input features in the channel dimension, connects the results along the channel dimension, passes the connected tensor through a convolution layer, and then passes it through the Sigmoid function to generate a spatial attention map. The attention map is multiplied by the input feature to spatially emphasize or weaken certain areas of the original feature. The specific calculation formula for spatial attention is as follows:

[0183] M spatial =σ(Conv 7×7 ([AvgPool(F);MaxPool(F)])) (3.1)

[0184] F out =F×M spatial (3.2)

[0185] Where F is the input feature map, AvgPool(·) and MaxPool(·) are the average pooling and maximum pooling operations along the channel dimension respectively; Conv 7×7 is a 7×7 convolution with 2 input channels and 1 output channel; σ is the Sigmoid function. spatial is the generated spatial attention map. Output feature map F out From the input feature map F and the spatial attention map Mspatial Multiply them together to get .

[0186] S3.5: Construct occupancy prediction head. The input feature map of size (8, 256, 160, 120) is first processed by a convolutional layer to obtain a feature map of size (8, 120, 160, 256), and then the channel to height operation is performed to output a size of (8, 120, 160, 16, 18), which represents the model's prediction scores for all voxels in the 120×160×16 three-dimensional voxel space on 18 categories.

[0187] The training of the occupancy prediction teacher model in step S4 to obtain an occupancy prediction teacher model with good performance and evaluating the performance of the teacher model specifically includes the following steps:

[0188] S4.1: Read three adjacent frames of front camera images, the transformation matrix from the camera coordinate system to the vehicle coordinate system, the transformation matrix from the vehicle coordinate system to the global coordinate system, the camera intrinsic parameters, the rotation and translation matrices for image enhancement, and the enhancement matrix in the BEV space from the training set at a time, which are used as data to be fed into the teacher model during training.

[0189] S4.2: Execute the forward process during training, returning the predicted values ​​of the occupancy network and the depth predicted values ​​predicted by the 2D to 3D feature conversion module, with sizes of (8, 120, 160, 16) and (8, 88, 16, 44) respectively.

[0190] S4.3: Read the real occupancy of the voxel according to the storage path of the voxel truth value contained in the S1.5 pkl file. The size is (8,200,200,16). (200,200,16) refers to the three-dimensional space with 200, 200, and 16 voxels in the front, back, left, right, and top and bottom directions, respectively, with the vehicle as the center. The actual length of each voxel is 0.4m. The true value contains labels from 0 to 17, representing 18 categories for classifying the voxel space.

[0191] S4.4: According to the storage path of the point cloud data contained in the pkl file in S1.5, the point cloud data is read, and the true value of the depth is generated through the point cloud data to supervise the depth value predicted by the 2D to 3D feature conversion module.

[0192] Furthermore, step S4.4 is implemented by the following sub-steps:

[0193] S4.4.1: Read the three-dimensional coordinates (x, y, z) of the point cloud data from the pkl file.

[0194] S4.4.2: Use the transformation matrix from the vehicle coordinate system to the camera coordinate system to convert the read point cloud data into the camera coordinate system.

[0195] S4.4.3: The point cloud data converted to the camera coordinate system is further projected onto the image plane: Through the camera intrinsic parameter matrix, the points in the three-dimensional space are projected onto the pixel coordinates (u, v) on the two-dimensional image plane while maintaining the depth information d (i.e., distance).

[0196] S4.4.4: Generate depth map: For each point projected onto the image plane, record its corresponding depth value. Fill in the corresponding depth value d at the pixel coordinate (u, v) to generate a depth map. Select the point closest to the camera as the depth value of the pixel in the depth map.

[0197] S4.5: Create a mask with a size of (8, 160, 120, 16) to exclude the true occupancy values ​​outside the field of view of the front camera. The range of the field of view of the front camera in the BEV space is as follows: Figure 4 As shown: The length of the BEV space in the front-to-back direction is 120 voxels, and the width in the left-to-right direction is 160 voxels. The ego vehicle is located on the center line of the BEV space, 20 voxels behind, and the two vertices of the base of the triangle of its field of view are 10 voxels away from each side.

[0198] S4.6: Calculate the binary cross entropy loss between the predicted depth and the true depth:

[0199]

[0200] where d and They are the true depth label and the predicted depth respectively.

[0201] S4.7: Calculate the cross entropy loss between the predicted occupancy and the true occupancy:

[0202]

[0203] C is the total number of categories, 18. i It is the one-hot true label of the i-th class obtained from the true voxel occupancy value in S4.3. is the probability of the i-th class predicted by the model. When calculating the loss, the mask is used to exclude the loss outside the field of view of the front camera.

[0204] S4.8: Final loss L total for:

[0205] L total =λ depth L depth +λ occ L occ (4.3)

[0206] where λ depth and λ occ is the weight of depth loss and occupancy loss, taking 0.05 and 1. Return the final loss and perform the back propagation process to update the gradient.

[0207] S4.9: Put the trained teacher model in the evaluation (eval) mode, perform forward reasoning, calculate the mIoU (mean intersection over union) value between the reasoning result and the true value of the occupancy prediction, and evaluate the performance of the teacher model through the mIoU value. The calculation of the mIoU value is as follows:

[0208] For each category c, calculate its IoU (intersection over union) value. IoU is defined as the intersection of the predicted area and the true area divided by the union of the predicted area and the true area. The formula is as follows:

[0209]

[0210] Where: TP(c) is the true positive of category c. FP(c) is the false positive of category c. FN(c) is the false negative of category c.

[0211] mIoU is the average IoU of all categories. The formula is as follows:

[0212]

[0213] Where C is the number of categories.

[0214] In step S5, the intermediate layer features and outputs of the teacher network are used to perform distillation training on the student model to obtain a lightweight occupancy prediction student model with improved performance, and the performance of the student model is evaluated, such as Figure 5 As shown, the specific steps include:

[0215] S5.1: Read and fix the weights of the trained teacher model, and put the teacher model in evaluation (eval) mode and the student model in training (train) mode.

[0216] S5.2: In one training cycle, read three adjacent frames of front camera images from the training set at a time and feed them into the teacher model to perform forward reasoning to obtain the feature map T of the last convolutional layer in the occupancy prediction head. feat and the output T of the occupancy prediction head out . The dimensions are (8,256,160,120) and (8,120,160,288).

[0217] S5.3: In a training cycle, the image of the current frame is sent to the student model to perform the forward process of training, and the output of the occupancy prediction head is obtained, and the feature map S of the last convolutional layer in the occupancy prediction head is obtained. feat and the output S of the occupancy prediction head out . The dimensions are (8,256,160,120) and (8,120,160,288).

[0218] S5.4: For S feat After one MLP layer and T feat Calculate L2 loss:

[0219]

[0220] The MLP layer consists of 1×1 convolution + ReLU + 1×1 convolution, N is the batch size during training, which is 8. When calculating the L2 loss, a mask is used to calculate only the valid voxels within the field of view of the front camera.

[0221] S5.5: From T out and S out The occupancy prediction in the height direction is decoded, and the prediction scores y of the teacher model and the student model for all voxels in the three-dimensional voxel space for all categories of objects are obtained. T and S The dimensions are (8,120,160,16,18).

[0222] S5.6: For the predictions of the teacher model and the student model in a training batch, use the mask to find the valid voxels within the field of view of the front camera, and calculate the KL divergence as the distillation loss for the prediction scores of the 18 voxel spaces corresponding to the 18 categories of the teacher model and the student model. The distillation loss is calculated as follows:

[0223]

[0224] It is used to transform the eigenvalues ​​into a probability distribution as shown below:

[0225]

[0226] y T and S are the outputs of the teacher model and the student model, respectively, representing the prediction scores of valid voxels for C = 18 categories in a 3D voxel space of size W·H·Z. T is the distillation temperature, which is used to smooth the prediction distribution of the student model. The larger T is, the more attention is paid to a wider area in the 3D space.

[0227] During the entire training process, the distillation temperature changes according to the following formula, following the distillation strategy of high temperature first and then low temperature. The temperature T in each round of training is calculated according to the following formula:

[0228]

[0229] Where T0 is the starting temperature, which is 4. end is the end temperature, take 2. E total is the total number of training rounds, which is 24. e is the current number of training rounds.

[0230] S5.7: For the 18 prediction categories, they are divided into three categories: foreground objects, background objects, and air. Different weights are given to these three categories of objects when calculating the distillation loss. The specific division and weights of the three categories of foreground, background, and air are shown in Table 2:

[0231] Table 2 Foreground / background / air division and weight

[0232]

[0233] S5.8: The total losses during the distillation process are as follows:

[0234] L total =λ seg L seg +λ occ L occ +λ feat L feat +λ cw3d L cw3d (5.5)

[0235] Where L seg is the 2D semantic loss of the student model itself, that is, the cross entropy loss function between the 2D semantic map and the 2D semantic pseudo-label:

[0236]

[0237] in is the model's true category y at pixel (i, j) i，j The predicted probability of . H and W are the height and width of the feature map respectively.

[0238] L occ is the occupancy prediction loss of the student model itself, which is calculated the same as formula (4.2) in S4.7.

[0239] L feat is the L2 distillation loss of the feature map.

[0240] L cw3d is the distillation loss of the final output.

[0241] λ seg, occ , feat and λ cw3d are the weights of the four. They are set to 0.5, 1, 5e-7 and 1. For the distillation loss, a cosine annealing learning rate strategy is set: from the initial learning rate of 0.0004 to 1% of the initial learning rate according to the cosine function over the entire 24 training cycles. The final loss is returned and the back-propagation process is performed to update the gradient.

[0242] S5.9: Put the student model after distillation training in the evaluation (eval) mode, perform forward reasoning, calculate the mIoU (mean intersection over union) value between the reasoning result and the true value of the occupancy prediction, and evaluate the performance of the student model by the mIoU value. The calculation of the mIoU value is the same as formula (4.5) in S4.9.

[0243] The student model is forward-reasoned in step S6, and monocular ranging is completed according to the reasoning result, such as Figure 6 As shown, the specific steps include:

[0244] S6.1: The vehicle-mounted forward-view camera image, the conversion matrix from the camera coordinate system to the vehicle coordinate system, the conversion matrix from the vehicle coordinate system to the global coordinate system, and the camera internal parameters are input into the student model. The student model is in evaluation (eval) mode and performs forward reasoning to obtain a prediction with a shape of (120, 160, 16).

[0245] S6.2: Initialize the data structure, including: an array of the same shape as the prediction: labeled array, used to store the connected region identifier. A Boolean array: visited, used to record the visited voxels. A list: region props, used to store the properties of each connected region, such as the number of voxels, category, the voxel closest to the ego vehicle, and the distance.

[0246] S6.3: For the area within the field of view of the front camera, use a breadth-first search strategy to find the connected domain of each object of interest. The connected domain adopts 6-connectivity, that is, it is considered that the 6 voxels above, below, left, right, front and back of a voxel are connected to it. For each category of interest, traverse all voxels. When a voxel that meets the requirements is found, use a breadth-first search to find all connected voxels: use a queue to store the voxel coordinates of the current connected domain, for each connected domain, traverse all its voxels, and calculate the Euclidean distance of each voxel to the vehicle.

[0247] Furthermore, step S6.3 is implemented by the following sub-steps:

[0248] S6.3.1: Traverse all voxels, and the voxels that meet the requirements meet the following conditions: 1. The voxel is within the field of view of the front camera. 2. The voxel belongs to the category of interest. 3. The voxel has not been visited.

[0249] S6.3.2: For voxels that meet the requirements, initialize the breadth-first search queue and add the current voxel coordinates to the queue, and mark it as visited. Initialize the voxel count of the current connected domain to 1, and record the current voxel as the voxel closest to the ego vehicle, and the distance from the current voxel to the ego vehicle is the minimum distance.

[0250] S6.3.3: Perform a breadth-first search until the queue is empty. Each time a voxel coordinate is taken from the queue, check whether its 6 adjacent voxels meet the requirements of S6.3.1. For voxels that meet the requirements, mark them as visited and add them to the queue. At the same time, update the voxel counter of the connected domain and calculate its distance to the ego vehicle. If it is less than the previously calculated result, update the minimum distance.

[0251] S6.3.4: After the search is completed, if the voxel counter of the current connected domain is greater than 1, the connected domain is recorded in the labeled array, and the properties of the connected domain, including the number of voxels, category, the voxel closest to the vehicle, and the distance, are recorded in the region props, and the traversal continues to the next voxel.

[0252] S6.4: For each connected region, take the minimum distance recorded in the region props as the distance from the instance of current interest to the vehicle.

[0253] S6.5: If the distance is less than a preset safety threshold, an alarm message is generated.

[0254] After experiments, the teacher model has a computational load of 132.5 GFLOPS, a parameter load of 58.67M, and an mIoU value of 33.37. The original student model has a computational load of 90.14 GFLOPS, a parameter load of 44.47M, and an mIoU value of 27.88. After using the improved 2D feature processing part and the improved BEV feature processing part and distillation training, the student model has a computational load of 92.64 GFLOPS, a parameter load of 48.33M, and an mIoU value of 29.82.

[0255] Figure 7 The occupancy prediction inference results of the student model after distillation training using the present invention are given, which can clearly distinguish structures such as houses, roads, trees, and vehicles. Figure 8 This is the effect diagram of monocular ranging using the occupancy prediction inference results of the student model, which measures the distance of vehicles and pedestrians within the field of view of the front camera.

[0256] Example 2

[0257] The present embodiment relates to a monocular ranging device for automobile assisted driving based on a lightweight occupancy prediction network, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement a monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network in embodiment 1.

[0258] Example 3

[0259] This embodiment relates to a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network is implemented in this embodiment 1.

[0260] The present invention is expected to solve the technical bottlenecks faced by existing monocular ranging systems and provide strong support for the intelligent development of the field of automobile assisted driving. This will not only help improve the safety and accuracy of vehicles in complex traffic environments, but also reduce the hardware requirements and costs of the system and expand the application scope of intelligent driving technology. In addition, through more accurate environmental perception and real-time response capabilities, the present invention will provide a higher user experience and safety guarantee for intelligent driving.

[0261] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network, comprising the following steps: S1: Construct 3D occupancy prediction dataset; S2: Construct a monocular occupancy prediction teacher model based on multi-frame temporal fusion and temporal stereo depth estimation; S3: Construct a lightweight monocular occupancy prediction student model based on single-frame input; S4: train the occupancy prediction teacher model to obtain an occupancy prediction teacher model with good performance, and evaluate the performance of the teacher model; S5: Use the intermediate layer features and outputs of the teacher network to perform distillation training on the student model, obtain a lightweight occupancy prediction student model with improved performance, and evaluate the performance of the student model; S6: Perform forward reasoning on the student model and complete monocular ranging based on the reasoning results.

2. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 1, characterized in that: Step S1 is specifically implemented by the following steps: S1.1: Collect sensor data: Use a collection vehicle equipped with multiple sensors to collect raw data on the road environment; S1.2: Labeled point cloud data: In the collected data, the semantics of the 3D point cloud of the key frames are manually labeled; the values ​​in the labels are encoded as integers starting from 0, and each integer corresponds to a category; S1.3: Generate occupancy prediction ground truth labels: Voxelize the annotated point cloud data to generate accurate 3D ground truth values; specifically, S1.3.1: Aggregate multi-frame point clouds to generate dense point cloud data; S1.3.2: Use the K-nearest neighbor algorithm (KNN) to assign semantic labels to unlabeled frames. For each point in the unlabeled frame, find the points in the nearest K labeled frames and assign a semantic label to the point based on the majority voting principle. S1.3.3: For the aggregated dense point cloud, the VDBFusion method is used to perform mesh reconstruction (Meshreconstruction); S1.3.4: Densely sample the reconstructed grid and continue to use KNN to semantically label the sampled points to obtain accurate 3D truth values; S1.4: Divide the training set, validation set and test set; S1.5: Save the data information in a format that is easy to read, including: pkl format, which is easy to read during training and verification; the pkl file contains: (1) Data identification: including the tokens of the current frame and its previous and next frames, whether it is a key frame, and the scene token of the frame data; (2) Conversion matrix: including the external parameter matrix of the lidar, the internal and external parameter matrix of the camera, the conversion matrix from the camera coordinate system to the vehicle coordinate system, and the conversion matrix from the vehicle coordinate system to the global coordinate system; (3) Storage path: including the point cloud data storage path, camera image storage path, 2D semantic pseudo-label storage path, and occupancy (voxel) true value storage path; The ego-vehicle coordinate system, global coordinate system, 2D semantic pseudo-label, and occupancy true value described in step S1.5 are as follows: S1.5.1: The ego vehicle coordinate system is a local coordinate system with the vehicle (ego vehicle) as the reference point, used to describe sensor data and the position of objects relative to the vehicle; the origin is located at the center of the ego vehicle, that is, the geometric center of the vehicle chassis; the X axis points to the front of the vehicle; the Y axis points to the left of the vehicle; and the Z axis points to the top of the vehicle. S1.5.2: The global coordinate system is a fixed world coordinate system used to describe the absolute position of objects in the entire environment; it does not change with the movement of the vehicle; the origin is located at the point where the latitude and longitude in the GPS are both 0°; the X axis points to the east; the Y axis points to the north; and the Z axis points to the sky; S1.5.3: 2D semantic pseudo-labels are obtained by projecting the semantically annotated 3D point cloud in S1.2 onto a 2D plane and then performing morphological processing and smoothing operations; S1.5.4: The ground truth occupancy (voxel) contains all scene keyframes, and the shape is (D x ,D y ,D z ), indicating that the size is D x ×D y ×D z The predicted semantic class index of each voxel in the three-dimensional space of D; Referring to step S1.3, compared with the semantic index of the point cloud in S1.2, an air (free) class is added, which represents the voxel that does not contain any object in the three-dimensional space; D x , D y , D z They are the number of voxels in the front-back, left-right, and up-down directions.

3. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 1, characterized in that ,,Step S2 is specifically implemented by the following steps: S2.1: Set the input camera, input image size, image enhancement matrix, BEV spatial grid size, depth prediction range, multi-frame input index, BEV feature enhancement matrix, optimizer, and learning rate; S2.2: Construct 2D feature extraction module: Use ResNet50 network as the backbone network to extract multi-scale features; send features at different levels into FPN (feature pyramid network) to integrate features at different scales; S2.3: Construct a 2D to 3D feature conversion module: Depth estimation predicts depth and context features simultaneously; when predicting depth, combine monocular depth prediction and multi-view depth prediction based on MVS (temporal multi-view stereo); for multi-view depth prediction, first predict the depth center (μ) and depth range (σ), then use μ and σ to generate the depth distribution, and finally combine the monocular depth prediction and the weighted multi-view depth to obtain the final depth; use the Bev Pooling operation to obtain the BEV feature; for the continuous n-frame images of the temporal input, extract their respective BEV features, and uniformly project the BEV features of the non-current key frame to the BEV coordinate system of the current key frame, and fuse the BEV features of multiple frames through the splicing operation along the channel dimension; The Bev Pooling operation in step S2.3 projects the 2D features into the BEV space based on the camera intrinsic and extrinsic parameters and the pre-calculated frustum pseudo point cloud, which is implemented by the following sub-steps: S2.3.1: Based on the preset depth distribution and image size, construct a frustum pseudo point cloud, which means the pixel coordinates and possible depth distribution of each pixel on the image; S2.3.2: Subtract the image enhancement translation matrix from the pixel coordinates of the pseudo point cloud and multiply it by the inverse of the image enhancement rotation matrix to compensate for the impact of the image enhancement operation; multiply the compensated enhanced pixel coordinates by the inverse matrix of the camera to the ego vehicle and the camera internal parameter to convert them to the ego vehicle coordinates; multiply the ego vehicle coordinates by the BEV feature enhancement matrix to synchronize the data enhancement in the BEV space; subtract the three minimum values ​​of the preset three-dimensional direction of the BEV space from the ego vehicle coordinates and divide them by the length of each voxel to obtain the voxel coordinates of the frustum pseudo point cloud in the BEV space, exclude the frustum points that fall outside the preset range, and generate the index of the frustum pseudo point cloud in the BEV space; S2.3.3: Filter out the context features and depth value indexes that need to be converted according to the index value, sort the frustum pseudo point cloud according to the voxel index, and make the pseudo point clouds belonging to the same voxel adjacent; use multi-threaded parallel processing, and for multiple pseudo point clouds falling into the same voxel, perform weighted summation of features according to the predicted depth, and finally obtain the BEV feature; S2.4: Construct BEV feature processing module: ResNet50 network is used as the backbone network to send features at different levels into FPN (feature pyramid network) to integrate features at different scales; S2.5: Construct an occupancy prediction head; first, input the input feature map into a convolutional layer for processing, and then perform a channel to height operation to decode the height direction voxel prediction for all categories from the channel dimension of the feature map.

4. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 3, characterized in that ,,The channel to height operation in step S2.5 is implemented by the following sub-steps: S2.5.1: For input channels C f The feature map is output after the full connection layer + activation function + full connection layer, and the number of channels is D z ×C; where D z and C are the number of voxels in the height direction and the total number of voxel categories in S1.5.4, respectively; S2.5.2: Split the output channel into a size of (D x ,D y ,D z ,C) feature map, which represents the model’s prediction scores for all voxels in the three-dimensional voxel space in C categories.

5. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 1, characterized in that ,,Step S3 is specifically implemented by the following steps: S3.1: Set the input camera, input image size, image enhancement matrix, BEV spatial grid size, depth prediction range, BEV feature enhancement matrix, optimizer, and learning rate; S3.2: Construct an improved 2D feature extraction module: Use the ResNet50 network as the backbone network to extract multi-scale features; send features at different levels to the FPN (feature pyramid network) to integrate features at different scales; send the output features of the FPN layer to two residual blocks and then upsample them, supervised by 2D semantic pseudo-labels; concatenate the features of the FPN layer and the features after passing through the two residual blocks along the channel dimension and pass through a convolution layer to enhance the 2D features; S3.3: Construct a 2D to 3D feature conversion module: send the features obtained in S3.2 to a convolutional layer, output the coupled features of the context features and the depth estimation; separate the context features and the depth estimation from the coupled features; use the same Bev Pooling operation as S2.3 to obtain the BEV features; S3.4: Construct an improved BEV feature processing module, with the ResNet50 network as the backbone network; add a spatial attention (SA) module after each residual block of the ResNet50 network to spatially enhance the feature extraction of the BEV feature map; the calculation formula of spatial attention is as follows: M spatial =σ(Conv 7×7 ([AvgPool(F);MaxPool(F)])) (3.1) F out =F×M spatial (3.2) Where F is the input feature map, AvgPool(·) and MaxPool(·) are the average pooling and maximum pooling operations along the channel dimension respectively; Conv 7×7 is a 7×7 convolution with 2 input channels and 1 output channel; σ is the Sigmoid function; M spatial is the generated spatial attention map; output feature map F out From the input feature map F and the spatial attention map M spatial Multiply them together to get; S3.5: Construct an occupancy prediction head; the input feature map is first processed by a convolutional layer, and then a channel to height operation is performed to decode the height direction voxels for all categories from the channel dimension of the feature map.

6. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 1, characterized in that ,,Step S4 is specifically implemented by the following steps: S4.1: Read multiple front camera images input in time series from the training set at once, the conversion matrix from the camera coordinate system to the vehicle coordinate system, the conversion matrix from the vehicle coordinate system to the global coordinate system, the camera intrinsic parameters, the rotation and translation matrix for image enhancement, and the enhancement matrix in the BEV space, which are used as data to be fed into the teacher model during training; S4.2: Execute the forward process during training, returning the predicted value of the occupancy network and the depth predicted value predicted by the 2D to 3D feature conversion module; S4.3: Read the true occupancy of the voxel according to the storage path of the true value of the voxel contained in the S1.5pkl file; S4.4: Read the point cloud data according to the storage path of the point cloud data contained in the S1.5pkl file, and generate the true value of the depth through the point cloud data to supervise the depth value predicted by the 2D to 3D feature conversion module; Step S4.4 is implemented by the following sub-steps: S4.4.1: Read the three-dimensional coordinates (x, y, z) of the point cloud data from the pkl file; S4.4.2: Use the transformation matrix from the vehicle coordinate system to the camera coordinate system to convert the read point cloud data into the camera coordinate system; S4.4.3: Further project the point cloud data converted to the camera coordinate system onto the image plane: Project the points in the three-dimensional space onto the pixel coordinates (u, v) on the two-dimensional image plane using the camera intrinsic parameter matrix while maintaining the depth information d (i.e., distance); S4.4.4: Generate depth map: For each point projected onto the image plane, record its corresponding depth value; fill in the corresponding depth value d at the pixel coordinate (u, v) to generate a depth map; select the point closest to the camera as the depth value of the pixel in the depth map; S4.5: Create a mask to exclude the ground truth occupancy values ​​outside the field of view of the front camera; S4.6: Calculate the binary cross entropy loss between the predicted depth and the true depth: where d and They are the true depth label and the predicted depth respectively; S4.7: Calculate the cross entropy loss between the predicted occupancy and the true occupancy: C is the total number of voxel categories in S1.5.4; i is the one-hot true label of the i-th class obtained from the true voxel occupancy value in S4.3; is the probability of the i-th category predicted by the model; when calculating the loss, the mask is used to exclude the loss outside the field of view of the front camera; S4.8: Final loss L total for: L total =λ1L depth +λ2L occ (4.3) Where λ1 and λ2 are the weights of depth loss and occupancy loss; Return the final loss and perform the back-propagation process to update the gradient; S4.9: Put the trained teacher model in the evaluation (eval) mode, perform forward reasoning, calculate the mIoU (mean intersection over union) value between the reasoning result and the true value of the occupancy prediction, and evaluate the performance of the teacher model by the mIoU value; the calculation of the mIoU value is as follows: For each category c, calculate its IoU (intersection over union) value; IoU is defined as the intersection of the predicted area and the true area divided by the union of the predicted area and the true area; the formula is as follows: Where: TP(c) is the true positive of category c; FP(c) is the false positive of category c; FN(c) is the false negative of category c. mIoU is the average IoU of all categories; the formula is as follows: Where C is the number of categories.

7. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 1, characterized in that ,, the step S5 is specifically implemented by the following steps: S5.1: Read and fix the weights of the trained teacher model, and put the teacher model in evaluation (eval) mode and the student model in training (train) mode; S5.2: In one training cycle, read three adjacent frames of front camera images from the training set at a time and feed them into the teacher model to perform forward reasoning to obtain the feature map T of the last convolutional layer in the occupancy prediction head. feat and the output of the occupancy prediction head T out ; S5.3: In a training cycle, the image of the current frame is sent to the student model to perform the forward process of training, and the feature map S of the last convolutional layer in the occupancy prediction head is obtained. feat and the output S of the occupancy prediction head out ; S5.4: For S feat After one MLP layer and T feat Calculate L2 loss: The MLP layer consists of 1×1 convolution + ReLU + 1×1 convolution, where N is the batch size during training. When calculating the L2 loss, a mask is used to calculate only the valid voxels within the field of view of the front camera. S5.5: From T out and S out The occupancy prediction in the height direction is decoded, and the prediction scores y of the teacher model and the student model for all voxels in the three-dimensional voxel space for all categories of objects are obtained. T and s ; S5.6: For the predictions of the teacher model and the student model in a training batch, use the mask to find the valid voxels within the field of view of the front camera, and calculate the k1 divergence of the prediction scores of the voxel space corresponding to all categories of the teacher model and the student model as the distillation loss. The distillation loss is calculated as follows: It is used to transform the eigenvalues ​​into a probability distribution as shown below: y T and s are the outputs of the teacher model and the student model, respectively, representing the prediction scores of C categories for valid voxels in a three-dimensional voxel space of size W·H·Z; T is the distillation temperature, which is used to smooth the prediction distribution of the student model; the larger T is, the more attention is paid to a wider area in the three-dimensional space; During the entire training process, the distillation temperature changes according to the following formula, following the distillation strategy of high temperature first and then low temperature. The temperature T in each round of training is calculated according to the following formula: Where T0 is the starting temperature, T end is the end temperature, E total is the total number of training rounds, e is the current number of training rounds; S5.7: The predicted categories of voxels in S1.5.4 are divided into three categories: foreground objects, background objects and air according to actual needs; foreground objects can include vehicles and pedestrians on the road and other objects of focus; background objects can include trees, roads, buildings, etc.; different weights are given to these three types of objects when calculating the distillation loss; S5.8: The total losses during the distillation process are as follows: L total =λ seg L seg +λ occ L occ +λ feat L feat +λ cw3d L cw3d (5.5) Where L seg is the 2D semantic loss of the student model itself, that is, the cross entropy loss function between the 2D semantic map and the 2D semantic pseudo-label: in is the model's true category y at pixel (i, j) i，j The predicted probability of; H and W are the height and width of the feature map respectively; L occ is the occupancy prediction loss of the student model itself, which is calculated the same as formula (4.2) in S4.7; L feat is the L2 distillation loss of the feature map; L cw3d is the distillation loss of the final output; λ seg , occ , feat and λ cw3d are the weights of the four respectively; for distillation loss, set the cosine annealing learning rate strategy; Return the final loss and perform the back-propagation process to update the gradient; S5.9: Put the student model that has completed distillation training in evaluation (eval) mode, perform forward reasoning, calculate the mIoU (mean intersection over union) value between the reasoning result and the true value of the occupancy prediction, and evaluate the performance of the student model through the mIoU value; the calculation of the mIoU value is the same as formula (4.5) in S4.

9.

8. The monocular ranging method for automobile assisted driving based on lightweight occupancy prediction network as claimed in claim 1, characterized in that ,, the step S6 is specifically implemented by the following steps: S6.1: Input the vehicle-mounted forward-view camera image, the conversion matrix from the camera coordinate system to the vehicle coordinate system, the conversion matrix from the vehicle coordinate system to the global coordinate system, and the camera internal parameters into the student model. The student model is in evaluation (eval) mode and performs forward reasoning to obtain a prediction of shape (Dx, Dy, Dz); S6.2: Initialize the data structure, including: an array of the same shape as the prediction: labeled array, used to store the connected domain identifier; a Boolean array: visited, used to record the visited voxels; a list: regionprops, used to store the properties of each connected region, such as the number of voxels, category, the voxel closest to the vehicle, and the distance; S6.3: Use the breadth-first search strategy to find the connected domain of each object of interest and the voxel coordinates closest to the ego vehicle, and record the connected domains that meet the requirements; for each connected domain, traverse all its voxels and calculate the Euclidean distance from each voxel to the ego vehicle; Furthermore, step S6.3 is implemented by the following sub-steps: S6.3.1: Traverse all voxels, and the voxels that meet the requirements meet the following conditions:

1. The voxel is within the field of view of the front camera; 2. The voxel belongs to the category of interest; 3. The voxel has not been visited; S6.3.2: For voxels that meet the requirements, initialize the breadth-first search queue and add the current voxel coordinates to the queue, and mark it as visited; initialize the voxel counter of the current connected domain to 1, and record the current voxel as the voxel closest to the ego vehicle, and the distance from the current voxel to the ego vehicle is the minimum distance; S6.3.3: Perform breadth-first search until the queue is empty; take out a voxel coordinate from the queue each time and check whether its 6 adjacent voxels meet the requirements of S6.3.1; for voxels that meet the requirements, mark them as visited and add them to the queue, update the voxel counter of the connected domain, and calculate the distance from the voxel to the ego vehicle. If it is less than the previously calculated result, update the minimum distance; S6.3.4: After the search is completed, if the voxel counter of the current connected domain is greater than 1, the connected domain is recorded in the labeled array, and the properties of the connected domain, including the number of voxels, category, the voxel closest to the ego vehicle, and the distance, are recorded in the region props, and the traversal continues to the next voxel; S6.4: For each connected domain, find the minimum value among all distances and use it as the minimum distance of the current connected domain and the distance from the instance of current interest to the vehicle; S6.5: If the distance is less than a preset safety threshold, an alarm message is generated.

9. A monocular ranging device for automobile assisted driving based on a lightweight occupancy prediction network, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a monocular ranging method for automobile assisted driving based on a lightweight occupancy prediction network as described in any one of claims 1 to 8 is implemented.

Images