Depth estimation model training method, system and evaluation method based on random activation

Abstract

The present invention provides a random activation-based depth estimation model training method, system, and evaluation method, including collecting image data and point cloud data of a road scene; projecting the point cloud data onto the image to obtain a fusion image, and then obtaining a baseline true depth map; dividing the image data into a training set and a test set, using the images in the training set as training images and sequentially inputting them into the depth estimation model to be trained, and computing and processing them to obtain a predicted depth map; masking the baseline true depth map using a random activation method to obtain a masked baseline depth map; calculating the model's loss value using a loss function based on the masked baseline depth map and the predicted depth map; determining whether the model has converged; if not, updating the model parameters using the loss value through a backpropagation algorithm; if so, completing the training and obtaining a trained depth estimation model. The present invention uses less point cloud data to train a model with accuracy approaching or even exceeding that of full point cloud training.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a depth estimation model training method, system and evaluation method based on random activation. Background Art

[0002] Depth estimation is a computer vision task that aims to estimate the depth from a 2D image. This task takes as input an RGB image and outputs a depth image, which contains information about the distance to objects in the image from the viewpoint, which is usually the camera that captured the image.

[0003] Currently, supervised deep learning-based solutions are extremely dependent on the quality of the data we input to the model. For the task of depth estimation, we first need to project the 3D point cloud data collected by the lidar onto the camera's imaging plane based on the internal and external parameters between the lidar and the camera to obtain the ground truth data used for supervised training. If the quality of the supervised data is poor, the effectiveness of the trained model will inevitably be reduced.

[0004] Chinese patent CN107274445B discloses an image depth estimation method and system, which includes: constructing a depth estimation network, the depth estimation network includes: an encoding part, a convolution connection part and a decoding part, the deconvolution layer of the decoding part is connected to the last convolution layer in the convolution block of the same scale as the encoding part to form a final deconvolution layer; selecting two consecutive images in the sample image and the depth map of one of the images as training samples, using the training samples to train the depth estimation network to obtain a trained depth estimation network; collecting a test image, extracting the current frame image and the previous frame image of the test image; inputting the color channels of the current frame image and the previous frame image into the trained depth estimation network to obtain the depth map of the current frame image.

[0005] The above solution relies on the data quality of the collected images. In one case, the FOV of a single lidar is limited, resulting in blind spots. Projecting the point cloud onto the camera imaging plane may only cover an area in the middle of the image. The areas near the upper and lower edges of the image are blind spots that the point cloud cannot cover. If used for training, only data from the limited area covered by the point cloud can be used. In another case, the point cloud data generated by a single lidar is not dense enough, resulting in the data points we construct for supervised training being too sparse, making it difficult for the model to learn effective depth estimation capabilities with a small amount of data. In this case, we usually compensate for this with hardware, that is, adding multiple lidars to make the point cloud dense enough to improve model performance, but this will lead to problems such as increased costs. Summary of the Invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a depth estimation model training method, system and evaluation method based on random activation.

[0007] In a first aspect, the present invention provides a depth estimation model training method based on random activation, comprising:

[0008] Step S1: using a camera and a lidar to collect image data and point cloud data of a road scene respectively;

[0009] Step S2: Projecting the point cloud data onto the image to obtain a fusion map based on the intrinsic and extrinsic parameters of the camera and the lidar, and selecting a highly matched fusion map as a reference true depth map;

[0010] Step S3: dividing the image data into a training set and a test set, and using the images in the training set as training images;

[0011] Step S4: inputting the training images into the depth estimation model to be trained in sequence, and obtaining a predicted depth map through calculation and processing;

[0012] Step S5: performing mask processing on the reference true depth map by a random activation method to obtain a masked reference depth map;

[0013] Step S6: Calculating a loss value of the model using a loss function according to the mask reference depth map and the predicted depth map;

[0014] Step S7: Determine whether the model has converged. If not, update the model parameters using the loss value through the back propagation algorithm. If converged, the training is completed and a trained depth estimation model is obtained.

[0015] Preferably, the number of the laser radar used to collect road scene point cloud data is one;

[0016] The high degree of matching means that the two-dimensional points projected by the point cloud onto the image accurately correspond to the objects in the image in terms of position, and the average deviation between the two is less than a preset threshold.

[0017] Preferably, the step S3 further comprises: performing data enhancement on the images in the training set to obtain training images;

[0018] The data enhancement includes horizontally flipping the image according to a preset probability, and / or adjusting the brightness, and / or adjusting the contrast, and / or adjusting the color.

[0019] Preferably, the depth estimation model adopts an encoder-decoder network structure, the encoder part includes a dense feature extractor, and the decoder part includes a context information extractor, three local plane guidance layers, a dimensionality reduction layer, three convolution layers, three upsampling layers and five downsampling layers;

[0020] The context information extractor is a spatial pyramid pooling layer based on void convolution; the three local plane guiding layers are respectively the first, second and third local plane guiding layers, each local plane guiding layer includes a dimensionality reduction link, a unit normal vector calculation link, an activation function link and a locally defined depth calculation link, which is used to guide low-resolution features to full resolution, and then the local plane guiding layers are connected together, and the final depth prediction is performed through the last convolution layer; the three convolution layers are respectively the first, second and third convolution layers; the three upsampling layers are respectively the first, second and third upsampling layers; the five downsampling layers are respectively the first, second, third, fourth and fifth downsampling layers.

[0021] Preferably, step S4 includes:

[0022] Step S4.1: The training images are sequentially input into the dense feature extractor of the depth estimation model to be trained to obtain original feature maps. The original feature maps are then input into the first, second, and third downsampling layers for 8x, 4x, and 2x downsampling, respectively, to obtain first, second, and third intermediate feature maps.

[0023] Step S4.2: Input the first intermediate feature map into the context information extractor to obtain a first intermediate input feature map containing multi-scale semantic information; then input the first intermediate input feature map into the first local plane guide layer to obtain a first intermediate depth prediction result, and at the same time input the first intermediate input feature map into the first upsampling layer for 2x upsampling to obtain a fourth intermediate feature map; then input the first intermediate depth prediction result into the fourth downsampling layer for 4x downsampling to obtain a fifth intermediate feature map; then concatenate the second intermediate feature map, the fourth intermediate feature map, and the fifth intermediate feature map and input the concatenated feature map into the first convolutional layer for fusion to obtain a second intermediate input feature map;

[0024] Step S4.3: Input the second intermediate input feature map into the second local plane guide layer to obtain a second intermediate depth prediction result, and input the second intermediate input feature map into the second upsampling layer for 2x upsampling to obtain a sixth intermediate feature map; then input the second intermediate depth prediction result into the fifth downsampling layer for 2x downsampling to obtain a seventh intermediate feature map; then concatenate the third intermediate feature map, the sixth intermediate feature map, and the seventh intermediate feature map, and input the concatenated features into the second convolutional layer for fusion to obtain a third intermediate input feature map;

[0025] Step S4.4: Input the third intermediate input feature map into a third local plane guiding layer to obtain a third intermediate depth prediction result. Simultaneously, input the third intermediate input feature map into a third upsampling layer for 2x upsampling to obtain an eighth intermediate feature map. Then, input the eighth intermediate feature map into a dimensionality reduction layer for dimensionality reduction to obtain a fourth intermediate depth prediction result.

[0026] Step S4.5: splicing the first intermediate depth prediction result, the second intermediate depth prediction result, the third intermediate depth prediction result and the fourth intermediate depth prediction result, and inputting the concatenated result into a third convolutional layer for fusion to obtain a predicted depth map.

[0027] Preferably, the random activation method includes:

[0028] Solution 1: Randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both preset values, to obtain a masked reference depth map;

[0029] Or scheme 2: randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both random values, to obtain a masked reference depth map;

[0030] Alternatively, solution three: use the target detection model to find the target box in the image of the reference ground truth depth map, select all points within the target box for activation, and obtain a masked reference depth map;

[0031] Alternatively, solution 4 is to use the target detection model to find the target box in the image of the reference ground truth depth map, and then randomly select a point in each target box. With this point as the center, select points within the range of height p above and below it and width q to the left and right of it for activation, where p and q are both random values, to obtain the masked reference depth map.

[0032] Alternatively, solution five: using a salt and pepper noise model to randomly activate the points in the point cloud of the reference ground truth depth map to obtain a masked reference depth map;

[0033] Or scheme six: randomly place rectangular boxes in the reference true value depth map, the number, position and size of the rectangular boxes are randomly set, select all points within the rectangular boxes for activation, and obtain the mask reference depth map.

[0034] Preferably, the loss function includes a logarithmic scale invariant loss function, the formula is

[0035]

[0036] d=log(y pred )-log(y gt )

[0037] Among them, α and λ are hyperparameters, y pred is the prediction result of the model, y gt is the true value input to the model, d is the logarithmic difference between the predicted result and the true value, L is the final calculated loss value, and N is the total number of point clouds in the training sample.

[0038] In a second aspect, the present invention provides a depth estimation model training system based on random activation, comprising:

[0039] Data acquisition module: uses cameras and lidar to collect image data and point cloud data of road scenes respectively;

[0040] Data fusion module: projects the point cloud onto the image to obtain a fusion map based on the intrinsic and extrinsic parameters of the camera and the lidar, and selects the fusion map with a high degree of matching as the reference true value depth map;

[0041] Training preparation module: divides the image data into a training set and a test set, and uses the images in the training set as training images;

[0042] Model training module: includes a depth prediction unit, a random activation unit, a loss calculation unit and a convergence judgment unit; the depth prediction unit obtains a predicted depth map through calculation based on the training images input in sequence; the random activation unit masks the reference true depth map through a random activation method to obtain a masked reference depth map; the loss calculation unit calculates the loss value of the model through a loss function based on the masked reference depth map and the predicted depth map; the convergence judgment unit determines whether the model converges. If not, the model parameters are updated using the loss value through the back propagation algorithm. If converged, the training is completed and the trained depth estimation model is obtained.

[0043] Preferably, the model training module further comprises a data enhancement unit, configured to perform data enhancement on the images in the training set to obtain training images;

[0044] The data enhancement includes horizontally flipping the image according to a preset probability, and / or adjusting the brightness, and / or adjusting the contrast, and / or adjusting the color.

[0045] Preferably, the depth estimation model adopts an encoder-decoder network structure, the encoder part includes a dense feature extractor, and the decoder part includes a context information extractor, three local plane guidance layers, a dimensionality reduction layer, three convolution layers, three upsampling layers and five downsampling layers;

[0046] The context information extractor is a spatial pyramid pooling layer based on void convolution; the three local plane guiding layers are respectively the first, second and third local plane guiding layers, each local plane guiding layer includes a dimensionality reduction link, a unit normal vector calculation link, an activation function link and a locally defined depth calculation link, which is used to guide low-resolution features to full resolution, and then the local plane guiding layers are connected together, and the final depth prediction is performed through the last convolution layer; the three convolution layers are respectively the first, second and third convolution layers; the three upsampling layers are respectively the first, second and third upsampling layers; the five downsampling layers are respectively the first, second, third, fourth and fifth downsampling layers.

[0047] Preferably, the calculation process to obtain the predicted depth map includes:

[0048] Inputting the training images into the dense feature extractor of the depth estimation model to be trained in sequence to obtain original feature maps, and then the original feature maps are respectively input into the first, second and third downsampling layers for 8x, 4x and 2x downsampling to obtain first, second and third intermediate feature maps;

[0049] Inputting the first intermediate feature map into the context information extractor to obtain a first intermediate input feature map containing multi-scale semantic information; then inputting the first intermediate input feature map into the first local plane guide layer to obtain a first intermediate depth prediction result, and at the same time inputting the first intermediate input feature map into the first upsampling layer for 2x upsampling to obtain a fourth intermediate feature map; then inputting the first intermediate depth prediction result into the fourth downsampling layer for 4x downsampling to obtain a fifth intermediate feature map; then concatenating the second intermediate feature map, the fourth intermediate feature map, and the fifth intermediate feature map and inputting the convolutional layer for fusion to obtain a second intermediate input feature map;

[0050] Input the second intermediate input feature map into the second local plane guide layer to obtain a second intermediate depth prediction result, and input the second intermediate input feature map into the second upsampling layer for 2x upsampling to obtain a sixth intermediate feature map; then input the second intermediate depth prediction result into the fifth downsampling layer for 2x downsampling to obtain a seventh intermediate feature map; then concatenate the third intermediate feature map, the sixth intermediate feature map, and the seventh intermediate feature map, and input the concatenated convolutional layer for fusion to obtain a third intermediate input feature map;

[0051] Inputting the third intermediate input feature map into a third local plane guiding layer to obtain a third intermediate depth prediction result, and simultaneously inputting the third intermediate input feature map into a third upsampling layer for 2x upsampling to obtain an eighth intermediate feature map; then inputting the eighth intermediate feature map into a dimensionality reduction layer for dimensionality reduction to obtain a fourth intermediate depth prediction result;

[0052] The first intermediate depth prediction result, the second intermediate depth prediction result, the third intermediate depth prediction result, and the fourth intermediate depth prediction result are spliced and input into the third convolutional layer for fusion to obtain a predicted depth map.

[0053] Preferably, the random activation method includes:

[0054] Solution 1: Randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both preset values, to obtain a masked reference depth map;

[0055] Or scheme 2: randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both random values, to obtain a masked reference depth map;

[0056] Alternatively, solution three: use the target detection model to find the target box in the image of the reference ground truth depth map, select all points within the target box for activation, and obtain a masked reference depth map;

[0057] Alternatively, solution 4 is to use the target detection model to find the target box in the image of the reference ground truth depth map, and then randomly select a point in each target box. With this point as the center, select points within the range of height p above and below it and width q to the left and right of it for activation, where p and q are both random values, to obtain the masked reference depth map.

[0058] Alternatively, solution five: using a salt and pepper noise model to randomly activate the points in the point cloud of the reference ground truth depth map to obtain a masked reference depth map;

[0059] Or scheme six: randomly place rectangular boxes in the reference true value depth map, the number, position and size of the rectangular boxes are randomly set, select all points within the rectangular boxes for activation, and obtain the mask reference depth map.

[0060] Preferably, the loss function includes a logarithmic scale invariant loss function, the formula is

[0061]

[0062] d=log(ypred )-log(y gt )

[0063] Among them, α and λ are hyperparameters, y pred is the prediction result of the model, y gt is the true value input to the model, d is the logarithmic difference between the predicted result and the true value, L is the final calculated loss value, and N is the total number of point clouds in the training sample.

[0064] In a third aspect, the present invention provides an evaluation method for a depth estimation model based on random activation, comprising: evaluation scheme 1: setting the depth range to be evaluated to 0 to a first preset distance; sequentially inputting images in a test set into the depth estimation model to be evaluated, and obtaining a predicted depth map through computational processing; calculating the error or accuracy of the predicted depth map and the reference true depth map using an evaluation index for the entire image area; and evaluating the performance of the depth estimation model based on the error or accuracy;

[0065] Or evaluation scheme two: set the depth range to be evaluated to 0 to a first preset distance; input the images in the test set into the depth estimation model to be evaluated in sequence, and obtain a predicted depth map after calculation and processing; use the target detection model to detect the image, and if the detection result contains the target object, use the evaluation index to calculate the error or accuracy only for the area where the target object is located in the predicted depth map and the reference true depth map; if the detection result does not contain the target object, use the evaluation index to calculate the error or accuracy for the entire image area of the predicted depth map and the reference true depth map; evaluate the performance of the depth estimation model based on the error or accuracy;

[0066] Or evaluation scheme three: set the depth range to be evaluated to 0 to a second preset distance; input the images in the test set into the depth estimation model to be evaluated in sequence, and obtain a predicted depth map after calculation and processing; use the target detection model to detect the image, and if the detection result contains the target object, calculate the error or accuracy using the evaluation index only for the area where the target object is located in the predicted depth map and the reference true depth map; if the detection result does not contain the target object, calculate the error or accuracy using the evaluation index for the entire image area of the predicted depth map and the reference true depth map; evaluate the performance of the depth estimation model based on the error or accuracy;

[0067] Preferably, the second preset distance is smaller than the first preset distance.

[0068] Preferably, the method for obtaining the reference true depth map includes:

[0069] Use cameras and lidar to collect image data and point cloud data of road scenes respectively;

[0070] According to the intrinsic and extrinsic parameters of the camera and the lidar, the point cloud is projected onto the image to obtain a fusion map, and a highly matched fusion map is selected as the reference true value depth map.

[0071] Preferably, the evaluation indicators include:

[0072] Evaluation indicator 1, absolute relative error:

[0073]

[0074] Or evaluation indicator two, square relative error:

[0075]

[0076] Or evaluation indicator three, root mean square error:

[0077]

[0078] Or evaluation indicator four, logarithmic root mean square error:

[0079]

[0080] Or evaluation indicator five, logarithmic scale invariant error:

[0081]

[0082] Or the sixth evaluation indicator, accuracy when the threshold is 1.25:

[0083]

[0084] Or evaluation indicator seven, the threshold is 1.25 2 The accuracy of:

[0085]

[0086] Or evaluation indicator eight, threshold is 1.25 3 The accuracy of:

[0087]

[0088] Among them, y pred is the prediction result of the model, y gtis the true value input to the model, N is the number of valid points involved in the calculation, log(·) means taking the logarithm, max(·) means taking the maximum value, and ∑(·) means summing; II(x) in evaluation indicators six, seven, and eight is an indicator function, that is, when x meets the conditions, the function returns 1, otherwise it returns 0; evaluation indicators one, two, three, four, and five are error indicators, and evaluation indicators six, seven, and eight are accuracy indicators. AbsRel, SqRel, RMSE, LogRMSE, SILog, a1, a2, and a3 represent absolute relative error, square relative error, root mean square error, logarithmic root mean square error, logarithmic scale invariant error, accuracy when the threshold is 1.25, and threshold is 1.25, respectively. 2 When the precision and threshold are 1.25 3 time accuracy.

[0089] Compared with the prior art, the present invention has the following beneficial effects:

[0090] 1. The present invention uses less point cloud data to train a model with accuracy that approaches or even exceeds that of full point cloud training.

[0091] 2. The present invention can improve the model accuracy problem caused by point cloud sparsity by using less point cloud training data, and since the data during training is reduced, the amount of calculation is reduced, thereby reducing the training time of the model.

[0092] 3. The training method proposed in the present invention avoids the increase of hardware costs while achieving ideal results, which is conducive to the implementation of business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0094] Figure 1 A schematic diagram of a workflow of an embodiment of the present invention;

[0095] Figure 2 Schematic diagram of the depth estimation model in the present invention;

[0096] Figure 3 for Figure 1 Schematic diagram of the process structure of step S4;

[0097] Figure 4 Schematic diagram of six schemes of the random activation method of the present invention;

[0098] Figure 5 A schematic diagram of the structure of a depth estimation model training system according to an embodiment of the present invention;

[0099] Figure 6This is a structural diagram of a depth estimation model training system according to another embodiment of the present invention. DETAILED DESCRIPTION

[0100] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0101] According to the present invention, a supervised training method for a depth estimation model based on random activation point cloud is provided. Figure 1 Shown, including:

[0102] Step S1: Collect image data and point cloud data of the road scene using a camera and a lidar respectively.

[0103] In one embodiment, the number of the laser radar used to collect road scene point cloud data is one.

[0104] Step S2: Based on the intrinsic and extrinsic parameters of the camera and the lidar, the point cloud is projected onto the image to obtain a fusion map, and a highly matched fusion map is selected as the reference truth depth map. Specifically, the point cloud is projected onto the camera imaging plane using the intrinsic and extrinsic parameters of the radar and the camera to obtain a fusion map. A highly matched fusion map is selected from multiple fusion maps as the reference truth depth map. High matching means that the two-dimensional points projected onto the image by the point cloud correspond precisely to the objects in the image in position, and the average deviation between the two is less than a preset threshold. The highly matched fusion map is used as the reference truth depth map to ensure that the two-dimensional points projected from the point cloud and the objects in the image are perfectly matched.

[0105] In one embodiment, the preset threshold of step S2 is 90%. When the positional correspondence between the two-dimensional points projected onto the image and the objects in the image reaches more than 90%, a high degree of match is achieved.

[0106] Step S3: Divide the image data into a training set and a test set, and use the images in the training set as training images.

[0107] In one embodiment, the method further includes performing data augmentation on the images in the training set to obtain training images; the data augmentation includes horizontally flipping the images according to a preset probability, and / or adjusting the image brightness, and / or adjusting the image contrast, and / or adjusting the image color, so as to increase the differences between different batches of images used for training and enhance the generalization ability of the model.

[0108] In one embodiment, the structure of the depth estimation model is as follows Figure 2As shown, the depth estimation model adopts an encoder-decoder network structure, the encoder part includes a dense feature extractor, and the decoder part includes a context information extractor, three local plane guidance layers, a dimensionality reduction layer, three convolution layers, three upsampling layers and five downsampling layers; the context information extractor is a spatial pyramid pooling layer based on void convolution; the three local plane guidance layers are the first, second and third local plane guidance layers, each local plane guidance layer includes a dimensionality reduction link, a unit normal vector calculation link, an activation function link and a locally defined depth calculation link, which is used to guide low-resolution features to full resolution, and then the local plane guidance layers are connected together, and the final depth prediction is performed through the last convolution layer; the three convolution layers are the first, second and third convolution layers, respectively; the three upsampling layers are the first, second and third upsampling layers, respectively; the five downsampling layers are the first, second, third, fourth and fifth downsampling layers, respectively.

[0109] Step S4: inputting the training images into the depth estimation model to be trained in sequence, and obtaining a predicted depth map through calculation and processing.

[0110] In one embodiment, Figure 3 As shown, step S4 includes:

[0111] Step S4.1: The training images are sequentially input into the dense feature extractor of the depth estimation model to be trained to obtain original feature maps. The original feature maps are then input into the first, second, and third downsampling layers for 8x, 4x, and 2x downsampling, respectively, to obtain first, second, and third intermediate features.

[0112] Step S4.2: Input the first intermediate feature map into the context information extractor to obtain a first intermediate input feature map containing multi-scale semantic information; then input the first intermediate input feature map into the first local plane guide layer to obtain a first intermediate depth prediction result, and at the same time input the first intermediate input feature map into the first upsampling layer for 2x upsampling to obtain a fourth intermediate feature map; then input the first intermediate depth prediction result into the fourth downsampling layer for 4x downsampling to obtain a fifth intermediate feature map; then concatenate the second intermediate feature map, the fourth intermediate feature map, and the fifth intermediate feature map and input the concatenated feature map into the first convolutional layer for fusion to obtain a second intermediate input feature map;

[0113] Step S4.3: Input the second intermediate input feature map into the second local plane guide layer to obtain a second intermediate depth prediction result, and input the second intermediate input feature map into the second upsampling layer for 2x upsampling to obtain a sixth intermediate feature map; then input the second intermediate depth prediction result into the fifth downsampling layer for 2x downsampling to obtain a seventh intermediate feature map; then concatenate the third intermediate feature map, the sixth intermediate feature map, and the seventh intermediate feature map, and input the concatenated features into the second convolutional layer for fusion to obtain a third intermediate input feature map;

[0114] Step S4.4: Input the third intermediate input feature map into a third local plane guiding layer to obtain a third intermediate depth prediction result. Simultaneously, input the third intermediate input feature map into a third upsampling layer for 2x upsampling to obtain an eighth intermediate feature map. Then, input the eighth intermediate feature map into a dimensionality reduction layer for dimensionality reduction to obtain a fourth intermediate depth prediction result.

[0115] Step S4.5: splicing the first intermediate depth prediction result, the second intermediate depth prediction result, the third intermediate depth prediction result and the fourth intermediate depth prediction result, and inputting the concatenated result into a third convolutional layer for fusion to obtain a predicted depth map.

[0116] Step S5: performing mask processing on the reference true depth map using a random activation method to obtain a masked reference depth map.

[0117] In one embodiment, Figure 4 As shown, the random activation method includes:

[0118] Solution 1: Randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both preset values, to obtain a masked reference depth map;

[0119] Or scheme 2: randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both random values, to obtain a masked reference depth map;

[0120] Alternatively, solution three: use the target detection model to find the target box in the image of the reference ground truth depth map, select all points within the target box for activation, and obtain a masked reference depth map;

[0121] Alternatively, solution 4 is to use the target detection model to find the target box in the image of the reference ground truth depth map, and then randomly select a point in each target box. With this point as the center, select points within the range of height p above and below it and width q to the left and right of it for activation, where p and q are both random values, to obtain the masked reference depth map.

[0122] Alternatively, solution five: using a salt and pepper noise model to randomly activate the points in the point cloud of the reference ground truth depth map to obtain a masked reference depth map;

[0123] Or scheme six: randomly place rectangular boxes in the reference true value depth map, the number, position and size of the rectangular boxes are randomly set, select all points within the rectangular boxes for activation, and obtain the mask reference depth map.

[0124] Step S6: Calculate the loss value of the model using a loss function according to the mask reference depth map and the predicted depth map.

[0125] Specifically, the loss function adopts a logarithmic scale invariant loss function, and the formula is:

[0126]

[0127] d=log(y pred )-log(y gt )

[0128] Among them, α and λ are hyperparameters. A higher λ will make the model more focused on minimizing the variance of the loss. α is used to scale the loss value, making it easier for the model to converge to the optimal result. pred is the prediction result of the model, y gt is the true value input to the model, d is the logarithmic difference between the predicted result and the true value, L is the final calculated loss value, and N is the total number of point clouds in the training sample.

[0129] Step S7: Determine whether the model has converged. If not, update the model parameters using the loss value through a backpropagation algorithm. If converged, training is complete, and a trained depth estimation model is obtained. The method for updating the model parameters includes calculating the gradient of the training loss value with respect to each parameter in the model and updating the parameters according to the optimization algorithm.

[0130] In one embodiment, the optimization algorithm uses the Adam optimizer.

[0131] The present invention also provides a depth estimation model training system based on random activation. Those skilled in the art can implement the depth estimation model training system based on random activation by executing the step flow of the depth estimation model training method based on random activation, that is, the depth estimation model training method based on random activation can be understood as a preferred implementation of the depth estimation model training system based on random activation.

[0132] The present invention provides a depth estimation model training system based on random activation, such as Figure 5 Shown, including:

[0133] Data acquisition module: uses cameras and lidar to collect image data and point cloud data of road scenes respectively.

[0134] Data fusion module: Based on the intrinsic and extrinsic parameters of the camera and the lidar, the point cloud is projected onto the image to obtain a fusion map, and the highly matched fusion map is selected as the reference true value depth map.

[0135] Training preparation module: divides the image data into a training set and a test set, and uses the images in the training set as training images.

[0136] Model training module: includes a depth prediction unit, a random activation unit, a loss calculation unit and a convergence judgment unit; the depth prediction unit obtains a predicted depth map based on the training images input in sequence; the random activation unit uses a random activation method to mask the reference true depth map to obtain a masked reference depth map; the loss calculation unit calculates the loss value of the model through a loss function based on the masked reference depth map and the predicted depth map; the convergence judgment unit determines whether the model converges. If not, the model parameters are updated using the loss value through the back propagation algorithm. If converged, the training is completed and the trained depth estimation model is obtained.

[0137] In one embodiment, Figure 6 As shown, the model training module also includes a data enhancement unit, which performs data enhancement on the images in the training set to obtain training images; the data enhancement includes horizontally flipping the images according to a preset probability, and / or brightness adjustment, and / or contrast adjustment, and / or color adjustment, so as to increase the differences between different batches of images used for training and enhance the generalization ability of the model.

[0138] In one embodiment, the depth estimation model adopts an encoder-decoder network structure; Figure 2 As shown, the encoder contains a dense feature extractor; the decoder contains a context information extractor, three local plane guidance layers, a dimensionality reduction layer, three convolutional layers, three upsampling layers and two downsampling layers;

[0139] The context information extractor is a spatial pyramid pooling layer based on void convolution; the three local plane guiding layers are respectively the first, second and third local plane guiding layers, each local plane guiding layer includes a dimensionality reduction link, a unit normal vector calculation link, an activation function link and a locally defined depth calculation link, which is used to guide low-resolution features to full resolution, and then the local plane guiding layers are connected together, and the final depth prediction is performed through the last convolution layer; the three convolution layers are respectively the first, second and third convolution layers; the three upsampling layers are respectively the first, second and third upsampling layers; the five downsampling layers are respectively the first, second, third, fourth and fifth downsampling layers.

[0140] Furthermore, the specific steps of the depth prediction unit, random activation unit, and loss calculation unit in the model training module are described as follows:

[0141] The method for the depth prediction unit in the model training module to obtain a predicted depth map according to the training images input sequentially includes:

[0142] First, the training images are sequentially input into the dense feature extractor of the depth estimation model to be trained to obtain the original feature maps. Then, the original feature maps are respectively input into the first, second, and third downsampling layers for 8x, 4x, and 2x downsampling to obtain the first, second, and third intermediate feature maps.

[0143] Then, the first intermediate feature map is input into the context information extractor to obtain a first intermediate input feature map containing multi-scale semantic information; then the first intermediate input feature map is input into the first local plane guide layer to obtain a first intermediate depth prediction result, and at the same time, the first intermediate input feature map is input into the first upsampling layer for 2x upsampling to obtain a fourth intermediate feature map; then the first intermediate depth prediction result is input into the fourth downsampling layer for 4x downsampling to obtain a fifth intermediate feature map; then the second intermediate feature map, the fourth intermediate feature map and the fifth intermediate feature map are spliced and input into the first convolution layer for fusion to obtain a second intermediate input feature map.

[0144] Next, the second intermediate input feature map is input into the second local plane guide layer to obtain a second intermediate depth prediction result, and at the same time, the second intermediate input feature map is input into the second upsampling layer for 2x upsampling to obtain a sixth intermediate feature map; then the second intermediate depth prediction result is input into the fifth downsampling layer for 2x downsampling to obtain a seventh intermediate feature map; then the third intermediate feature map, the sixth intermediate feature map and the seventh intermediate feature map are spliced and input into the second convolutional layer for fusion to obtain a third intermediate input feature map.

[0145] Next, the third intermediate input feature map is input into the third local plane guide layer to obtain a third intermediate depth prediction result. At the same time, the third intermediate input feature map is input into the third upsampling layer for 2x upsampling to obtain an eighth intermediate feature map; then the eighth intermediate feature map is input into the dimensionality reduction layer for dimensionality reduction to obtain a fourth intermediate depth prediction result.

[0146] Finally, the first intermediate depth prediction result, the second intermediate depth prediction result, the third intermediate depth prediction result, and the fourth intermediate depth prediction result are concatenated and input into the third convolutional layer for fusion to obtain a predicted depth map.

[0147] The random activation methods used by the random activation unit in the model training module include the following six schemes:

[0148] Solution 1: Randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both preset values, to obtain a masked reference depth map.

[0149] Or scheme 2: randomly select a point from the point cloud of the reference true depth map, take the point as the center, select points within the range of m height above and below it and n width to the left and right of it for activation, where m and n are both random values, and obtain the masked reference depth map.

[0150] Or solution three: use the target detection model to find the target box in the image of the reference true value depth map, select all points within the target box for activation, and obtain the mask reference depth map.

[0151] Or solution four: use the target detection model to find the target box in the image of the reference ground truth depth map, then randomly select a point in each target box, and with this point as the center, select points within the range of p heights above and below it and q widths to the left and right of it for activation, where p and q are both random values, to obtain a masked reference depth map.

[0152] Or scheme five: use the salt and pepper noise model to randomly activate the points of the point cloud in the reference true depth map to obtain a masked reference depth map.

[0153] Or scheme six: randomly place rectangular boxes in the reference true value depth map, the number, position and size of the rectangular boxes are randomly set, select all points within the rectangular boxes for activation, and obtain the mask reference depth map.

[0154] The loss function of the loss calculation unit in the model training module adopts the logarithmic scale invariant loss function, and the formula is

[0155]

[0156] d=log(y pred )-log(y gt )

[0157] Among them, α and λ are hyperparameters. A higher λ will make the model more focused on minimizing the variance of the loss. α is used to scale the loss value, making it easier for the model to converge to the optimal result. pred is the prediction result of the model, y gt is the true value input to the model, d is the logarithmic difference between the predicted result and the true value, L is the final calculated loss value, and N is the total number of point clouds in the training sample.

[0158] According to the present invention, a method for evaluating a depth estimation model based on random activation is provided, which is used to evaluate a depth estimation model trained using the depth estimation model training method based on random activation in the present invention, or to evaluate a depth estimation model trained by the depth estimation model training system based on random activation in the present invention.

[0159] Specifically, there are three options:

[0160] Evaluation Scheme 1: Set the depth range to be evaluated to a value between 0 and a first preset distance; sequentially input the images in the test set into the depth estimation model to be evaluated, and calculate and process them to obtain a predicted depth map; use the evaluation metric to calculate the error or accuracy of the predicted depth map and the ground truth depth map for the entire image area; and evaluate the performance of the depth estimation model based on the error or accuracy.

[0161] In one embodiment, the first preset distance is 80 meters.

[0162] Or evaluation scheme two: set the depth range to be evaluated to 0 to a first preset distance; input the images in the test set into the depth estimation model to be evaluated in sequence, and obtain a predicted depth map after calculation and processing; use the target detection model to detect the image, and if the detection result contains the target object, use the evaluation index to calculate the error or accuracy only for the area where the target object is located in the predicted depth map and the reference true depth map; if the detection result does not contain the target object, use the evaluation index to calculate the error or accuracy for the entire image area of the predicted depth map and the reference true depth map; evaluate the performance of the depth estimation model based on the error or accuracy;

[0163] In one embodiment, the second preset distance is 50 meters.

[0164] Or evaluation plan three: set the depth range to be evaluated to 0 to a second preset distance; input the images in the test set into the depth estimation model to be evaluated in turn, and obtain a predicted depth map after calculation and processing; use the target detection model to detect the image. If the detection result contains the target object, the error or accuracy is calculated using the evaluation index only for the area where the target object is located in the predicted depth map and the baseline true depth map. If the detection result does not contain the target object, the error or accuracy is calculated using the evaluation index for the entire image area of the predicted depth map and the baseline true depth map; based on the error or accuracy, the performance of the depth estimation model is evaluated.

[0165] In one embodiment, the third preset distance is 30 meters.

[0166] The second preset distance is less than the first preset distance. The method for obtaining a baseline true depth map includes: collecting image data and point cloud data of a road scene using a camera and a lidar, respectively; projecting the point cloud onto the image to obtain a fused image based on intrinsic and extrinsic parameters of the camera and the lidar, and selecting a highly matched fused image as the baseline true depth map.

[0167] The evaluation indicators include the following six indicators:

[0168] Evaluation indicator 1, absolute relative error:

[0169]

[0170] Or evaluation indicator two, square relative error:

[0171]

[0172] Or evaluation indicator three, root mean square error:

[0173]

[0174] Or evaluation indicator four, logarithmic root mean square error:

[0175]

[0176] Or evaluation metric five, logarithmic scale invariant error:

[0177]

[0178] Or the sixth evaluation indicator, accuracy when the threshold is 1.25:

[0179]

[0180] Or evaluation indicator seven, the threshold is 1.25 2 The accuracy of:

[0181]

[0182] Or evaluation indicator eight, threshold is 1.25 3 The accuracy of:

[0183]

[0184] Among them, y pred is the prediction result of the model, y gt is the true value input to the model, N is the number of valid points involved in the calculation, log(·) means taking the logarithm, max(·) means taking the maximum value, ∑(·) means summing; is an indicator function, that is, when x meets the conditions, the function returns 1, otherwise it returns 0; evaluation indicators 1, 2, 3, 4, and 5 are error indicators, the lower the better, and evaluation indicators 6, 7, and 8 are accuracy indicators, the higher the better. AbsRel, SqRel, RMSE, LogRMSE, SILog, a1, a2, and a3 represent absolute relative error, square relative error, root mean square error, logarithmic root mean square error, logarithmic scale invariant error, accuracy when the threshold is 1.25, and threshold 1.25, respectively. 2 When the precision and threshold are 1.25 3 time accuracy.

[0185] Furthermore, in order to compare the differences between the random activation training method proposed in this patent and the original training method (Baseline), the following evaluation schemes are used:

[0186] Evaluation Option 1: Evaluate the full point cloud. The results of this evaluation method are as follows:

[0187]

[0188] As can be seen, under this evaluation scheme, both Scheme 1 and Scheme 5 achieved the best results in four indicators. Overall, the random activation scheme outperforms the baseline.

[0189] Evaluation Option 2: Because the Baseline solution uses the full point cloud for training, while the method proposed in this patent uses only a portion of the point cloud for training, using the full point cloud for both approaches would be unfair. We need to consider adjusting the evaluation method based on the application scenario. In autonomous driving scenarios, we only focus on foreground objects, such as pedestrians, vehicles, and obstacles. Therefore, it is more fair to evaluate only the point cloud in the target area. When the target of interest is absent from the image, we evaluate the full point cloud.

[0190] The target objects in this evaluation method are found by the target detection algorithm. The results of this evaluation method are as follows:

[0191]

[0192]

[0193] As can be seen, under this evaluation method, all indicators of Schemes 3 and 6 are better than the baseline, and Scheme 3 achieves the best performance on all eight indicators, which is expected because it has the same data domain as the training set. The second best solution is Scheme 6. Both solutions use rectangular boxes for activation, which has certain advantages for our scenario. Overall, all solutions are better than the baseline.

[0194] Evaluation Plan 3: Going a step further, the default evaluation depth range is 0-80 meters, but our business scenario does not require detection at such a long distance, so based on Evaluation Plan 2, the depth is limited to 0-30 meters. The results of this evaluation method are as follows:

[0195]

[0196] It can be seen that under this evaluation method, all indicators of Schemes 3, 4, and 6 are better than the Baseline, and Scheme 3 is the best scheme.

[0197] Overall, for the application scenarios in the field of autonomous driving, Solution 3 is the optimal cloud activation solution, which achieves the best evaluation effect by generating a mask reference depth map through the target frame.

[0198] The present invention makes improvements at the software level, that is, there is no need to add hardware to generate better supervision data. Instead, it makes adjustments to the training strategy of the depth estimation model and proposes a depth estimation model training method, system and evaluation method based on random activation. The model effect of training with less point cloud data is close to or even exceeds that of full point cloud training.

[0199] This method involves training and testing phases. It uses the KITTI autonomous driving dataset as test data. The encoder network extracts multi-scale features from the original image, and the decoder network reconstructs the depth map from the feature map. The focus is on calculating the loss function and evaluating the model's performance using evaluation metrics.

[0200] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0201] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A depth estimation model training method based on random activation, characterized in that: include: Step S1: using a camera and a lidar to collect image data and point cloud data of a road scene respectively; The number of the laser radar used to collect road scene point cloud data is one; Step S2: Projecting the point cloud data onto the image to obtain a fused image based on the intrinsic and extrinsic parameters of the camera and the lidar, and selecting a highly matched fused image as a reference ground truth depth map; highly matched means that the two-dimensional points projected from the point cloud onto the image precisely correspond in position to the objects in the image, and the average deviation between the two is less than a preset threshold; Step S3: dividing the image data into a training set and a test set, and using the images in the training set as training images; Step S4: inputting the training images into the depth estimation model to be trained in sequence, and obtaining a predicted depth map through calculation and processing; Step S5: performing mask processing on the reference true depth map by a random activation method to obtain a masked reference depth map; Step S6: Calculate the loss value of the model through the loss function according to the mask reference depth map and the predicted depth map; the loss function includes a logarithmic scale invariant loss function, the formula is d=log(y pred )-log(y gt ) Among them, α and λ are hyperparameters, y pred is the prediction result of the model, y gt is the true value input to the model, d is the logarithmic difference between the predicted result and the true value, L is the final calculated loss value, and N is the total number of point clouds in the training sample; Step S7: Determine whether the model has converged. If not, update the model parameters using the loss value through the back propagation algorithm. If converged, the training is completed and a trained depth estimation model is obtained.

2. The depth estimation model training method based on random activation according to claim 1, characterized in that The step S3 further includes: performing data enhancement on the images in the training set to obtain training images; The data enhancement includes horizontally flipping the image according to a preset probability, and / or adjusting the brightness, and / or adjusting the contrast, and / or adjusting the color.

3. The depth estimation model training method based on random activation according to claim 1, characterized in that The depth estimation model adopts an encoder-decoder network structure. The encoder part includes a dense feature extractor, and the decoder part includes a context information extractor, three local plane guidance layers, a dimensionality reduction layer, three convolutional layers, three upsampling layers and five downsampling layers. The context information extractor is a spatial pyramid pooling layer based on dilated convolution; the three local plane guiding layers are respectively the first, second and third local plane guiding layers, each of which includes a dimensionality reduction step, a unit normal vector calculation step, an activation function step and a locally defined depth calculation step, and is used to guide low-resolution features to full resolution. The local plane guiding layers are then connected together and the final depth prediction is performed through the final convolutional layer; The three convolutional layers are respectively the first, second and third convolutional layers; the three upsampling layers are respectively the first, second and third upsampling layers; the five downsampling layers are respectively the first, second, third, fourth and fifth downsampling layers.

4. The depth estimation model training method based on random activation according to claim 3, characterized in that Step S4 includes: Step S4.1: The training images are sequentially input into the dense feature extractor of the depth estimation model to be trained to obtain original feature maps. The original feature maps are then input into the first, second, and third downsampling layers for 8x, 4x, and 2x downsampling, respectively, to obtain first, second, and third intermediate feature maps. Step S4.2: Input the first intermediate feature map into the context information extractor to obtain a first intermediate input feature map containing multi-scale semantic information; then input the first intermediate input feature map into the first local plane guide layer to obtain a first intermediate depth prediction result, and at the same time input the first intermediate input feature map into the first upsampling layer for 2x upsampling to obtain a fourth intermediate feature map; then input the first intermediate depth prediction result into the fourth downsampling layer for 4x downsampling to obtain a fifth intermediate feature map; then concatenate the second intermediate feature map, the fourth intermediate feature map, and the fifth intermediate feature map and input the concatenated feature map into the first convolutional layer for fusion to obtain a second intermediate input feature map; Step S4.3: Input the second intermediate input feature map into the second local plane guide layer to obtain a second intermediate depth prediction result, and input the second intermediate input feature map into the second upsampling layer for 2x upsampling to obtain a sixth intermediate feature map; then input the second intermediate depth prediction result into the fifth downsampling layer for 2x downsampling to obtain a seventh intermediate feature map; then concatenate the third intermediate feature map, the sixth intermediate feature map, and the seventh intermediate feature map, and input the concatenated features into the second convolutional layer for fusion to obtain a third intermediate input feature map; Step S4.4: Input the third intermediate input feature map into a third local plane guiding layer to obtain a third intermediate depth prediction result. Simultaneously, input the third intermediate input feature map into a third upsampling layer for 2x upsampling to obtain an eighth intermediate feature map. Then, input the eighth intermediate feature map into a dimensionality reduction layer for dimensionality reduction to obtain a fourth intermediate depth prediction result. Step S4.5: splicing the first intermediate depth prediction result, the second intermediate depth prediction result, the third intermediate depth prediction result and the fourth intermediate depth prediction result, and inputting the concatenated result into a third convolutional layer for fusion to obtain a predicted depth map.

5. The depth estimation model training method based on random activation according to claim 1, characterized in that The random activation method comprises: Solution 1: Randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both preset values, to obtain a masked reference depth map; Or scheme 2: randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both random values, to obtain a masked reference depth map; Alternatively, solution three: use the target detection model to find the target box in the image of the reference ground truth depth map, select all points within the target box for activation, and obtain a masked reference depth map; Alternatively, solution 4 is to use the target detection model to find the target box in the image of the reference ground truth depth map, and then randomly select a point in each target box. With this point as the center, select points within the range of height p above and below it and width q to the left and right of it for activation, where p and q are both random values, to obtain the masked reference depth map. Alternatively, solution five: using a salt and pepper noise model to randomly activate the points in the point cloud of the reference ground truth depth map to obtain a masked reference depth map; Or scheme six: randomly place rectangular boxes in the reference true value depth map, the number, position and size of the rectangular boxes are randomly set, select all points within the rectangular boxes for activation, and obtain the mask reference depth map.

6. A depth estimation model training system based on random activation, characterized in that include: Data acquisition module: uses cameras and lidar to collect image data and point cloud data of road scenes respectively; The number of the laser radar used to collect road scene point cloud data is one; Data fusion module: Based on the intrinsic and extrinsic parameters of the camera and the lidar, the point cloud is projected onto the image to obtain a fused image, and the fused image with a high degree of matching is selected as the reference ground truth depth map; high matching means that the two-dimensional points projected from the point cloud onto the image precisely correspond in position to the objects in the image, and the average deviation between the two is less than a preset threshold; Training preparation module: divides the image data into a training set and a test set, and uses the images in the training set as training images; Model training module: includes a depth prediction unit, a random activation unit, a loss calculation unit and a convergence judgment unit; the depth prediction unit obtains a predicted depth map through calculation based on the training images input in sequence; the random activation unit masks the reference true depth map through a random activation method to obtain a masked reference depth map; the loss calculation unit calculates the loss value of the model through a loss function based on the masked reference depth map and the predicted depth map; the convergence judgment unit determines whether the model has converged. If not, the model parameters are updated using the loss value through the back propagation algorithm. If converged, the training is completed to obtain a trained depth estimation model; The loss function includes a logarithmic scale invariant loss function, which is formulated as d=log(y pred )-log(y gt ) Among them, α and λ are hyperparameters, y pred is the prediction result of the model, y gt is the true value input to the model, d is the logarithmic difference between the predicted result and the true value, L is the final calculated loss value, and N is the total number of point clouds in the training sample.

7. The random activation-based depth estimation model training system according to claim 6, characterized in that The model training module further includes a data enhancement unit for performing data enhancement on the images in the training set to obtain training images; The data enhancement includes horizontally flipping the image according to a preset probability, and / or adjusting the brightness, and / or adjusting the contrast, and / or adjusting the color.

8. The random activation-based depth estimation model training system according to claim 6, characterized in that The depth estimation model adopts an encoder-decoder network structure. The encoder part includes a dense feature extractor, and the decoder part includes a context information extractor, three local plane guidance layers, a dimensionality reduction layer, three convolutional layers, three upsampling layers and five downsampling layers. The context information extractor is a spatial pyramid pooling layer based on dilated convolution; the three local plane guiding layers are respectively the first, second and third local plane guiding layers, each of which includes a dimensionality reduction step, a unit normal vector calculation step, an activation function step and a locally defined depth calculation step, and is used to guide low-resolution features to full resolution. The local plane guiding layers are then connected together and the final depth prediction is performed through the final convolutional layer; The three convolutional layers are respectively the first, second and third convolutional layers; the three upsampling layers are respectively the first, second and third upsampling layers; the five downsampling layers are respectively the first, second, third, fourth and fifth downsampling layers.

9. The random activation-based depth estimation model training system according to claim 8, characterized in that The calculation process to obtain the predicted depth map includes: Inputting the training images into the dense feature extractor of the depth estimation model to be trained in sequence to obtain original feature maps, and then the original feature maps are respectively input into the first, second and third downsampling layers for 8x, 4x and 2x downsampling to obtain first, second and third intermediate feature maps; Inputting the first intermediate feature map into the context information extractor to obtain a first intermediate input feature map containing multi-scale semantic information; then inputting the first intermediate input feature map into the first local plane guide layer to obtain a first intermediate depth prediction result, and at the same time inputting the first intermediate input feature map into the first upsampling layer for 2x upsampling to obtain a fourth intermediate feature map; then inputting the first intermediate depth prediction result into the fourth downsampling layer for 4x downsampling to obtain a fifth intermediate feature map; then concatenating the second intermediate feature map, the fourth intermediate feature map, and the fifth intermediate feature map and inputting the convolutional layer for fusion to obtain a second intermediate input feature map; Input the second intermediate input feature map into the second local plane guide layer to obtain a second intermediate depth prediction result, and input the second intermediate input feature map into the second upsampling layer for 2x upsampling to obtain a sixth intermediate feature map; then input the second intermediate depth prediction result into the fifth downsampling layer for 2x downsampling to obtain a seventh intermediate feature map; then concatenate the third intermediate feature map, the sixth intermediate feature map, and the seventh intermediate feature map, and input the concatenated convolutional layer for fusion to obtain a third intermediate input feature map; Inputting the third intermediate input feature map into a third local plane guiding layer to obtain a third intermediate depth prediction result, and simultaneously inputting the third intermediate input feature map into a third upsampling layer for 2x upsampling to obtain an eighth intermediate feature map; then inputting the eighth intermediate feature map into a dimensionality reduction layer for dimensionality reduction to obtain a fourth intermediate depth prediction result; The first intermediate depth prediction result, the second intermediate depth prediction result, the third intermediate depth prediction result, and the fourth intermediate depth prediction result are spliced and input into the third convolutional layer for fusion to obtain a predicted depth map.

10. The random activation-based depth estimation model training system according to claim 8, characterized in that The random activation method comprises: Solution 1: Randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both preset values, to obtain a masked reference depth map; Or scheme 2: randomly select a point from the point cloud of the reference ground truth depth map, and with this point as the center, select points within a range of m height above and below it and n width to the left and right of it for activation, where m and n are both random values, to obtain a masked reference depth map; Alternatively, solution three: use the target detection model to find the target box in the image of the reference ground truth depth map, select all points within the target box for activation, and obtain a masked reference depth map; Alternatively, solution 4 is to use the target detection model to find the target box in the image of the reference ground truth depth map, and then randomly select a point in each target box. With this point as the center, select points within the range of height p above and below it and width q to the left and right of it for activation, where p and q are both random values, to obtain the masked reference depth map. Alternatively, solution five: using a salt and pepper noise model to randomly activate the points in the point cloud of the reference ground truth depth map to obtain a masked reference depth map; Or scheme six: randomly place rectangular boxes in the reference true value depth map, the number, position and size of the rectangular boxes are randomly set, select all points within the rectangular boxes for activation, and obtain the mask reference depth map.

11. A method for evaluating a depth estimation model based on random activation, characterized in that: A depth estimation model obtained by the random activation-based depth estimation model training method according to any one of claims 1 to 5, and an evaluation method thereof, comprising: evaluation scheme 1: setting the depth range to be evaluated to 0 to a first preset distance; inputting the images in the test set into the depth estimation model to be evaluated in sequence, and obtaining a predicted depth map through calculation and processing; calculating the error or accuracy of the predicted depth map and the reference true depth map using an evaluation index for the entire image area; and evaluating the performance of the depth estimation model based on the error or accuracy; Or evaluation scheme two: set the depth range to be evaluated to 0 to a first preset distance; input the images in the test set into the depth estimation model to be evaluated in sequence, and obtain a predicted depth map after calculation and processing; use the target detection model to detect the image, and if the detection result contains the target object, use the evaluation index to calculate the error or accuracy only for the area where the target object is located in the predicted depth map and the reference true depth map; if the detection result does not contain the target object, use the evaluation index to calculate the error or accuracy for the entire image area of the predicted depth map and the reference true depth map; evaluate the performance of the depth estimation model based on the error or accuracy; Or evaluation plan three: set the depth range to be evaluated to 0 to a second preset distance; input the images in the test set into the depth estimation model to be evaluated in turn, and obtain a predicted depth map after calculation and processing; use the target detection model to detect the image. If the detection result contains the target object, the error or accuracy is calculated using the evaluation index only for the area where the target object is located in the predicted depth map and the baseline true depth map. If the detection result does not contain the target object, the error or accuracy is calculated using the evaluation index for the entire image area of the predicted depth map and the baseline true depth map; based on the error or accuracy, the performance of the depth estimation model is evaluated.

12. The evaluation method of the depth estimation model based on random activation according to claim 11, characterized in that The second preset distance is smaller than the first preset distance.

13. The evaluation method of the depth estimation model based on random activation according to claim 11, characterized in that The method for obtaining the reference ground truth depth map includes: Use cameras and lidar to collect image data and point cloud data of road scenes respectively; According to the intrinsic and extrinsic parameters of the camera and the lidar, the point cloud is projected onto the image to obtain a fusion map, and a highly matched fusion map is selected as the reference true value depth map.

14. The evaluation method of the depth estimation model based on random activation according to claim 11, characterized in that The evaluation indicators include: Evaluation indicator 1, absolute relative error: Or evaluation indicator two, square relative error: Or evaluation indicator three, root mean square error: Or evaluation indicator four, logarithmic root mean square error: Or evaluation metric five, logarithmic scale invariant error: Or the sixth evaluation indicator, accuracy when the threshold is 1.25: Or evaluation indicator seven, the threshold is 1.25 2 The accuracy of: Or evaluation indicator eight, threshold is 1.25 3 The accuracy of: Among them, y pred is the prediction result of the model, y gt is the true value input to the model, N is the number of valid points involved in the calculation, log(·) means taking the logarithm, max(·) means taking the maximum value, ∑(·) means summing; is an indicator function, that is, when x meets the conditions, the function returns 1, otherwise it returns 0; evaluation indicators one, two, three, four, and five are error indicators, evaluation indicators six, seven, and eight are precision indicators, AbsRel, SqRel, RMSE, LogRMSE, SILog, a1, a2, and a3 represent absolute relative error, square relative error, root mean square error, logarithmic root mean square error, logarithmic scale invariant error, precision when the threshold is 1.25, and threshold 1.25, respectively. 2 When the precision and threshold are 1.25 3 time accuracy.

Images