Monocular depth estimation method and device based on joint loss function supervision

By adopting the joint loss function supervision method in monocular depth estimation, combining RGB images and point cloud data for alignment and double loss supervision, the problem of insufficient accuracy of monocular depth estimation is solved, and the accuracy and robustness of the model are improved. , suitable for depth image generation under complex environmental conditions.

CN119090933APending Publication Date: 2024-12-06GOSUNCN TECH GRP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411134184.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing monocular depth estimation methods have problems of insufficient accuracy and low accuracy in practical applications, especially during the data acquisition process, which is affected by the surrounding environment and acquisition methods, resulting in abnormal outliers affecting the model performance.

Method used

Using a joint loss function supervision method, a data set is constructed by aligning RGB images and point cloud data, a deep estimation network model is designed and a dual loss supervision mechanism is introduced, and the model parameters are adjusted to reduce the impact of abnormal outliers and improve the model prediction. accuracy.

Benefits of technology

It effectively reduces the interference of abnormal outliers in the dataset on the prediction results, improves the stability and robustness of the depth estimation network, improves the accuracy of the model prediction results, and can generate high-precision depth images under complex environmental conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090933A_ABST
    Figure CN119090933A_ABST
Patent Text Reader

Abstract

The invention discloses a monocular depth estimation method based on joint loss function supervision, and the method comprises the steps: obtaining an RGB image and point cloud data, and carrying out the alignment operation of the RGB image and the point cloud data, so as to construct a data set which comprises a training set, a verification set and a test set; constructing a depth estimation network model, and inputting the training set into the depth estimation network model for training to obtain model parameters and a depth prediction image; based on the model parameters, inputting the verification set into a depth estimation network model for evaluation so as to adjust the model parameters until the joint loss function of the verification set converges, and obtaining optimal model parameters; according to the optimal model parameters, inputting the test set into the depth estimation network model for performance evaluation to obtain a trained depth estimation network model; and inputting the RGB image into the trained depth estimation network model for prediction to obtain a depth image. The depth estimation network model is supervised through the joint loss function, and the accuracy of the model prediction result is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of depth estimation, and in particular to a monocular depth estimation method and device based on joint loss function supervision. Background Art

[0002] In recent years, monocular depth estimation has received extensive attention in the field of computer vision, especially in technical fields such as autonomous driving and robot navigation. Existing monocular depth estimation methods are mainly divided into traditional methods and deep learning-based methods. Traditional methods rely on optical geometric constraints, such as scale-invariant feature transformation and restoration of structure in motion, while deep learning-based methods use the powerful encoding ability and generalization performance of deep convolutional neural networks to perform depth estimation. Although these methods have achieved good results in some application scenarios, they still face many challenges and limitations in practical applications.

[0003] Among them, deep learning-based methods are mainly divided into supervised learning and self-supervised learning. Supervised learning uses the powerful encoding ability and generalization performance of deep convolutional neural networks to take color images and depth truth image pairs as the input of the network model and output the corresponding depth map in an end-to-end manner; self-supervised learning usually takes monocular camera image sequences as input, adopts the corresponding network architecture, and unifies the two tasks of depth map and pose estimation into one framework. The supervision information of self-supervised learning mainly comes from view synthesis.

[0004] However, for environments such as roadside scenes, cameras are usually fixed stationary on poles on the side of the road. Increasing the number of cameras to become binocular will increase costs, and reusing solutions in the field of autonomous driving is not feasible in terms of cost and configuration. Although self-supervised or unsupervised monocular depth estimation methods do not require a complex image acquisition process, most of them are only applicable to situations where parallax exists, and their accuracy is low. Especially in the data acquisition process, due to the influence of the surrounding environment and the acquisition method, there are often abnormal outliers in the collected data. These abnormal outliers will affect the performance of the model during the training process. How to reduce their impact and ensure the accuracy of the monocular depth estimation network to improve the accuracy of model predictions is a technical problem that urgently needs to be solved in this field. Summary of the invention

[0005] The present invention provides a monocular depth estimation method based on joint loss function supervision, so as to solve the technical problem that the monocular depth estimation network in the prior art is insufficient in precision, resulting in low accuracy of model prediction results.

[0006] According to one aspect of the present invention, the present invention provides a monocular depth estimation method based on joint loss function supervision, the method comprising:

[0007] Acquire an RGB image and point cloud data, and align the RGB image and the point cloud data to construct a data set, wherein the data set includes a training set, a validation set, and a test set;

[0008] Constructing a depth estimation network model, inputting the training set into the depth estimation network model for training, and obtaining model parameters and a depth prediction image;

[0009] Based on the model parameters, inputting the validation set into the depth estimation network model for performance evaluation to adjust the model parameters until the joint loss function of the validation set converges to obtain optimal model parameters;

[0010] According to the optimal model parameters, inputting the test set into the depth estimation network model for performance evaluation to obtain a trained depth estimation network model;

[0011] The RGB image is input into the trained depth estimation network model for prediction to obtain a depth image.

[0012] Furthermore, the RGB image and the point cloud data are obtained, and the RGB image and the point cloud data are aligned to construct a data set, wherein the data set includes a training set, a validation set, and a test set including:

[0013] The RGB image is captured by a monocular camera and the point cloud data is collected by a laser radar;

[0014] Based on the calibration algorithm, the laser radar coordinate system is converted to the monocular camera coordinate system to obtain the external parameter matrix, and the monocular camera coordinate system is converted to the RGB image coordinate system to obtain the internal parameter matrix;

[0015] Based on the external parameter matrix, the point cloud data is converted into the monocular camera coordinate system to obtain depth information;

[0016] Based on the intrinsic parameter matrix, the depth information is converted into the RGB image coordinate system to obtain a pixel position;

[0017] According to the pixel position, a depth value corresponding to the depth information is obtained, and the depth value is assigned to the pixel position corresponding to the RGB image to obtain a true depth image.

[0018] Furthermore, after obtaining the depth value corresponding to the depth information according to the pixel position and assigning the depth value to the pixel position corresponding to the RGB image to obtain the true depth image, the method further includes:

[0019] Constructing the data set according to the RGB image and the depth truth image;

[0020] The data set is randomly divided into the training set, the validation set and the test set.

[0021] Furthermore, the depth estimation network model is constructed, the training set is input into the depth estimation network model for training, and the model parameters and the depth prediction image are obtained, including:

[0022] Constructing the depth estimation network model, wherein the depth estimation network model includes an encoder and a decoder;

[0023] Preprocessing the training set to enhance image features of the training set;

[0024] Inputting the preprocessed training set into the encoder for downsampling to obtain coding layer features;

[0025] The coding layer features are input into the decoder for upsampling to obtain the model parameters and the depth prediction image.

[0026] Furthermore, the depth estimation network model is a U-shaped fully convolutional network, and the U-shaped fully convolutional network is the encoder and the decoder structure; wherein,

[0027] The encoder has five coding layers, the downsampling step length of each coding layer is 2, and the fifth coding layer is also provided with ASPP;

[0028] The decoder has four decoding layers in total, and the step length of upsampling performed by each decoding layer is also 2.

[0029] Furthermore, the depth prediction image includes an intermediate depth prediction image output by the second decoding layer and a final depth prediction image output by the fourth decoding layer, the intermediate depth prediction image is used to supervise the calculation of the loss function of the second decoding layer, and the final depth prediction image is used to supervise the calculation of the loss function of the fourth decoding layer.

[0030] Furthermore, the joint loss function includes:

[0031] L=γL1+(1-γ)L2

[0032] Among them, L represents the joint loss function, L1 represents the loss function of the decoding layer in the fourth layer, L2 represents the loss function of the decoding layer in the second layer, γ represents the balance factor, and the value range of γ is [0,1].

[0033] According to another aspect of the present invention, a monocular depth estimation device based on joint loss function supervision is provided, comprising:

[0034] A data set generation module, used for acquiring RGB images and point cloud data, and aligning the RGB images and the point cloud data to construct a data set, wherein the data set includes a training set, a validation set, and a test set;

[0035] A depth estimation network module is used to construct a depth estimation network model, input the training set into the depth estimation network model for training, and obtain model parameters and depth prediction images;

[0036] A joint loss function module, used to input the validation set into the depth estimation network model for performance evaluation based on the model parameters, so as to adjust the model parameters until the joint loss function of the validation set converges to obtain the optimal model parameters;

[0037] A model training and evaluation module, used to input the test set into the depth estimation network model for performance evaluation according to the optimal model parameters to obtain a trained depth estimation network model;

[0038] The depth image output module is used to input the RGB image into the trained depth estimation network model for prediction to obtain a depth image.

[0039] According to another aspect of the present invention, there is provided an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor;

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of claims 1 to 7.

[0041] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

[0042] Compared with the prior art, the monocular depth estimation method based on joint loss function supervision in the embodiment of the present invention has the following beneficial effects:

[0043] The embodiment of the present invention constructs a depth estimation network model, and uses a data set obtained by aligning the RGB image collected by a monocular camera and the point cloud data collected by a lidar for training and performance evaluation. A joint loss function is designed in the training process to achieve dual loss supervision of the learning process of the depth estimation network model, thereby ensuring that the model can be effectively trained and evaluated, and reducing the interference of abnormal outliers in the data set on the prediction results, so that the depth estimation network model can predict the RGB image under the optimal model parameters, obtain the optimal solution of the depth estimation network model, and obtain a high-precision depth image corresponding to the RGB image, so as to effectively cope with various complex environmental conditions, such as illumination changes, occlusion, etc., to ensure that the generated depth image can be directly applied to tasks such as target detection and distance measurement in roadside scenes, with strong practicality, improved the stability and robustness of the overall depth estimation network, and improved the accuracy of the model prediction results.

[0044] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0046] Figure 1 is a flow chart of a monocular depth estimation method based on joint loss function supervision provided by an embodiment of the present invention;

[0047] Figure 2 is a flowchart of step S10 in the monocular depth estimation method based on joint loss function supervision provided by an embodiment of the present invention;

[0048] Figure 3 is a flowchart of step S20 in the monocular depth estimation method based on joint loss function supervision provided by an embodiment of the present invention;

[0049] Figure 4 is a schematic diagram of a depth estimation network model in a monocular depth estimation method based on joint loss function supervision provided by an embodiment of the present invention;

[0050] Figure 5 is a schematic diagram of a monocular depth estimation device based on joint loss function supervision provided by an embodiment of the present invention;

[0051] Figure 6 Schematic diagram of an electronic device and a storage medium according to an embodiment of the present invention.

[0052] In the figure, 100, a monocular depth estimation device based on joint loss function supervision; 101, a data set generation module; 102, a depth estimation network module; 103, a joint loss function module; 104, a model training evaluation module; 105, a depth image output module; 200, an electronic device; 201, a computing unit; 202, a ROM; 203, a RAM; 204, a bus; 205, an I / O interface; 206, an input unit; 207, an output unit; 208, a storage unit; 209, a communication unit. DETAILED DESCRIPTION

[0053] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present invention. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0054] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features. Therefore, the definition of "first" and "second" features may explicitly or implicitly include one or more of the features, and in the description of the present invention, "at least" means one or more than one, unless otherwise clearly and specifically defined.

[0055] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0056] Terminology explanation section:

[0057] ASPP, short for Atrous Spatial Pyramid Pooling, is an effective module commonly used in semantic segmentation networks, such as the DeepLab series, to capture contextual information at different scales. ASPP obtains multi-scale information by using dilated convolutions with different dilation rates.

[0058] Backbone refers to the backbone network used for feature extraction in the encoder. The backbone network plays a key role in various visual tasks. It is responsible for extracting high-level feature representations from the input data. These representations are usually used by subsequent task-specific modules (such as classifiers, regressors, segmentation heads, etc.).

[0059] Supervised learning is a learning method that uses labeled data for training. The model makes predictions by learning the relationship between input data and corresponding labels.

[0060] Unsupervised learning is a learning method that is trained on unlabeled data. The model makes predictions or generates new data representations by discovering structures or patterns in the data.

[0061] Self-supervised learning is a special type of unsupervised learning. The model uses the characteristics of the data itself to generate pseudo labels for training. It can be regarded as a bridge between unsupervised learning and supervised learning.

[0062] ResNet-50 is a deep learning model, a type of residual network. The "50" in its name refers to its depth, that is, it contains 50 layers of neural network structure. ResNet-50 was proposed by researchers at Microsoft Research Asia to solve the gradient vanishing and gradient exploding problems that occur when training very deep neural networks.

[0063] Figure 1 It is a flowchart of a monocular depth estimation method based on joint loss function supervision provided by an embodiment of the present invention.

[0064] like Figure 1 As shown, an embodiment of the present invention provides a monocular depth estimation method based on joint loss function supervision, the method comprising:

[0065] S10, obtaining RGB images and point cloud data, and aligning the RGB images and point cloud data to construct a data set, wherein the data set includes a training set, a validation set and a test set; S20, constructing a depth estimation network model, inputting the training set into the depth estimation network model for training, and obtaining model parameters and depth prediction images; S30, based on the model parameters, inputting the validation set into the depth estimation network model for performance evaluation, so as to adjust the model parameters until the joint loss function of the validation set converges, and obtain the optimal model parameters; S40, according to the optimal model parameters, inputting the test set into the depth estimation network model for performance evaluation, and obtaining the trained depth estimation network model; S50, inputting the RGB image into the trained depth estimation network model for prediction, and obtaining a depth image.

[0066] The embodiment of the present invention constructs a depth estimation network model, and uses a data set obtained by aligning the RGB image collected by a monocular camera and the point cloud data collected by a lidar for training and performance evaluation, and designs a joint loss function in the training process to achieve dual loss supervision of the depth estimation network model learning process, thereby ensuring that the model can be effectively trained and evaluated, and reducing the interference of abnormal outliers in the data set on the prediction results, so that the depth estimation network model can predict the RGB image under the optimal model parameters, obtain the optimal solution of the depth estimation network model, and obtain a high-precision depth image corresponding to the RGB image, so as to effectively cope with various complex environmental conditions, such as illumination changes, occlusion, etc., to ensure that the generated depth image can be directly applied to tasks such as target detection and distance measurement in roadside scenes, has strong practicality, improves the stability and robustness of the overall depth estimation network, and improves the accuracy of the model prediction results.

[0067] Among them, by designing the model training process and introducing the joint loss function to perform dual loss supervised learning on the performance evaluation process of the depth estimation network model, the weight of abnormal outliers in the data set collected by the monocular camera and lidar due to the complex outdoor environment can be effectively reduced, the prediction error of the depth estimation network model can be reduced, and the accuracy of the existing monocular depth estimation network can be greatly improved. Among them, the principle of supervised learning is: collect and annotate the collected data to form input-output pairs (for example, RGB images and their corresponding labels); then use the labeled data to train the model, and measure the prediction error of the model by calculating the loss function, and then use the optimization algorithm (such as gradient descent) to adjust the model parameters until the loss function tends to the optimal solution (the loss function converges) to reduce the model error; finally, use the validation set to evaluate the model performance, and adjust the model parameters and hyperparameters according to the evaluation results, and use the test set to further evaluate the performance and generalization ability of the model to improve the prediction effect of the model, and finally obtain a trained depth estimation network model.

[0068] It should be emphasized that the purpose of using the validation set to evaluate the performance of the depth estimation network model is to observe the performance changes of the model on the validation set to decide whether to stop the training of the model in advance and adjust the model parameters, or take other measures to optimize the performance of the model; among them, the model parameters include learning rate, batch size and optimizer, etc., which are mainly adjusted through manual intervention. In the process of building the depth estimation network model, both the validation set and the test set are used to evaluate the performance of the depth estimation network model. The main difference between the two is the different uses and data distributions. The validation set is mainly used to adjust the model parameters during the training process to optimize the model performance, while the test set is only used to evaluate its generalization ability after the model training is completed.

[0069] In addition, learning-based methods are mainly divided into supervised learning, unsupervised learning, and self-supervised learning. Supervised learning requires a large amount of labeled data and is suitable for scenarios where labels are easy to obtain. Compared with unsupervised learning and self-supervised learning, it can usually achieve the highest accuracy; unsupervised learning does not require labeled data and is suitable for scenarios where data labeling is difficult; self-supervised learning does not require manual labeling and is trained on unsupervised data by generating pseudo labels. Therefore, introducing a joint loss function in the depth estimation network model to implement the supervised learning process of the model is very important for improving the accuracy of the depth estimation network and improving the accuracy of the model's prediction results. It is suitable for various complex environmental conditions, such as lighting changes, occlusion, etc., to ensure that the generated depth image can be directly applied to tasks such as target detection and distance measurement in roadside scenes, and has strong practicality.

[0070] Figure 2 It is a flowchart of step S10 in the monocular depth estimation method based on joint loss function supervision provided in an embodiment of the present invention.

[0071] like Figure 2 As shown, in an optional embodiment of the present invention, RGB images and point cloud data are obtained, and the RGB images and point cloud data are aligned to construct a data set, wherein the data set includes a training set, a validation set, and a test set, specifically:

[0072] S11, take RGB images with a monocular camera and collect point cloud data with a lidar; S12, based on a calibration algorithm, convert the lidar coordinate system to the monocular camera coordinate system to obtain an external parameter matrix, and convert the monocular camera coordinate system to the RGB image coordinate system to obtain an internal parameter matrix; S12, based on the external parameter matrix, convert the point cloud data to the monocular camera coordinate system to obtain depth information; S14, based on the internal parameter matrix, convert the depth information to the RGB image coordinate system to obtain the pixel position; S15, according to the pixel position, obtain the depth value corresponding to the depth information, and assign the depth value to the pixel position corresponding to the RGB image to obtain a true depth image.

[0073] Specifically, the RGB images captured by the monocular camera deployed in the roadside scene and the point cloud data collected by the corresponding LiDAR are used as input to construct a data set. Among them, the point cloud data acquired by the LiDAR provides accurate depth information, which is lacking in RGB images. By aligning the point cloud data with the RGB image, an image with depth information, namely the depth truth image, can be generated. It combines the features of the point cloud data and the RGB image data to achieve the reconstruction of the three-dimensional scene. For example, the point cloud data provides spatial coordinates, while the RGB image data provides color and texture information. The combination of the two can generate a realistic three-dimensional model. In some tasks, since manual annotation of three-dimensional information is very difficult and time-consuming, point cloud data can generate partial annotation information through automated methods to assist the data annotation process.

[0074] The actual working process of aligning the RGB image and the point cloud data to obtain the depth truth image is as follows: Based on the calibration algorithm, the laser radar coordinate system is converted to the monocular camera coordinate system to obtain the external parameter matrix T1, and the monocular camera coordinate system is converted to the RGB image coordinate system to obtain the internal parameter matrix T2; Based on the external parameter matrix T1, the point cloud data is converted to the monocular camera coordinate system to obtain the depth information. Taking the point cloud data with the value (x, y, z) as an example, there are:

[0075] (x c y c z c ) T =T1*(xyz) T ,

[0076] z represents the distance from the laser radar to the target, c represents the distance from the monocular camera to the target; then, based on the intrinsic parameter matrix T2, the depth information is converted to the RGB image coordinate system to obtain the pixel position:

[0077] (x i y i ) T =T2*(x c y c z c ) T ,

[0078] According to the pixel position, the depth value corresponding to the depth information is obtained, and the depth value is assigned to the pixel position corresponding to the RGB image to obtain a depth true value image. By aligning the point cloud data with the RGB image, an image with depth information, i.e., a depth true value image, can be generated. Combining the point cloud data and the RGB image data, it can realize the reconstruction of the three-dimensional scene, which is very important for the subsequent training of the depth estimation model and other related visual tasks. It should be noted that since the point cloud data collected by the lidar is sparse data relative to the pixel data of the RGB image taken by the monocular camera, there may be pixels in the RGB image that cannot correspond to the point cloud, and the depth values ​​of these pixels are usually assigned to 0.

[0079] Figure 2 It is a flowchart of step S10 in the monocular depth estimation method based on joint loss function supervision provided in an embodiment of the present invention.

[0080] like Figure 2 As shown, in an optional embodiment of the present invention, according to the pixel position, the depth value corresponding to the depth information is obtained, and the depth value is assigned to the pixel position corresponding to the RGB image to obtain the depth true value image, and it also includes: S16, constructing the data set according to the RGB image and the depth true value image; S17, randomly dividing the data set into a training set, a verification set and a test set.

[0081] Specifically, the generated RGB image and the corresponding depth truth image are used to construct a data set, and randomly selected and divided according to the ratio of training set: validation set: test set = 8:1:1. After the data set is divided, the training set and the validation set are used in the performance evaluation stage of the depth estimation network model, wherein the evaluation result of the validation set can be used to select the optimal weight to ensure that the depth estimation network model can predict the RGB image to be tested under its optimal model parameters to output a high-precision depth image. The test set is input into the network for performance evaluation after the depth estimation network model is trained, and its results are only used to evaluate the performance of the model in the test set to ensure that the depth estimation network model has been trained and can be directly used for the result prediction of the RGB image to be tested. It should be emphasized that the division ratio described in the optional embodiment of the present invention is only used to explain the specific implementation of the present application, and is intended to provide an example of implementing the technology, and does not mean to limit or restrict the present invention. In fact, the present invention can adopt various different division ratios to adapt to different application requirements and technical environments. Therefore, the division ratio mentioned in this example is only an example, and other applicable division ratios are not excluded.

[0082] Figure 3 is a flowchart of step S20 in the monocular depth estimation method based on joint loss function supervision provided by an embodiment of the present invention;

[0083] like Figure 3 As shown, in an optional embodiment of the present invention, a depth estimation network model is constructed, and a training set is input into the depth estimation network model for training, and the model parameters and the depth prediction image are obtained, including:

[0084] S21, construct a depth estimation network model, the depth estimation network model includes an encoder and a decoder; S22, preprocess the training set to enhance the image features of the training set; S23, input the preprocessed training set into the encoder for downsampling to obtain coding layer features; S24, input the coding layer features into the decoder for upsampling to obtain model parameters and depth prediction images.

[0085] Specifically, by constructing a depth estimation network model, the input training set is trained. In the encoder, downsampling is used to extract the features of the image from the preprocessed training set, while in the decoder, upsampling is used to convert these features into depth images. During the training process of the depth estimation network model, the model parameters can be updated by continuously optimizing the loss function until the loss function converges or the maximum number of iterations is reached. After the training is completed, these model parameters can be extracted and saved for model reasoning or further fine-tuning to optimize the accuracy of the monocular depth estimation network and improve the accuracy of the model prediction results.

[0086] It should be noted that before inputting the training set into the encoder, the training set needs to be preprocessed, such as normalization, flipping, cropping, color adjustment and other image feature enhancement operations, which can not only effectively increase the diversity of the training set data, thereby helping the depth estimation network model to better learn the invariance and universal features of the images in the data set, but also reduce the model's dependence on specific features of the training set, making it more generalized, thereby reducing the risk of overfitting and improving the performance of the model in practical applications.

[0087] Figure 4 It is a schematic diagram of a depth estimation network model in a monocular depth estimation method based on joint loss function supervision provided in an embodiment of the present invention.

[0088] like Figure 4 As shown, in an optional embodiment of the present invention, the depth estimation network model is a U-shaped fully convolutional network, and the U-shaped fully convolutional network is an encoder and decoder structure; wherein the encoder has a total of five encoding layers, and the step size of downsampling in each encoding layer is 2, and the fifth encoding layer is also provided with ASPP; the decoder has a total of four decoding layers, and the step size of upsampling in each decoding layer is also 2.

[0089] Specifically, the depth estimation network can adopt a U-shaped fully convolutional network, which mainly adopts a typical encoder-decoder structure, completes downsampling through convolution and pooling operations, and completes upsampling through nonlinear interpolation and convolution. The downsampling structure represents the encoding layer, and the upsampling represents the decoding layer. The workflow of the entire U-shaped fully convolutional network is:

[0090] Assume that the RGB training image of size H*W*3 in the preprocessed data set is sent to the encoder with ResNet-50 as the backbone to extract the feature information of the image. The encoder has five encoding layers, and each layer of the network performs convolution, regularization, activation, and pooling operations in sequence, and the step size of each layer is 2. After the input H*W*3 RGB image passes through the encoder, the resolution is downsampled to 1 / 32 of the original, and the size is (H / 32)*(W / 32)*C1. In order to obtain a larger receptive field of the RGB image and help the U-shaped full convolutional network extract more contextual information, ASPP is added to the last layer of the encoding layer, where the expansion rates of each pooling convolution are 3, 6, 12, 18, and 24 respectively. The input of the ASPP module is the features of each encoding layer in the encoder. After the ASPP module and convolution operation, the final encoder output encoding layer features are (H / 16)*(W / 16)*C2.

[0091] In this network, there are four decoding layers, and the upsampling step length of each layer is 2. The input of each decoding layer is the output features of the previous decoding layer and the corresponding encoding layer features. The purpose of introducing the encoding layer features is to alleviate the problem of information loss during downsampling. In the decoder, it first upsamples the output features of the previous decoding layer, and then fuses the encoding and decoding features of the same resolution with them at multi-scale features; finally, the fused features are convolved to obtain the final depth prediction image. The multi-level feature extraction of the encoder ensures the richness and complexity of semantic information, which is crucial for scene understanding in depth estimation tasks. The design of the decoder ensures that the output depth map has the same resolution as the input image, which improves the practicality and application value of the task.

[0092] Figure 4 It is a schematic diagram of a depth estimation network model in a monocular depth estimation method based on joint loss function supervision provided in an embodiment of the present invention.

[0093] like Figure 4 As shown, in an optional embodiment of the present invention, the depth prediction image includes an intermediate depth prediction image output by the second decoding layer and a final depth prediction image output by the fourth decoding layer, the intermediate depth prediction image is used to supervise the calculation of the loss function of the second decoding layer, and the final depth prediction image is used to supervise the calculation of the loss function of the fourth decoding layer.

[0094] Specifically, in order to improve the prediction performance of the depth estimation network, the network will output depth prediction images of size (H / 4)*(W / 4)*1 and H*W*1 in the second and fourth layers of the decoding layer, in addition to the normal output decoding features, to help the network to perform double supervision when calculating the loss function loss in the subsequent training process. It should be noted that the network uses the output of the last layer (i.e., the fourth layer) of the decoder as the final output depth image, while the depth image output by the intermediate layer (i.e., the second layer) decoding layer only serves as a supervisory function.

[0095] By using the depth prediction images of the second and fourth layers simultaneously, the network can benefit from supervisory signals of different levels and resolutions during training. The intermediate depth prediction image provides a rougher but earlier depth estimate, which is used as a supervisory signal when calculating the loss function. By comparing with the true depth image, it helps guide the network to learn more accurate depth features at an earlier stage, accelerating the convergence of the network in the initial learning stage; while the final depth prediction image provides a more accurate depth prediction, ensuring that the network has good performance and accuracy overall. By outputting depth images at different decoding layers, the network can comprehensively utilize feature information at different levels, thereby improving the global perception ability and local detail capture ability of depth estimation. This dual supervision strategy effectively utilizes the feature representations at different levels in the decoder and is an effective means to improve the performance of the depth estimation network.

[0096] Figure 4 It is a schematic diagram of a depth estimation network model in a monocular depth estimation method based on joint loss function supervision provided in an embodiment of the present invention.

[0097] like Figure 4 As shown, in an optional embodiment of the present invention, the joint loss function includes:

[0098] L=γL1+(1-γ)L2

[0099] Among them, L represents the joint loss function, L1 represents the loss function of the fourth decoding layer, L2 represents the loss function of the second decoding layer, γ represents the balance factor, and the value range of γ is [0,1].

[0100] Specifically, since monocular cameras and lidars need to be deployed in complex outdoor environments such as roadside scenes for data collection, point cloud data will have unevenness and some abnormal outliers, and the depth value of the training data is converted from point cloud data; in addition, the alignment operation of RGB images and point cloud data is required during the data conversion process. At this time, the dynamic targets in the picture have inherent data alignment errors and conversion accuracy errors, and these errors cannot be removed by other technical means. Therefore, these abnormal outliers exist for a long time, and their gradients in the model training process are large, which will guide the learning of the depth estimation network to these abnormal outliers. Therefore, in the process of learning the depth estimation network, in order to reduce the interference of erroneous data and obtain the optimal solution of the depth estimation network model, a joint loss function is designed in the process of evaluating the performance of the depth estimation network. During the training process, the weight of small errors (i.e., the difference between the predicted depth value and the actual depth value) is increased, and the influence of abnormal outliers is reduced, so as to improve the accuracy of the monocular depth estimation network and ensure that the model prediction results have a high accuracy. Finally, the depth image output by the trained depth estimation network model can be directly applied to tasks such as target detection and distance measurement in roadside scenes. The formula of the decoding layer loss function is as follows:

[0101]

[0102] In the formula, logx represents (logy-logy * ). First, the predicted depth value y and the actual depth value y * The actual difference (yy * ) is transformed into logarithmic space (logy-logy * ), (logy-logy * ) and the actual difference (yy * ), the loss function will not be too large in the early stage of training, the training will be more stable, and it will be easier to converge when the difference is large. Among them, θ and ɑ represent hyperparameters, which are:

[0103] C represents a constant, and B represents the gradient value;

[0104] According to the gradient value B between [-ɑ,ɑ], it can be seen that the two hyperparameters θ and ɑ jointly control the rate at which the loss function decreases. When the predicted depth value y and the actual difference transformed to the logarithmic space (logy-logy *) is smaller, the larger the gradient value B is. Therefore, when predicting RGB images, the depth estimation network model will pay more attention to errors in a small and medium range, thereby increasing the weight of small errors, thereby improving the accuracy of the monocular depth estimation network and ensuring that the model prediction results have a high accuracy rate. It should be noted that the value of the hyperparameter θ should not be too small, otherwise it will cause gradient explosion.

[0105] On the basis of the loss function of the decoding layer, in order to further improve the accuracy of depth prediction, the loss function loss is used not only in the last layer of the decoding layer (i.e., the fourth layer) to supervise the network, but also in the middle layer of the decoder (i.e., the second layer) to supervise the network. When training the depth estimation network model, the joint loss function used is:

[0106] L=γL1+(1-γ)L2

[0107] Among them, L represents the joint loss function, L1 represents the loss function of the fourth decoding layer, L2 represents the loss function of the second decoding layer, γ represents the balance factor, and the value range of γ is [0,1].

[0108] Therefore, in the joint loss function module, two sets of prediction and true value pairs are first input, such as the depth prediction image y1 of the middle layer of the decoder and the depth true value image y1 of the original depth image downsampled to the same resolution size. * 1. The depth prediction image y2 and the depth truth image y of the last layer of the decoder * 2. Then, the model parameters of the depth estimation network are iteratively updated until the joint loss function value of the validation set converges, and the training of the depth estimation network model is terminated. Finally, the model parameters with the best results in the validation set are selected as the optimal model parameters of the depth estimation network model.

[0109] Finally, the RGB image of the depth to be measured is input into the trained depth estimation network model for prediction, and the depth image corresponding to the RGB image can be obtained. The dual loss supervision of the learning process of the depth estimation network model ensures that the model can be effectively trained and evaluated, and reduces the interference of abnormal outliers in the data set on the prediction results, so that the depth estimation network model can predict the RGB image under the optimal model parameters, obtain the optimal solution of the depth estimation network model, and obtain a high-precision depth image corresponding to the RGB image, so as to effectively cope with various complex environmental conditions, such as lighting changes, occlusion, etc., to ensure that the generated depth image can be directly applied to tasks such as target detection and distance measurement in roadside scenes, and has strong practicality.

[0110] Figure 5 It is a schematic diagram of a monocular depth estimation device based on joint loss function supervision provided by an embodiment of the present invention.

[0111] like Figure 5 As shown, a monocular depth estimation device 100 based on joint loss function supervision includes:

[0112] The data set generation module 101 is used to obtain RGB images and point cloud data, and align the RGB images and point cloud data to construct a data set, wherein the data set includes a training set, a validation set, and a test set;

[0113] The depth estimation network module 102 is used to construct a depth estimation network model, input the training set into the depth estimation network model for training, and obtain model parameters and depth prediction images;

[0114] A joint loss function module 103 is used to input the validation set into the depth estimation network model for performance evaluation based on the model parameters, so as to adjust the model parameters until the joint loss function of the validation set converges to obtain the optimal model parameters;

[0115] A model training and evaluation module 104 is used to input the test set into the depth estimation network model for performance evaluation according to the optimal model parameters to obtain a trained depth estimation network model;

[0116] The depth image output module 105 is used to input the RGB image into the trained depth estimation network model for prediction to obtain a depth image.

[0117] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present invention, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0118] In the technical solution of the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0119] According to an embodiment of the present invention, the present invention also provides an electronic device, a readable storage medium and a computer program product.

[0120] Figure 6 A schematic block diagram of an example electronic device 200 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0121] like Figure 6 As shown, the device 200 includes a computing unit 201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 202 or a computer program loaded from a storage unit 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the device 200 can also be stored. The computing unit 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0122] A number of components in the device 200 are connected to the I / O interface 205, including: an input unit 206, such as a keyboard, a mouse, etc.; an output unit 207, such as various types of displays, speakers, etc.; a storage unit 208, such as a disk, an optical disk, etc.; and a communication unit 209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 209 allows the device 200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0123] The computing unit 201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 201 performs the various methods and processes described above, such as a power generation prediction method. For example, in some embodiments, a power generation prediction method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 200 via the ROM 202 and / or the communication unit 209. When the computer program is loaded into the RAM 203 and executed by the computing unit 201, one or more steps of a power generation prediction method described above may be performed. Alternatively, in other embodiments, the computing unit 201 may be configured to perform a power generation prediction method in any other appropriate manner (e.g., by means of firmware).

[0124] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, or tactile input).

[0128] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0129] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present invention should be included in the protection scope of the present invention.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A monocular depth estimation method based on joint loss function supervision, characterized in that: The method comprises: Acquire an RGB image and point cloud data, and align the RGB image and the point cloud data to construct a data set, wherein the data set includes a training set, a validation set, and a test set; Constructing a depth estimation network model, inputting the training set into the depth estimation network model for training, and obtaining model parameters and a depth prediction image; Based on the model parameters, inputting the validation set into the depth estimation network model for performance evaluation to adjust the model parameters until the joint loss function of the validation set converges to obtain optimal model parameters; According to the optimal model parameters, inputting the test set into the depth estimation network model for performance evaluation to obtain a trained depth estimation network model; The RGB image is input into the trained depth estimation network model for prediction to obtain a depth image.

2. The method according to claim 1, characterized in that The step of acquiring an RGB image and point cloud data and aligning the RGB image and the point cloud data to construct a data set, wherein the data set includes a training set, a validation set, and a test set. The RGB image is captured by a monocular camera and the point cloud data is collected by a laser radar; Based on the calibration algorithm, the laser radar coordinate system is converted to the monocular camera coordinate system to obtain the external parameter matrix, and the monocular camera coordinate system is converted to the RGB image coordinate system to obtain the internal parameter matrix; Based on the external parameter matrix, the point cloud data is converted into the monocular camera coordinate system to obtain depth information; Based on the intrinsic parameter matrix, the depth information is converted into the RGB image coordinate system to obtain a pixel position; According to the pixel position, a depth value corresponding to the depth information is obtained, and the depth value is assigned to the pixel position corresponding to the RGB image to obtain a true depth image.

3. The method according to claim 2, characterized in that After obtaining the depth value corresponding to the depth information according to the pixel position and assigning the depth value to the pixel position corresponding to the RGB image to obtain the depth true value image, the method further includes: Constructing the data set according to the RGB image and the depth truth image; The data set is randomly divided into the training set, the validation set and the test set.

4. The method according to claim 1, characterized in that: The constructing of the depth estimation network model, inputting the training set into the depth estimation network model for training, and obtaining model parameters and a depth prediction image comprises: Constructing the depth estimation network model, wherein the depth estimation network model includes an encoder and a decoder; Preprocessing the training set to enhance image features of the training set; Inputting the preprocessed training set into the encoder for downsampling to obtain coding layer features; The coding layer features are input into the decoder for upsampling to obtain the model parameters and the depth prediction image.

5. The method according to claim 4, characterized in that The depth estimation network model is a U-shaped fully convolutional network, and the U-shaped fully convolutional network is the encoder and the decoder structure; wherein, The encoder has five coding layers, the downsampling step length of each coding layer is 2, and the fifth coding layer is also provided with ASPP; The decoder has four decoding layers in total, and the step length of upsampling performed by each decoding layer is also 2.

6. The method according to claim 5, characterized in that The depth prediction image includes an intermediate depth prediction image output by the second decoding layer and a final depth prediction image output by the fourth decoding layer, the intermediate depth prediction image is used to supervise the calculation of the loss function of the second decoding layer, and the final depth prediction image is used to supervise the calculation of the loss function of the fourth decoding layer.

7. The method according to claim 6, characterized in that The joint loss function includes: L=γL1+(1-γ)L2 Among them, L represents the joint loss function, L1 represents the loss function of the decoding layer in the fourth layer, L2 represents the loss function of the decoding layer in the second layer, γ represents the balance factor, and the value range of γ is [0,1].

8. A monocular depth estimation device based on joint loss function supervision, characterized in that: include: A data set generation module, used for acquiring RGB images and point cloud data, and aligning the RGB images and the point cloud data to construct a data set, wherein the data set includes a training set, a validation set, and a test set; A depth estimation network module is used to construct a depth estimation network model, input the training set into the depth estimation network model for training, and obtain model parameters and depth prediction images; A joint loss function module, used to input the validation set into the depth estimation network model for performance evaluation based on the model parameters, so as to adjust the model parameters until the joint loss function of the validation set converges to obtain the optimal model parameters; A model training and evaluation module, used to input the test set into the depth estimation network model for performance evaluation according to the optimal model parameters to obtain a trained depth estimation network model; The depth image output module is used to input the RGB image into the trained depth estimation network model for prediction to obtain a depth image.

9. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.