Depth estimation model training, depth estimation method, apparatus, device, and medium

Abstract

The application discloses a kind of depth estimation model training, depth estimation method, device, equipment and medium, it is related to the field of automatic driving technology.The method comprises: obtaining target image, adjacent frame image;Depth estimation model and pose transformation model are constructed;Depth estimation model includes coding network and decoding network, coding network increases dense connection, channel attention mechanism, obtains the feature map of multiple scales;Decoding network fuses feature map to obtain depth estimation information;Pose transformation model obtains the pose transformation relationship between target image and adjacent frame image;Based on depth estimation information and pose transformation relationship, generate reconstructed image;Based on reconstructed image and target image, construct loss function, utilize it to complete the training of depth estimation model.The method increases dense connection, channel attention mechanism in coding network, the feature extracted is more effective, comprehensive, accurate, improves accuracy and inference speed under the condition that the number of layers of model increases less.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a depth estimation model training, depth estimation method, device, equipment and medium. Background Technology

[0002] Depth estimation is primarily used in autonomous driving systems to acquire information about the vehicle's surrounding environment. It is a crucial component of autonomous driving technology and a current research hotspot in the field. Currently, most mainstream methods for acquiring depth information are based on cameras or LiDAR. LiDAR can accurately measure the distance to objects within a certain space, but its limited scanning frequency, specular black hole problem, and high cost make its widespread application in autonomous driving difficult. Compared to LiDAR, camera-based vision solutions offer advantages such as high scanning frequency, long lifespan, and low cost. Based on camera type, vision solutions can be further divided into monocular camera-based and binocular camera-based solutions. For binocular camera solutions, traditional methods rely on feature point matching algorithms to find corresponding points in the left and right cameras. This method requires high accuracy in feature point matching and high synchronization between the left and right cameras. Monocular camera-based solutions do not have the synchronization limitations of binocular cameras and are cheaper; however, their accuracy is lower than that of binocular camera-based solutions.

[0003] Based on different training methods, monocular depth estimation can be divided into supervised learning and unsupervised learning. Supervised learning methods have relatively simple models, more accurate results, and easier training processes, but they require a large amount of labeled depth information as supervision signals during model training, and the cost of obtaining depth information is very high. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, embodiments of the present invention provide a depth estimation model training, depth estimation method, apparatus, device and medium.

[0005] In a first aspect, embodiments of the present invention provide a method for training a depth estimation model, comprising:

[0006] Acquire a target image and adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by a vehicle-mounted forward-looking monocular camera;

[0007] A depth estimation model and a pose transformation model are constructed. The depth estimation model includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network incorporates dense connection mechanisms and channel attention mechanisms into its residual structure to fuse shallow and deep features of the target image, obtaining feature maps of the target image at multiple scales. The depth estimation decoding network is used to fuse feature maps of the target image at multiple scales to obtain depth estimation information of the target image. The pose transformation model is used to obtain the pose transformation relationship between the target image and adjacent frame images.

[0008] Based on the depth estimation information and the pose transformation relationship, a reconstructed image is generated;

[0009] Based on the reconstructed image and the target image, a loss function is constructed, and the depth estimation model is trained using the loss function.

[0010] Secondly, embodiments of the present invention provide a depth estimation method, comprising: acquiring an image to be estimated; processing the image to be estimated using a pre-trained depth estimation model to obtain a depth map of the image to be estimated; and using the depth map as a depth estimation result of the image to be estimated, wherein the depth estimation model is trained using the depth estimation model training method of the present invention.

[0011] Thirdly, embodiments of the present invention provide a depth estimation model training apparatus, comprising:

[0012] The first acquisition module is used to acquire a target image and adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by a vehicle-mounted forward-looking monocular camera.

[0013] A construction module is used to build a depth estimation model and a pose transformation model. The depth estimation model includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network incorporates dense connection mechanisms and channel attention mechanisms into its residual structure to fuse shallow and deep features of the target image, obtaining feature maps of the target image at multiple scales. The depth estimation decoding network is used to fuse feature maps of the target image at multiple scales to obtain depth estimation information of the target image. The pose transformation model is used to obtain the pose transformation relationship between the target image and adjacent frame images.

[0014] The reconstruction module is used to generate a reconstructed image based on the depth estimation information and the pose transformation relationship;

[0015] The parameter training module is used to construct a loss function based on the reconstructed image and the target image, and to use the loss function to train the depth estimation model.

[0016] Fourthly, embodiments of the present invention provide a depth estimation device, comprising: a second acquisition module for acquiring an image to be estimated; and a depth determination module for processing the image to be estimated using a pre-trained depth estimation model to obtain a depth map of the image to be estimated, and using the depth map as the depth estimation result of the image to be estimated.

[0017] Fifthly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the depth estimation model training method or depth estimation method of the embodiments of the present invention.

[0018] In a sixth aspect, embodiments of the present invention provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the depth estimation model training method or depth estimation method of the embodiments of the present invention.

[0019] One embodiment of the above invention has the following advantages or beneficial effects:

[0020] The depth estimation model training method of this invention obtains depth estimation information of a target image through a depth estimation model, obtains the pose transformation relationship between the target image and adjacent frame images through a pose transformation model, generates a reconstructed image based on the depth estimation information and pose transformation relationship, constructs a loss function based on the reconstructed image and the target image, and optimizes the parameters of the depth estimation model using the loss function to complete the training of the depth estimation model. The depth estimation model includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network incorporates dense connection mechanisms and channel attention mechanisms into its residual structure to fuse shallow features of the target image and... Deep features are obtained by acquiring feature maps at multiple scales of the target image, making the extracted features more effective, comprehensive, and accurate. This improves accuracy and inference speed with only a slight increase in the number of model layers. The depth estimation decoding network is used to fuse feature maps at multiple scales of the target image to obtain depth estimation information. This allows for the full fusion of shallow and deep features of the target image, ensuring that the localization information represented by shallow features is fully integrated with the semantic information represented by deep features. The resulting depth estimation results are highly accurate, thereby improving the accuracy, inference speed, and robustness of the depth estimation model. It can adapt to most driving scenarios and provide accurate depth information for autonomous driving systems.

[0021] The depth estimation method of this invention uses a depth estimation model to analyze the two-dimensional image acquired by the vehicle's forward-looking monocular camera to determine the distance between objects in front and the vehicle body. This allows for the rapid and accurate input of the depth information corresponding to the video frame after the forward-looking monocular camera acquires the video frame. It enables the detection and estimation of the distance between the vehicle and objects such as vehicles, people, and buildings in front under different weather and road conditions. This forms a highly accurate, robust, and real-time depth estimation system, which is convenient for subsequent three-dimensional target recognition and for providing environmental information for obstacle avoidance and path planning of autonomous vehicles.

[0022] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0023] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0024] Figure 1 A flowchart of the depth estimation model training method according to an embodiment of the present invention is shown;

[0025] Figure 2 The diagram shows the structure of the depth estimation encoding network in the depth estimation model training method according to an embodiment of the present invention.

[0026] Figure 3 A structural diagram of the second coding layer of the depth estimation coding network according to an embodiment of the present invention is shown;

[0027] Figure 4 The diagram shows the structure of the depth estimation decoding network in the depth estimation model training method of this invention.

[0028] Figure 5 This diagram illustrates a sub-process of the depth estimation model training method according to an embodiment of the present invention.

[0029] Figure 6 A flowchart of the depth estimation method according to an embodiment of the present invention is shown;

[0030] Figure 7 A flowchart of another embodiment of the depth estimation method of the present invention is shown;

[0031] Figure 8 A schematic diagram of the structure of a depth estimation model training device according to an embodiment of the present invention is shown;

[0032] Figure 9 A schematic diagram of the depth estimation device according to an embodiment of the present invention is shown;

[0033] Figure 10 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0034] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] In the field of deep learning, generally speaking, the more layers a model contains and the more complex it is, the more complex the nonlinear mapping relationships it can construct, and the more accurate the model's output. However, as the number of layers and complexity increase, the inference speed of the model slows down, and the hardware resources consumed also increase. For depth estimation tasks with high real-time requirements, long inference times are unacceptable. To address the contradiction between model accuracy and model complexity, this invention provides a depth estimation model training method. This method proposes a novel network structure for the depth estimation model, which can improve accuracy with a small increase in the number of layers compared to the original model. It also features fast inference speed, strong robustness, and adaptability to most driving scenarios, providing accurate depth information for autonomous driving systems.

[0036] Figure 1 A flowchart illustrating the depth estimation model training method according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, the method includes:

[0037] Step S101: Acquire the target image and the adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by the vehicle-mounted forward-looking monocular camera.

[0038] This embodiment can acquire video data of a vehicle in motion from an onboard forward-looking monocular camera and decode the video data into a series of consecutive frames. The target image is any frame within this series of frames. Adjacent frames to the target image include the frame preceding and following the target image.

[0039] Step S102: Construct a depth estimation model and a pose transformation model.

[0040] This depth estimation model is used to obtain depth estimation information of a target image, i.e., the distance between objects and the vehicle body in the target image. The depth estimation model consists of an encoding part and a decoding part; that is, it includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network is used to acquire multiple features of the target image, while the depth estimation decoding network is used to obtain a depth estimate of the target image based on these multiple features, i.e., the distance between objects and the vehicle body in the target image.

[0041] In order to improve accuracy and inference speed with fewer layers in the model compared to the original, this invention incorporates dense connection mechanism and channel attention mechanism into the residual structure of the deep estimation coding network to fuse shallow features (also known as low-level features) and deep features (also known as high-level features) of the target image to obtain feature maps of the target image at multiple scales.

[0042] The depth estimation coding network includes a first coding layer and multiple second coding layers. The first coding layer includes a convolutional layer, a batch normalization (BN) layer, and an activation layer (RELU). The second coding layer is a residual structure, and includes multiple third coding layers and channel attention layers, with the multiple third coding layers connected by dense connections. The third coding layers include convolutional layers, batch normalization layers, and activation layers. In this embodiment, a max-pooling layer is also included between the first and second coding layers.

[0043] As an example, Figure 2 A schematic diagram of the structure of a depth estimation coding network according to an embodiment of the present invention is shown. Figure 3 This diagram illustrates the structure of the second coding layer in a depth estimation coding network according to an embodiment of the present invention. The depth estimation coding network in this example can be based on ResNet18, with dense connections and channel attention mechanisms added to the residual part. The network structure of the depth estimation coding network is as follows: Figure 2 As shown, this depth estimation coding network consists of one first coding layer and three second coding layers. Figure 3As shown, the second encoding layer is a residual structure, containing three third encoding layers consisting of convolutional layers, batch normalization layers, and activation layers, and a channel attention layer. The input of each third encoding layer is composed of the outputs of all previous third encoding layers and the input of the entire second encoding layer. For example, the input of the third third encoding layer is formed by concatenating the original input, the output of the first third encoding layer, and the output of the second third encoding layer along the channel dimension. In the channel attention layer, the output of the third third encoding layer is first subjected to global average pooling along the channels to obtain a 1x1xC tensor (where C is the number of output channels of the third encoding layer). Then, information is fused between channels using 1x1 convolutional kernels and activation layers. Next, a 1x1 convolutional layer and activation layer are used to obtain the weights of each channel of the input features; these weights are the output of the channel attention layer. Finally, the output of the third third encoding layer is multiplied by the weights of each channel, and the result is added to the input of the entire second input layer as the residual, serving as the output of the depth estimation encoding network.

[0044] In an optional embodiment, for the second coding layer, when the number of output channels of the second coding layer is inconsistent with the number of input channels, convolution and normalization operations are performed on the input information of the second coding layer to make the number of input channels of the second coding layer consistent with the number of output channels. When the number of output channels of the second coding layer is inconsistent with the number of input channels, the input and output information of the second coding layer cannot be added. To solve the problem of not being able to add, this embodiment requires performing convolution and normalization operations on the input information of the second coding layer when the number of output channels of the second coding layer is inconsistent with the number of input channels, so that the number of input channels of the second coding layer is consistent with the number of output channels.

[0045] The depth estimation coding network of this invention, compared with networks without dense connections, enhances the information transfer between feature maps, utilizes existing features more effectively, and halves the output channels of the first third coding layer, limiting the increase in the number of parameters. Furthermore, the channel self-attention mechanism deepens the fusion between channels, strengthening channels that are more effective for depth estimation. The 1x1 convolution used also has fewer parameters than fully connected layers, further effectively preventing a significant increase in computational cost and overfitting.

[0046] To fully integrate the shallow and deep features of the target image obtained by the depth estimation coding network, this invention proposes to utilize deep features to shallow features, and combine features from the same layer with a densely connected depth estimation decoding network, thereby fully integrating the semantic information of deep features and the localization information of shallow features to obtain accurate depth estimation information.

[0047] Furthermore, the depth estimation decoding network includes a first decoding layer and multiple second decoding layers. The input to the first decoding layer is the feature map output by the first encoding layer and the feature information output by the adjacent second decoding layer. The input to the second decoding layer is the feature map output by the corresponding second encoding layer and the feature information output by the second decoding layer of the previous level.

[0048] Furthermore, the first and second decoding layers include extraction nodes, fusion nodes, and output nodes. The fusion nodes are densely connected to each other and to the output nodes. Extraction nodes are used to acquire feature maps output by the first or second coding layer; fusion nodes are used to fuse feature information output by extraction nodes in the same layer, as well as feature information output by extraction and fusion nodes in the previous layer; output nodes are used to obtain depth maps of the target image at different scales based on the feature information output by the fusion nodes.

[0049] As an example, the network structure of a depth estimation decoding network is as follows: Figure 4 As shown, this depth estimation decoding network and Figure 2 The depth estimation coding network shown corresponds to this. Figure 4 Each row in the diagram represents a decoding layer, with the first row being the first decoding layer and the second through fifth rows representing the second decoding layer. Nodes numbered 00, 10, 20, 30, and 40 in the diagram represent the output of the specified target layer of the depth estimation coding network. Figure 2 Layers 0 to 4, i.e., nodes numbered 00, 10, 20, 30, and 40, are the extraction nodes. Node number 40 is the output of the deepest layer of the encoding network, corresponding to... Figure 2 Layer 4 in the network. The decoding network passes features from the highest layer (represented by node 40) to the lower layers, that is, from deep features to shallow features. It differs from FPN (Feature Pyramid Network) in two main ways: first, the decoding layer containing nodes 00, 10, and 20 not only contains features from the encoding network, but also fuses high-level features from all adjacent decoding layers (in...). Figure 4In this diagram, the lower rows are referred to as high-level features. For example, in the decoding layer where node 10 is located, it not only contains the output features from the encoding network represented by node 10, but also features generated by upsampling and fusing the high-level features 20, 21, and 22 from adjacent decoding layers, represented by nodes 11, 12, and 13 respectively (nodes 11, 12, and 21 are fusion nodes). Each high-level feature is passed to the adjacent low-level layers, enhancing the information fusion between semantic layers. The second difference is that dense connections are used in the layers where nodes 00, 10, and 20 are located. That is, each node receives the features from its previous layer as input, concatenates these features with the high-level features on the channel, and uses a convolutional network to fuse the different features. The dense connection design enhances the fusion between features at the same level and better extracts and utilizes the features provided by the encoding network at each level. In the diagram, nodes 22, 13, and 04 are output nodes. After passing through convolutional layers and activation layers, they will output depth estimates between 0 and 1, that is, output depth maps at different scales (i.e., depth maps at different scales). Figure 4 (Disp1, Disp2, Disp3 in the text).

[0050] In an optional embodiment, the method further includes performing upsampling, convolution, and activation operations on the feature information output by the output nodes of the first decoding layer to obtain a depth map of the target image. Continuing with... Figure 4 For example, the depth map output by the output node numbered 04 is first upsampled, then subjected to convolution and activation operations, and its output is a depth map with the same resolution as the original target (i.e., Figure 4 (Disp0 in the middle).

[0051] The pose transformation model takes two adjacent images as input and uses them to obtain the pose transformation relationship between the two adjacent images, that is, the perspective transformation information (rotation angles and translations along three axes) from one image to another with 6 degrees of freedom. The structure of the pose transformation model is generally divided into an encoding part and a decoding part; that is, the pose transformation model includes a pose transformation encoding network and a pose transformation decoding network. The structure design of the pose transformation encoding network is basically the same as ResNet18, except that the number of input channels of the first convolutional layer is set to 6, because the input consists of two color images, each with 3 channels, and the output is the high-level features extracted by the model. The role of the pose transformation decoding network is to decode the features obtained by the pose transformation encoding network. By performing multiple convolutional kernel activation operations on the features of the encoding network, it obtains the perspective transformation information (rotation angles and translations along three axes) from the target image to the adjacent frame image with 6 degrees of freedom.

[0052] Step S103: Based on the depth estimation information and the pose transformation relationship, generate a reconstructed image. This reconstructed image is the target image for reconstruction.

[0053] Specifically, based on the camera imaging principle, the process of reconstructing the target image is as follows: Figure 5 As shown, it may include:

[0054] Step S501: Based on the pose transformation relationship, determine the projection transformation matrix. Generate the rotation matrix of the projection transformation according to the rotation angles of the three axes from the target image to the adjacent frame image output by the pose transformation model, and generate the translation matrix of the projection transformation along the translation amounts of the three axes.

[0055] Step S502: Based on the intrinsic parameter matrix of the monocular camera and the depth estimation information, determine the transformation relationship between the pixel coordinate system and the camera coordinate system of the target image, and based on the transformation relationship, determine the first coordinate of the pixel of the target image in the camera coordinate system;

[0056] Step S503: Based on the intrinsic parameter matrix and the projection transformation matrix, convert the first coordinates into second coordinates in the pixel coordinate system of the adjacent frame images;

[0057] Step S504: Based on the second coordinates, sample the adjacent frame images to generate a reconstructed image.

[0058] Furthermore, in this embodiment of the invention, the adjacent frame images of the target image include the preceding adjacent frame image and the following adjacent frame image. This embodiment of the invention can generate reconstructed images based on the preceding adjacent frame image and the following adjacent frame image, respectively. Therefore, obtaining the pose transformation relationship between the target image and the adjacent frame images can include: obtaining a first pose transformation relationship between the target image and the preceding adjacent frame image; and obtaining a second pose transformation relationship between the target image and the following adjacent frame image. Then, a reconstructed image is generated based on the first pose transformation relationship, and a second reconstructed image is generated based on the second pose transformation relationship.

[0059] Combining such as 2 and Figure 4 The depth estimation model outputs four depth maps at different scales. The adjacent frame images of the target image include the previous adjacent frame image and the next adjacent frame image. Therefore, a reconstructed image is generated for each depth map and the adjacent frame image, resulting in a total of eight reconstructed images.

[0060] Step S104: Based on the reconstructed image and the target image, construct a loss function and use the loss function to train the depth estimation model.

[0061] The design of the loss function has a significant impact on the performance of self-supervised deep learning algorithms. In this embodiment of the invention, based on a depth estimation model and a pose transformation model, the target image is reconstructed using adjacent frame images. The reconstructed image is then compared with the original target image to construct the loss, thereby training the two models.

[0062] The depth estimation model training method of this invention obtains depth estimation information of a target image through a depth estimation model, obtains the pose transformation relationship between the target image and adjacent frame images through a pose transformation model, generates a reconstructed image based on the depth estimation information and pose transformation relationship, constructs a loss function based on the reconstructed image and the target image, and optimizes the parameters of the depth estimation model using the loss function to complete the training of the depth estimation model. The depth estimation model includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network incorporates dense connection mechanisms and channel attention mechanisms into its residual structure to fuse shallow features of the target image and... Deep features are obtained by acquiring feature maps at multiple scales of the target image, making the extracted features more effective, comprehensive, and accurate. This improves accuracy and inference speed with only a slight increase in the number of model layers. The depth estimation decoding network is used to fuse feature maps at multiple scales of the target image to obtain depth estimation information. This allows for the full fusion of shallow and deep features of the target image, ensuring that the localization information represented by shallow features is fully integrated with the semantic information represented by deep features. The resulting depth estimation results are highly accurate, thereby improving the accuracy, inference speed, and robustness of the depth estimation model. It can adapt to most driving scenarios and provide accurate depth information for autonomous driving systems.

[0063] In an optional embodiment, the process of constructing a loss function based on the reconstructed image and the target image, and using the loss function to train the depth estimation model, may include:

[0064] Based on structural similarity and Manhattan distance, a loss function is determined between the reconstructed image and the target image;

[0065] Based on the loss function, calculate the target loss for each pixel between the reconstructed image and the target image;

[0066] The depth estimation model is trained based on the target loss.

[0067] This invention utilizes the structural similarity (SSIM) between the reconstructed image and the original target image, as well as the Manhattan distance, to calculate the accuracy of the reconstruction. SSIM measures similarity based on three aspects: brightness, contrast, and structure, while the Manhattan distance directly calculates the difference in pixel values ​​between the images.

[0068] Previous methods have always performed calculations in the RGB color space of the image. If the exposure intensity of the reconstructed image and the target image are inconsistent, even if the depth model and pose transformation model provide accurate estimates, there will still be a certain reconstruction error between the calculated reconstructed image and the original image. Similarly, when objects in the image are shadowed by different objects, the brightness of the corresponding pixels will vary due to the different shadows, ultimately causing the reconstruction error to fail to accurately reflect the accuracy of the designed model. This invention uses the LAB color space instead of the RGB mode. The LAB color space still has 3 channels, but unlike RGB, the LAB color space decouples the image brightness and color. Channel L is only responsible for brightness information, while channels A and B collect color information.

[0069] Before calculating the SSIM and Manhattan distance, this embodiment of the invention further includes converting the reconstructed image and the target image to LAB mode, normalizing the values ​​of the A and B channels of the reconstructed image and the target image respectively, and scaling the value of the L channel to a preset range. For example, the reconstructed image and the target image are converted from RGB mode to LAB mode, and the values ​​of the A and B channels are normalized to [-1, 1], while the range of channel L is scaled to [-α, α]. Here, α is a positive number less than 1. This is equivalent to reducing the influence of brightness on the SSIM and Manhattan distance, making the error of the target image insensitive to changes in brightness.

[0070] In an optional embodiment, the process of determining the loss function between the reconstructed image and the target image in the LAB mode based on structural similarity and Manhattan distance includes:

[0071] Based on structural similarity and Manhattan distance, the reconstruction loss and translation loss between the reconstructed image and the target image in the LAB mode are determined;

[0072] Based on the reconstruction loss and the translation loss, a loss function is determined between the reconstructed image and the target image in the LAB mode.

[0073] The reconstruction loss between the reconstructed image and the target image in the LAB mode is determined according to the following formula:

[0074]

[0075]

[0076] α and β represent weights, which are positive numbers between 0 and 1, and [i,j] represents the pixel in the i-th row and j-th column of the target image. This represents the reconstruction loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the structural similarity loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. These represent the Manhattan distance losses in the L, A, and B channels, respectively, between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. The value of m is either 1 or 2. Combined with... Figure 2 and Figure 4 In the example shown, k can be 1, 2, 3, or 4.

[0077] The formula for calculating SSIM is as follows:

[0078]

[0079] Where, μ src and μ dst These are the mean pixel values ​​of the target image and the reconstructed image, respectively, σ src and σ dst These are the variances of the pixel values ​​in the target image and the reconstructed image, respectively, σ. src,dst C1 and C2 represent the covariance of pixel values ​​between the target image and the reconstructed image, respectively, and are small constants. The SSIM loss is:

[0080] L SSIM = (1-SSIM) / 2

[0081] The SSIM loss value ranges from [0,1].

[0082] The loss corresponding to Manhattan distance is defined as:

[0083] L dis =|I src -I dst |1

[0084] The translation loss between the reconstructed image and the target image in the LAB mode is determined according to the following formula:

[0085]

[0086] Where γ represents the weight, which is a positive number between 0 and 1, and L neib,m L represents the translation loss between the reconstructed image generated using the m-th adjacent frame image and the target image. SSIM (I neib,m ,I dst) represents the structural similarity loss between the m-th adjacent frame image and the target image. This represents the Manhattan distance loss between the m-th adjacent frame image and the target image.

[0087] In this embodiment of the invention, the above formula calculates pixel-level reconstruction loss and motion loss (also referred to as reconstruction error and motion error). Reconstruction error measures the similarity between the reconstructed image and the target image. Translation error measures the similarity between adjacent frame images and the target image, and can also be seen as the similarity between the reconstructed images generated from adjacent frame images and the target image.

[0088] The reason for calculating translation loss in this invention is that the unsupervised algorithm used in this invention assumes that objects remaining stationary in different frames have an infinite depth (only when the object's depth is infinite can it remain stationary in the reconstructed image obtained after pose transformation). However, in reality, objects may remain relatively stationary to the camera. Without processing, the model would consider the distance between the object and the camera to be infinite, which is inconsistent with reality. Similar misclassifications can occur in low-texture regions of the target image. This invention utilizes the characteristic of small translation loss in such scenarios (relatively stationary objects are at the same position in adjacent frames and the target image, so the translation loss at this position is very small). It compares the reconstruction loss and translation loss at the same pixel. If the translation loss is less than the reconstruction loss, then the corresponding location is considered to be a stationary object or a low-texture region, and the loss generated at this location is not included in the loss function. In this way, the trained model will not be affected by the interference of stationary objects, thus providing a more accurate depth estimate.

[0089] Due to occlusion between adjacent frames, this embodiment of the invention selects the smaller value of the reconstruction error and the smaller value of the translation error at the pixel level as the reconstruction error and translation error under a certain depth map. That is, as shown in the following formula, for two reconstructed images generated using the same depth map but different adjacent frame images, the reconstruction loss and translation loss of each pixel in the two reconstructed images are compared, and the smaller reconstruction loss and the smaller translation loss are respectively used as the reconstruction loss and translation loss between the reconstructed image generated using the depth map and the target image. (Combined with...) Figure 4 The depth estimation model outputs four depth images at different scales. The adjacent frames of the target image include the previous adjacent frame image and the next adjacent frame image. Therefore, for each depth map, a reconstructed image can be generated using the previous adjacent frame image and the next adjacent frame image respectively. There are two reconstructed images corresponding to the depth map. The smaller value of the reconstruction error and the smaller value of the translation error are selected pixel by pixel from the two reconstructed images as the reconstruction error and translation error under the depth map.

[0090]

[0091] L neib =min(L neib,1 ,L neib,2 )

[0092] After determining the smaller reconstruction loss and smaller translation loss for each depth map, the smaller reconstruction loss is used as the target reconstruction loss, and the smaller translation loss is used as the target translation loss. Pixels whose target reconstruction loss is greater than or equal to the target translation loss error are ignored. The target reconstruction loss based on pixels whose target reconstruction loss is less than the target translation loss error is used as the target loss. Based on the target loss, a loss function is determined between the reconstructed image and the target image in the LAB mode. That is, as shown in the following formula, the final error in this embodiment of the invention only takes the reconstruction error formed by those pixels whose translation error is less than the translation error.

[0093]

[0094]

[0095] Where, loss k,m This represents the target loss between the reconstructed image generated using the k-th depth map and the target image. L represents the target reconstruction loss. neib M represents the target translation loss. k denoted as the mask matrix, if the reconstruction error of a certain pixel is less than the translation error, then the value of its corresponding position in M ​​is 1, otherwise it is 0.

[0096] In digital image processing, a mask is a two-dimensional matrix. Image masks are mainly used for: (1) Extracting Regions of Interest (ROIs): Multiplying a pre-made ROI mask with the image to be processed yields the ROI image. Image values ​​within the ROI remain unchanged, while image values ​​outside the ROI are all 0. (2) Masking: Using a mask to mask certain regions of the image, preventing them from participating in processing or the calculation of processing parameters, or only processing or statistically analyzing the masked areas. (3) Structural Feature Extraction: Detecting and extracting structural features in the image similar to the mask using similarity variables or image matching methods.

[0097] Finally, depth maps at all scales in the depth estimation model are included in the calculation of the training error, ensuring that different layers in the depth estimation decoding model learn the mapping relationship with the depth maps as much as possible through backpropagation of the error. The training error of the model is then taken as the average of the training errors corresponding to multiple depth maps, i.e.:

[0098]

[0099] Where L represents the average loss and n represents the number of depth maps.

[0100] This invention addresses the issue of inconsistent exposure intensities between adjacent video frames by proposing a method that replaces the RGB color space with a LAB color space and a three-channel weighted design for calculating the loss function. This new loss function calculates the L1 loss between images. Specifically, this invention decouples brightness and color in the LAB color space and assigns different weighting coefficients, reducing the impact of inconsistent light intensity caused by factors such as sunlight occlusion and camera hardware in real-world scenes on depth estimation. According to projection principles, objects stationary relative to a monocular camera are perceived as having infinite depth, leading to incorrect depth estimation. Similarly, low-texture regions exhibit similar issues. Therefore, this invention designs a new masking method, M... k [i,j] intelligently identifies the parts that are stationary relative to the camera and low-texture regions, and prevents the loss generated by these parts from participating in the training of the model, thus avoiding the generation of incorrect depth estimation results.

[0101] The depth estimation model training method of this invention uses a self-supervised deep learning method to estimate depth information. It has low requirements for the supporting embedded system, eliminates the need for synchronization of multiple camera times, and does not require labeled data samples, thus reducing the difficulty of model training, decreasing the steps of information extraction, and improving the efficiency of algorithm deployment. The encoding and decoding networks of the depth estimation model in this invention employ residual connections, dense connections, and channel attention structures to fully mine and utilize the feature information extracted from the image. This achieves good depth estimation accuracy with a low parameter count, achieving a good balance between model complexity and model accuracy. The loss function calculation method of this invention considers the impact of inconsistent exposure of the same object between different frames and inconsistent shadows formed by light sources on training errors. It uses a mask matrix to shield training errors caused by objects stationary relative to the camera and low-texture areas, enhancing the ability to judge depth information in the corresponding areas.

[0102] Figure 6 A flowchart of the depth estimation method according to an embodiment of the present invention is shown, as follows: Figure 6 As shown, the method includes:

[0103] Step S601: Obtain the image to be estimated;

[0104] Step S602: Process the image to be estimated using a pre-trained depth estimation model to obtain a depth map of the image to be estimated, and use the depth map as the depth estimation result of the image to be estimated. The depth estimation model is trained according to the depth estimation model training method described in any of the above embodiments.

[0105] This invention analyzes the image to be estimated using a depth estimation model, enabling the rapid and accurate input of the depth information corresponding to the video frame after the forward-looking monocular camera acquires the video frame. This achieves the detection and estimation of the distance between the vehicle and objects such as vehicles, people, and buildings ahead under different weather and road conditions, forming a depth estimation method with high accuracy, strong robustness, and real-time detection capability.

[0106] In an optional embodiment, combined with Figures 2-4 The depth estimation model shown obtains the depth map of the image to be estimated by acquiring the depth map output by the first decoding layer of the depth estimation model, and uses the depth map output by the first decoding layer as the depth estimation result of the image to be estimated. In other optional embodiments, depth maps output by other decoding layers of the depth estimation model can also be obtained as the depth estimation result of the image to be estimated.

[0107] Figure 7 A flowchart of another embodiment of the depth estimation method of the present invention is shown, as follows: Figure 7 As shown, the method includes:

[0108] Step S701: Obtain the image to be estimated;

[0109] Step S702: Mirror the image to be estimated to obtain a mirror image corresponding to the image to be estimated;

[0110] Step S703: Process the image to be estimated using a pre-trained depth estimation model to obtain a first depth map of the image to be estimated;

[0111] Step S704: Process the mirror image using the depth estimation model to obtain a second depth map of the mirror image;

[0112] Step S705: Based on the first depth map and the second depth map, determine the depth estimation result of the image to be estimated.

[0113] At this point, it is equivalent to having depth estimation maps of two images to be estimated. For autonomous driving scenarios, to ensure vehicle safety, the image with the closer distance between the two depth estimation maps is taken pixel-by-pixel as the final depth estimation result for the image to be estimated. Therefore, step S705 may include:

[0114] The second depth map is mirrored to obtain the third depth map;

[0115] For each pixel in the image to be estimated, the smaller depth in the first depth map and the third depth map is determined, and the smaller depth is used as the depth estimation result of the pixel.

[0116] The depth estimation method of this invention involves mirroring the image to be estimated to obtain a mirror image, using a depth estimation model to obtain a depth map of the image to be estimated and a depth map of the mirror image, which is equivalent to having two depth estimation maps of the image to be estimated. The method then takes the estimate of the image with the closer distance between the two depth maps pixel by pixel as the final depth estimation result of the image to be estimated, which can improve the safety of vehicle driving.

[0117] Figure 8 A schematic diagram of the structure of a depth estimation model training device 800 according to an embodiment of the present invention is shown, as follows: Figure 8 As shown, the device 800 includes:

[0118] The first acquisition module 801 is used to acquire a target image and adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by a vehicle-mounted forward-looking monocular camera.

[0119] Module 802 is used to construct a depth estimation model and a pose transformation model. The depth estimation model includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network incorporates a dense connection mechanism and a channel attention mechanism into its residual structure to fuse shallow and deep features of the target image, obtaining feature maps of the target image at multiple scales. The depth estimation decoding network is used to fuse feature maps of the target image at multiple scales to obtain depth estimation information of the target image. The pose transformation model is used to obtain the pose transformation relationship between the target image and adjacent frame images.

[0120] The reconstruction module 803 is used to generate a reconstructed image based on the depth estimation information and the pose transformation relationship;

[0121] The parameter training module 804 is used to construct a loss function based on the reconstructed image and the target image, and to use the loss function to train the depth estimation model.

[0122] Optionally, the depth estimation coding network includes a first coding layer and multiple second coding layers; the first coding layer includes a convolutional layer, a batch normalization layer, and an activation layer; the second coding layer is a residual structure, and the second coding layer includes multiple third coding layers and a channel attention layer, wherein the multiple third coding layers are connected in a dense connection manner.

[0123] Optionally, the third coding layer includes a convolutional layer, a batch normalization layer, and an activation layer.

[0124] Optionally, the method further includes: for the second coding layer, when the number of output channels of the second coding layer is inconsistent with the number of input channels of the second coding layer, performing convolution and normalization operations on the input information of the second coding layer so that the number of input channels of the second coding layer is consistent with the number of output channels of the second coding layer.

[0125] Optionally, the depth estimation decoding network includes a first decoding layer and multiple second decoding layers; the input of the first decoding layer is the feature map output by the first encoding layer and the feature information output by the adjacent second decoding layer; the input of the second decoding layer is the feature map output by the corresponding second encoding layer and the feature information output by the second decoding layer of the previous level.

[0126] Optionally, the first decoding layer and the second decoding layer include extraction nodes, fusion nodes, and output nodes; the extraction nodes are used to obtain feature maps output by the first encoding layer or the second encoding layer; the fusion nodes are used to fuse feature information output by extraction nodes in the same layer and feature information output by extraction nodes and fusion nodes in the previous layer; the output nodes are used to obtain depth maps of the target image at different scales based on the feature information output by the fusion nodes.

[0127] Optionally, the construction module is further configured to: perform upsampling, convolution, and activation operations on the feature information output by the output node of the first decoding layer to obtain a depth map of the target image.

[0128] Optionally, the pose transformation model includes a pose transformation encoding network and a pose transformation decoding network; the pose transformation encoding network is used to obtain pose transformation features between the target image and the adjacent frame images; the pose transformation decoding network is used to obtain the rotation angle and translation amount between the target image and the adjacent frame images based on the pose transformation features, and use the rotation angle and the translation amount as the pose transformation relationship between the target image and the adjacent frame images.

[0129] Optionally, the reconstruction module is further configured to: determine a projection transformation matrix based on the pose transformation relationship; determine the transformation relationship between the pixel coordinate system and the camera coordinate system of the target image based on the intrinsic parameter matrix of the monocular camera and the depth estimation information, and determine the first coordinate of the pixel of the target image in the camera coordinate system based on the transformation relationship; convert the first coordinate into a second coordinate in the pixel coordinate system of the adjacent frame image based on the intrinsic parameter matrix and the projection transformation matrix; and sample the adjacent frame image based on the second coordinate to generate a reconstructed image.

[0130] Optionally, the adjacent frame images include the previous adjacent frame image and the subsequent adjacent frame image;

[0131] The construction module is further configured to: obtain the first pose transformation relationship between the target image and the previous adjacent frame image; and obtain the second pose transformation relationship between the target image and the subsequent adjacent frame image.

[0132] Optionally, the parameter training module is further configured to: determine a loss function between the reconstructed image and the target image based on structural similarity and Manhattan distance; calculate the target loss for each pixel between the reconstructed image and the target image based on the loss function; and complete the training of the depth estimation model based on the target loss.

[0133] Optionally, the parameter training module is further configured to: convert the reconstructed image and the target image into LAB mode, and normalize the values ​​of the reconstructed image and the target image in the A channel and B channel, respectively, and scale the value of the L channel to a preset range; and determine the loss function between the reconstructed image and the target image in the LAB mode based on structural similarity and Manhattan distance.

[0134] Optionally, the parameter training module is further configured to: determine the reconstruction loss and translation loss between the reconstructed image and the target image in the LAB mode based on structural similarity and Manhattan distance; and determine the loss function between the reconstructed image and the target image in the LAB mode based on the reconstruction loss and the translation loss.

[0135] Optionally, the parameter training module is further configured to determine the reconstruction loss between the reconstructed image and the target image in the LAB mode according to the following formula:

[0136]

[0137]

[0138] Where α and β represent weights, and [i,j] represents the pixel in the i-th row and j-th column of the target image. This represents the reconstruction loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the structural similarity loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. These represent the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image in the L, A, and B channels, respectively.

[0139] Optionally, the parameter training module is further configured to determine the translation loss between the reconstructed image and the target image in the LAB mode according to the following formula:

[0140]

[0141] Where γ represents the weight, L neib,m L represents the translation loss between the reconstructed image generated using the m-th adjacent frame image and the target image. SSIM (I neib,m ,I dst ) represents the structural similarity loss between the m-th adjacent frame image and the target image. This represents the Manhattan distance loss between the m-th adjacent frame image and the target image.

[0142] Optionally, the parameter training module is further configured to: for two reconstructed images generated using the same depth map but different adjacent frame images, compare the reconstruction loss and translation loss of each pixel in the two reconstructed images, and use the smaller reconstruction loss and smaller translation loss as the reconstruction loss and translation loss between the reconstructed image generated using the depth map and the target image, respectively, and determine the loss function between the reconstructed image and the target image in the LAB mode based on the smaller reconstruction loss and smaller translation loss.

[0143] Optionally, the parameter training module is further configured to: use the smaller reconstruction loss as the target reconstruction loss, use the smaller translation loss as the target translation loss, ignore pixels where the target reconstruction loss is greater than or equal to the target translation loss error, use the target reconstruction loss based on pixels where the target reconstruction loss is less than the target translation loss error as the target loss, and determine a loss function between the reconstructed image and the target image in the LAB mode based on the target loss.

[0144] Optionally, the parameter training module is further configured to determine the target loss according to the following formula:

[0145]

[0146]

[0147] Where, loss k,m This represents the target loss between the reconstructed image generated using the k-th depth map and the target image. L represents the target reconstruction loss. neib M represents the target translation loss. k This represents the mask matrix.

[0148] Alternatively, the loss function can be determined according to the following formula:

[0149]

[0150] Where L represents the average loss and n represents the number of depth maps.

[0151] The depth estimation model training device of this invention uses a self-supervised deep learning method to estimate depth information. It has low requirements for the supporting embedded system, eliminates the need for synchronization of multiple camera times, and does not require labeled data samples, thus reducing the difficulty of model training, decreasing the steps of information extraction, and improving the efficiency of algorithm deployment. The encoding and decoding networks of the depth estimation model in this invention employ residual connections, dense connections, and channel attention structures to fully mine and utilize the feature information extracted from the image. This achieves good depth estimation accuracy with a low parameter count, achieving a good balance between model complexity and model accuracy. The loss function calculation method of this invention considers the impact of inconsistent exposure of the same object between different frames and inconsistent shadows formed by light sources on training errors. It uses a mask matrix to shield training errors caused by objects stationary relative to the camera and low-texture areas, enhancing the ability to judge depth information in the corresponding areas.

[0152] Figure 9 A schematic diagram of the structure of a depth estimation device 900 according to an embodiment of the present invention is shown, as follows: Figure 9 As shown, the depth estimation device 900 includes:

[0153] The second acquisition module 901 is used to acquire the image to be estimated;

[0154] The depth determination module 902 is used to process the image to be estimated using a pre-trained depth estimation model to obtain a depth map of the image to be estimated, and use the depth map as the depth estimation result of the image to be estimated.

[0155] Optionally, the depth determination module is further configured to: process the image to be estimated using a pre-trained depth estimation model to obtain a depth map output by the first decoding layer of the depth estimation model, and use the depth map output by the first decoding layer as the depth estimation result of the image to be estimated.

[0156] Optionally, the depth determination module is further configured to: mirror the image to be estimated to obtain a mirror image corresponding to the image to be estimated; process the image to be estimated using a pre-trained depth estimation model to obtain a first depth map of the image to be estimated; process the mirror image using the depth estimation model to obtain a second depth map of the mirror image; and determine the depth estimation result of the image to be estimated based on the first depth map and the second depth map.

[0157] Optionally, the depth determination module is further configured to: mirror and flip the second depth map to obtain a third depth map; for each pixel in the image to be estimated, determine the smaller depth in the first depth map and the third depth map, and use the smaller depth as the depth estimation result of the pixel.

[0158] The depth estimation device of this invention analyzes the image to be estimated through a depth estimation model, so that after the forward-looking monocular camera obtains a video frame, the depth information corresponding to the video frame can be quickly and accurately input. This enables the detection and estimation of the distance between the vehicle and objects such as vehicles, people, and buildings in front under different weather and road conditions, forming a depth estimation method with high accuracy, strong robustness, and real-time detection capability.

[0159] The above-described apparatus can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0160] This invention also provides an electronic device, such as... Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0161] Memory 1003 is used to store computer programs;

[0162] When the processor 1001 executes the program stored in the memory 1003, it performs the following steps: acquiring a target image and adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by a vehicle-mounted forward-looking monocular camera; constructing a depth estimation model and a pose transformation model; the depth estimation model includes a depth estimation encoding network and a depth estimation decoding network, wherein the depth estimation encoding network incorporates a dense connection mechanism and a channel attention mechanism into the residual structure of the depth estimation encoding network to fuse the shallow and deep features of the target image to obtain feature maps of the target image at multiple scales; the depth estimation decoding network... The network is used to fuse feature maps of the target image at multiple scales to obtain depth estimation information of the target image; the pose transformation model is used to obtain the pose transformation relationship between the target image and the adjacent frame images; based on the depth estimation information and the pose transformation relationship, a reconstructed image is generated; based on the reconstructed image and the target image, a loss function is constructed, and the depth estimation model is trained using the loss function; or, the following steps are implemented: obtaining the image to be estimated; processing the image to be estimated using the pre-trained depth estimation model to obtain the depth map of the image to be estimated, and using the depth map as the depth estimation result of the image to be estimated.

[0163] The communication bus 1004 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1004 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0164] The communication interface 1002 is used for communication between the aforementioned terminal and other devices.

[0165] The memory 1003 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor 1001.

[0166] The processor 1001 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0167] In another embodiment of the present invention, a computer-readable medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the depth estimation model training methods or depth estimation methods described in the above embodiments.

[0168] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the depth estimation model training methods or depth estimation methods described in the above embodiments.

[0169] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0170] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0172] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for training a depth estimation model, characterized in that, include: Acquire a target image and adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by a vehicle-mounted forward-looking monocular camera; A depth estimation model and a pose transformation model are constructed. The depth estimation model includes a depth estimation coding network and a depth estimation decoding network. The depth estimation coding network incorporates a dense connection mechanism and a channel attention mechanism into its residual structure to fuse the shallow and deep features of the target image and obtain feature maps of the target image at multiple scales. The depth estimation decoding network is used to fuse feature maps of the target image at multiple scales to obtain depth estimation information of the target image; The pose transformation model is used to obtain the pose transformation relationship between the target image and the adjacent frame images; Based on the depth estimation information and the pose transformation relationship, a reconstructed image is generated; Based on the reconstructed image and the target image, a loss function is constructed, and the depth estimation model is trained using the loss function. The step of generating a reconstructed image based on the depth estimation information and the pose transformation relationship includes: Based on the pose transformation relationship, determine the projection transformation matrix; Based on the intrinsic parameter matrix of the monocular camera and the depth estimation information, the transformation relationship between the pixel coordinate system and the camera coordinate system of the target image is determined, and based on the transformation relationship, the first coordinate of the pixel of the target image in the camera coordinate system is determined. Based on the intrinsic parameter matrix and the projection transformation matrix, the first coordinates are converted into second coordinates in the pixel coordinate system of the adjacent frame images; Based on the second coordinates, the adjacent frame images are sampled to generate a reconstructed image; The adjacent frame images include the previous adjacent frame images and the next adjacent frame images; Obtaining the pose transformation relationship between the target image and the adjacent frame images includes: Obtain the first pose transformation relationship between the target image and the previous adjacent frame image; and obtain the second pose transformation relationship between the target image and the subsequent adjacent frame image; The step of constructing a loss function based on the reconstructed image and the target image, and using the loss function to train the depth estimation model, includes: Based on structural similarity and Manhattan distance, a loss function is determined between the reconstructed image and the target image; Based on the loss function, calculate the target loss for each pixel between the reconstructed image and the target image; Based on the target loss, the depth estimation model is trained. Based on structural similarity and Manhattan distance, a loss function is determined between the reconstructed image and the target image, including: The reconstructed image and the target image are converted to LAB mode, and the values ​​of the reconstructed image and the target image in the A channel and B channel are normalized respectively, and the value of the L channel is scaled to a preset range. Based on structural similarity and Manhattan distance, a loss function is determined between the reconstructed image and the target image in the LAB mode; The loss function determined based on structural similarity and Manhattan distance between the reconstructed image and the target image in the LAB mode includes: Based on structural similarity and Manhattan distance, the reconstruction loss and translation loss between the reconstructed image and the target image in the LAB mode are determined; Based on the reconstruction loss and the translation loss, a loss function is determined between the reconstructed image and the target image in the LAB mode; The reconstruction loss between the reconstructed image and the target image in the LAB mode is determined according to the following formula: in, and They represent the weights, This represents the pixel in the i-th row and j-th column of the target image. This represents the reconstruction loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the structural similarity loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. , , These represent the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image in the L, A, and B channels, respectively.

2. The method of claim 1, wherein, The depth estimation coding network unit includes a first coding layer and multiple second coding layers; The first coding layer includes a convolutional layer, a batch normalization layer, and an activation layer; The second coding layer is a residual structure, which includes multiple third coding layers and a channel attention layer. The multiple third coding layers are connected in a dense interconnection manner.

3. The method of claim 2, wherein, The third coding layer includes a convolutional layer, a batch normalization layer, and an activation layer.

4. The method of claim 2, wherein, The method further includes: For the second coding layer, when the number of output channels of the second coding layer is inconsistent with the number of input channels of the second coding layer, convolution and normalization operations are performed on the input information of the second coding layer to make the number of input channels of the second coding layer consistent with the number of output channels of the second coding layer.

5. The method of claim 2, wherein, The depth estimation decoding network includes a first decoding layer and multiple second decoding layers; The input to the first decoding layer is the feature map output by the first encoding layer and the feature information output by the adjacent second decoding layer; The input to the second decoding layer is the feature map output by the corresponding second encoding layer and the feature information output by the second decoding layer of the previous level.

6. The method of claim 5, wherein, The first decoding layer and the second decoding layer include extraction nodes, fusion nodes and output nodes, and the fusion nodes are connected to each other and to the output nodes in a dense connection manner; The extraction node is used to obtain the feature map output by the first coding layer or the second coding layer; The fusion node is used to fuse the feature information output by the extraction nodes at the same level as well as the feature information output by the extraction nodes and fusion nodes at the next higher level. The output node is used to obtain depth maps of the target image at different scales based on the feature information output by the fusion node.

7. The method of claim 6, wherein, The method further includes: The feature information output by the output node of the first decoding layer is subjected to upsampling, convolution, and activation operations to obtain the depth map of the target image.

8. The method of claim 1, wherein, The pose transformation model includes a pose transformation encoding network and a pose transformation decoding network; The pose transformation coding network is used to obtain the pose transformation features between the target image and the adjacent frame images; The pose transformation decoding network is used to obtain the rotation angle and translation amount between the target image and the adjacent frame image based on the pose transformation features, and to use the rotation angle and the translation amount as the pose transformation relationship between the target image and the adjacent frame image.

9. The method of claim 1, wherein, The translation loss between the reconstructed image and the target image in the LAB mode is determined according to the following formula: wherein, represents a weight, represents a translation loss between a reconstructed image generated using the mth adjacent frame image and the target image, represents a structural similarity loss between the mth adjacent frame image and the target image, represents a Manhattan distance loss between the mth adjacent frame image and the target image.

10. The method of claim 9, wherein, The step of determining the loss function between the reconstructed image and the target image in the LAB mode based on the reconstruction loss and the translation loss includes: For two reconstructed images generated using the same depth map but different adjacent frame images, the reconstruction loss and translation loss of each pixel in the two reconstructed images are compared, and the smaller reconstruction loss and smaller translation loss are respectively used as the reconstruction loss and translation loss between the reconstructed image generated using the depth map and the target image. Based on the smaller reconstruction loss and smaller translation loss, a loss function is determined between the reconstructed image and the target image in the LAB mode.

11. The method of claim 10, wherein, The step of determining the loss function between the reconstructed image and the target image in the LAB mode based on the smaller reconstruction loss and the smaller translation loss includes: The smaller reconstruction loss is used as the target reconstruction loss, and the smaller translation loss is used as the target translation loss. Pixels whose target reconstruction loss is greater than or equal to the target translation loss are ignored. The target reconstruction loss based on pixels whose target reconstruction loss is less than the target translation loss is used as the target loss. Based on the target loss, a loss function between the reconstructed image and the target image in the LAB mode is determined.

12. The method of claim 11, wherein, The target loss is determined according to the following formula: wherein, denotes a target loss between a reconstructed image generated by the kth depth map and the target image, denotes a target reconstruction loss, denotes a target translation loss, denotes a mask matrix.

13. The method of claim 11, wherein, The loss function is determined according to the following formula: wherein, represents the average loss, n represents the number of depth maps.

14. A depth estimation method, characterized by, include: Obtain the image to be estimated; The image to be estimated is processed using a pre-trained depth estimation model to obtain a depth map of the image to be estimated, and the depth map is used as the depth estimation result of the image to be estimated, wherein the depth estimation model is trained by the method according to any one of claims 1-13.

15. The method of claim 14, wherein, The image to be estimated is processed using a pre-trained depth estimation model to obtain a depth map of the image to be estimated, and the depth map is used as the depth estimation result of the image to be estimated, including: The image to be estimated is processed using a pre-trained depth estimation model to obtain a depth map output by the first decoding layer of the depth estimation model, and the depth map output by the first decoding layer is used as the depth estimation result of the image to be estimated.

16. The method according to claim 14 or 15, characterized in that The image to be estimated is processed using a pre-trained depth estimation model to obtain a depth map of the image to be estimated, and the depth map is used as the depth estimation result of the image to be estimated, including: The image to be estimated is mirrored to obtain a mirror image corresponding to the image to be estimated; The image to be estimated is processed using a pre-trained depth estimation model to obtain a first depth map of the image to be estimated. The mirror image is processed using the depth estimation model to obtain a second depth map of the mirror image; Based on the first depth map and the second depth map, the depth estimation result of the image to be estimated is determined.

17. The method of claim 16, wherein, The step of determining the depth estimation result of the image to be estimated based on the first depth map and the second depth map includes: The second depth map is mirrored to obtain the third depth map; For each pixel in the image to be estimated, the smaller depth in the first depth map and the third depth map is determined, and the smaller depth is used as the depth estimation result of the pixel.

18. A depth estimation model training apparatus, comprising: include: The first acquisition module is used to acquire a target image and adjacent frame images adjacent to the target image, wherein the target image and the adjacent frame images are images captured by a vehicle-mounted forward-looking monocular camera. A construction module is used to construct a depth estimation model and a pose transformation model. The depth estimation model includes a depth estimation encoding network and a depth estimation decoding network. The depth estimation encoding network incorporates a dense connection mechanism and a channel attention mechanism into the residual structure of the depth estimation encoding network to fuse the shallow and deep features of the target image and obtain feature maps of the target image at multiple scales. The depth estimation decoding network is used to fuse feature maps of the target image at multiple scales to obtain depth estimation information of the target image; The pose transformation model is used to obtain the pose transformation relationship between the target image and the adjacent frame images; The reconstruction module is used to generate a reconstructed image based on the depth estimation information and the pose transformation relationship; The parameter training module is used to construct a loss function based on the reconstructed image and the target image, and to use the loss function to train the depth estimation model. The reconstruction module is further configured to: determine the projection transformation matrix based on the pose transformation relationship; determine the transformation relationship between the pixel coordinate system and the camera coordinate system of the target image based on the intrinsic parameter matrix of the monocular camera and the depth estimation information; and determine the first coordinate of the pixel of the target image in the camera coordinate system based on the transformation relationship. Based on the intrinsic parameter matrix and the projection transformation matrix, the first coordinates are converted into second coordinates in the pixel coordinate system of the adjacent frame images; based on the second coordinates, the adjacent frame images are sampled to generate a reconstructed image; The adjacent frame images include the previous adjacent frame images and the next adjacent frame images; The construction module is further configured to: obtain the first pose transformation relationship between the target image and the previous adjacent frame image; and obtain the second pose transformation relationship between the target image and the subsequent adjacent frame image; The parameter training module is further configured to: determine a loss function between the reconstructed image and the target image based on structural similarity and Manhattan distance; calculate the target loss for each pixel between the reconstructed image and the target image based on the loss function; and complete the training of the depth estimation model based on the target loss. The parameter training module is further configured to: convert the reconstructed image and the target image into LAB mode, and normalize the values ​​of the reconstructed image and the target image in the A and B channels respectively, and scale the value of the L channel to a preset range; and determine the loss function between the reconstructed image and the target image in the LAB mode based on structural similarity and Manhattan distance. The parameter training module is further configured to: determine the reconstruction loss and translation loss between the reconstructed image and the target image in the LAB mode based on structural similarity and Manhattan distance; and determine the loss function between the reconstructed image and the target image in the LAB mode based on the reconstruction loss and the translation loss. The parameter training module is also used to determine the reconstruction loss between the reconstructed image and the target image in the LAB mode according to the following formula: in, and They represent the weights, This represents the pixel in the i-th row and j-th column of the target image. This represents the reconstruction loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the structural similarity loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. This represents the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image. , , These represent the Manhattan distance loss between the reconstructed image generated using the k-th depth map and the m-th adjacent frame image and the target image in the L, A, and B channels, respectively.

19. An electronic device, comprising: include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-17.

20. A computer readable medium having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-17.

Images