A method and device for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency

By extracting pixels with ambiguous boundaries in the object in the image and processing them using an adaptive blur module, the photometric loss function is corrected to make it more fairly evaluate the depth prediction in high-frequency areas, solving the problem that the photometric loss function cannot be fairly evaluated in the prior art. , improves the performance of the monocular depth estimation model.

CN114998410BActive Publication Date: 2024-11-12PEKING UNIV SHENZHEN GRADUATE SCHOOL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210392984.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-15
Publication Date
2024-11-12
Estimated Expiration
2042-04-15

AI Technical Summary

Technical Problem

The photometric reprojection loss function of the existing self-supervised monocular depth estimation model cannot fairly evaluate the depth prediction value in the high-frequency region of the image, resulting in limited performance improvement.

Method used

By extracting pixels with ambiguous boundaries in the object in the image and processing them using the boundary blur mask module, combined with the adaptive blur module to process pixels in the high-frequency region, the photometric loss function is corrected so that it can more accurately evaluate depth predictions at different spatial frequencies.

Benefits of technology

The performance of the self-supervised monocular depth estimation model is improved, the accuracy of depth estimation is enhanced, and the fairness and accuracy of the photometric loss function are improved while keeping the inference time unchanged.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114998410B_ABST
    Figure CN114998410B_ABST
Patent Text Reader

Abstract

The method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency of the present invention comprises the steps of: S1. extracting and masking ambiguous pixels at the boundary of an object in an image; S2. processing the input image obtained after masking in step S1 with an adaptive blur module; and S3. replacing the original image with the image processed in steps S1 and S2 to participate in the training process of an existing monocular depth estimation model. The method of the present invention can be universally applied to almost all monocular depth estimation algorithms based on self-supervised learning, and it examines the influence of regions of different spatial frequencies of an image on the learning process of a self-supervised depth estimation model, making the loss function more robust; the method improves the accuracy of a large number of existing open source self-supervised monocular depth estimation models in the community while keeping the inference time completely unchanged.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of monocular image depth estimation, and more specifically to a method and device for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency. Background Art

[0002] Self-supervised monocular depth estimation has gained widespread attention in recent years because it only requires a monocular camera instead of expensive radar, RGB cameras and other equipment. Specifically, given a target image, the monocular depth estimation model calculates its dense depth map, and then, through other reference frames of the same scene at different shooting angles, and a known (or estimated) relative camera pose, the reconstruction map can be generated by sampling the reference frame. The quality of depth prediction during model training is evaluated by the pixel-by-pixel photometric distance between the target image and the reconstructed image.

[0003] The main problems and defects that exist now are: although the performance of self-supervised monocular depth estimation is getting closer and closer to the supervised depth estimation algorithm, some defects of this photometric reprojection loss function limit its further performance improvement. The anti-aliasing technology used to smooth the boundaries of objects in the image brings unreasonable photometric loss to the depth estimation model; the photometric loss function cannot fairly evaluate the depth prediction value in the high-frequency area of ​​the image. Due to the universality of this defect, designing a general, efficient and generalized method has become the main difficulty in solving the problem.

[0004] The significance of solving the above problems and defects is that the accuracy of the photometric loss function is affected by the different spatial frequencies of the image. Correcting this effect so that the photometric loss function can more fairly and accurately evaluate depth prediction will further improve the existing self-supervised monocular depth estimation models in general. Summary of the invention

[0005] The present invention provides a method and device for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, corrects the inherent defects of the photometric loss function, and further enhances the performance of a large number of existing self-supervised monocular depth estimation models in the current community.

[0006] The technical solution of the present invention is as follows:

[0007] According to one aspect of the present invention, a method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency is provided, comprising the following steps: S1. extracting and masking ambiguous pixels at the boundary of an object in an image; S2. processing the input image obtained after masking in step S1 with an adaptive blur module; and S3. replacing the original image with the image processed in steps S1 and S2 to participate in the training process of an existing monocular depth estimation model.

[0008] Preferably, in the above-mentioned method for improving the performance of the self-supervised monocular depth estimation model based on spatial frequency, in step S1, a boundary fuzzy mask module is used to extract and mask ambiguous pixels at the boundary of objects in the image to eliminate the adverse effects of anti-aliasing technology on the depth estimation of object boundaries.

[0009] Preferably, in the above-mentioned method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, in step S1, given an input target frame I t , first calculate the horizontal u and vertical v spatial frequencies of the pixel at i,j pixel by pixel As shown in formula (1-4):

[0010]

[0011] The mask μ is set to extract the pixels whose gradients in opposite directions have different signs, as shown in formula (5), where [·] is the Iverson bracket,

[0012]

[0013] Initial boundary blur mask A t Calculated by formula (6):

[0014]

[0015] Source frame I t+n , calculate its blur mask A according to the same method t+n ; Then the depth map D estimated by the depth network of the depth estimation model t And the camera pose T is obtained according to the pose network of the depth estimation model t+n→t , will I t Each pixel in is projected onto I t+n , calculate the sampler when generating the reconstructed frame Then the blur mask of the reconstructed frame is obtained by bilinear sampling Final blur mask for photometric reprojection loss (pe) After taking the maximum value (max) pixel by pixel of the fuzzy mask of the target frame and the reconstructed image, the threshold value δ is obtained, as shown in formula (7):

[0016]

[0017] Preferably, in the above-mentioned method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, in step S2, pixels located in the high-frequency area of ​​the input image are first identified, and formula (3) is used as the spatial frequency High frequency points are defined as points where the frequency is greater than the threshold λ:

[0018]

[0019] Then Do an average pooling with a stride of 1 and use the Iverson bracket again to determine whether the pixel is in the high frequency region:

[0020]

[0021] Where s is the kernel size of the average pooling, and pixel q belongs to the s×s neighborhood of p;

[0022] Finally, the adaptive blurred image is calculated by the following formula:

[0023]

[0024] where w blur (p) is the weight to balance the original image and the Gaussian blurred image, It is a Gaussian blurred image.

[0025] Preferably, in the above-mentioned method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, in step S3, the image processed by steps S1 and S2 is used to participate in the calculation process of the photometric loss, instead of the original image participating in the training of the existing monocular depth estimation model, so that the monocular depth estimation model finally achieves a better depth estimation accuracy. The photometric loss function corrected by the above operation is as follows:

[0026]

[0027] According to another aspect of the present invention, there is provided a device for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, comprising: a boundary fuzzy mask module and an adaptive fuzzy module, wherein: the boundary fuzzy mask module is used to extract and mask ambiguous pixels at the boundary of an object in an image to eliminate the adverse effects of anti-aliasing technology on the depth estimation of the object boundary, and remove pixels that bring unreasonable photometric reprojection losses to the model in dense depth estimation tasks; the adaptive fuzzy module is used to process the input image obtained after masking, so that the photometric loss function can more fairly evaluate the predicted depth in the high-frequency area of ​​the image, while revealing mismatched pixels in the high-frequency area of ​​the image.

[0028] According to the technical solution of the present invention, the beneficial effects produced are:

[0029] The method of improving the performance of the self-supervised monocular depth estimation model based on spatial frequency of the present invention examines the influence of pixels of different spatial frequencies on the accuracy of this loss function, as most existing self-supervised monocular depth estimation methods use the photometric loss function to optimize the network. The present invention proposes a fuzzy mask to avoid pixel interference at the object boundary for anti-aliasing in the training of the depth estimation network, and an adaptive fuzzy module to make the photometric loss function more fair and accurate in the high-frequency area of ​​the image. It provides substantial performance improvement for a large number of self-supervised monocular depth estimation algorithms currently open source in the community, while keeping their inference time completely unchanged.

[0030] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0032] Figure 1 It is a flow chart of a method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency of the present invention;

[0033] Figure 2 It is a specific operation flow chart of the method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency of the present invention;

[0034] Figure 3 This is an analysis of the unreasonable loss caused by anti-aliased pixels at the boundaries of image objects to the depth estimation model;

[0035] Figure 4 It is a diagram illustrating the effect of the adaptive fuzzy module proposed in the present invention. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0037] like Figure 1 and Figure 2 As shown, the method for improving the accuracy of a self-supervised monocular depth estimation model based on spatial frequency of the present invention comprises the following steps:

[0038] S1. Extract and mask the pixels with ambiguous object boundaries in the image.

[0039] This step corresponds to Figure 2The boundary blur mask module in is used to eliminate the adverse effects of anti-aliasing technology on object boundary depth estimation, that is, to remove pixels that bring unreasonable photometric reprojection loss to the model in the dense depth estimation task.

[0040] Given an input target frame I t , first calculate the horizontal (u) and vertical (v) spatial frequencies of each pixel (such as the pixel at i, j) As shown in formula (1-4).

[0041]

[0042] Since these pixels are used to smooth the color mutation at the edge of the object, so that it changes from abrupt to gradual, their color values ​​should be the weighted sum of the surrounding pixels. Therefore, the mask μ is set to extract the pixels with opposite signs of the gradient in the opposite direction, as shown in formula (5), where [·] is the Iverson bracket.

[0043]

[0044] Initial boundary blur mask A t Calculated by formula (6).

[0045]

[0046] Source frame I t+n , calculate its blur mask A according to the same method t+n Then, the depth map D estimated by the depth network of the depth estimation model t And the camera pose T is obtained according to the pose network of the depth estimation model t+n→t , will I t Each pixel in is projected onto I t+n , calculate the sampler when generating the reconstructed frame Then the blur mask of the reconstructed frame is obtained by bilinear sampling Final blur mask for photometric reprojection loss (pe) After taking the maximum value (max) pixel by pixel of the blur mask between the target frame and the reconstructed image, the value is obtained through the threshold value δ, as shown in formula (7).

[0047]

[0048] S2. Process the input image obtained after masking in step S1 using the adaptive blur module.

[0049] This step is used to make the photometric loss function more fairly evaluate the predicted depth in the high-frequency areas of the image, while revealing mismatched pixels in the high-frequency areas of the image.

[0050] In this step, we first identify the pixels located in the high-frequency region of the input image. For simplicity, we use formula (3) as the spatial frequency High frequency points are defined as points where the frequency is greater than the threshold λ:

[0051]

[0052] Then Do an average pooling with a stride of 1 and use the Iverson bracket again to determine whether the pixel is in the high frequency region:

[0053]

[0054] where s is the kernel size of the average pooling and pixel q belongs to the s×s neighborhood of p.

[0055] Finally, the adaptive blurred image is calculated by the following formula:

[0056]

[0057] where w blur (p) is the weight to balance the original image and the Gaussian blurred image, It is a Gaussian blurred image.

[0058] S3. Replace the original image with the image processed by steps S1 and S2 to participate in the training process of the existing monocular depth estimation model.

[0059] This step is used to finally eliminate the adverse effects of different spatial frequencies on the loss function. The images processed by S1 and S2 are used to participate in the calculation process of photometric loss, replacing the original images in the existing monocular depth estimation model (such as Figure 2 The training of the monocular depth estimation model finally achieves better depth estimation accuracy.

[0060] The photometric loss function corrected by the above operation is as follows:

[0061]

[0062] exist Figure 3 In the figure, (a) is the RGB image input during training; (b) is the loss graph of the 8th round of training. For most objects, the loss only occurs in the boundary area. (c) Image block shows that the pixels on the image boundary are smoothly transitioned, but these pixels are ambiguous, that is, they are neither from the black chimney nor from the white sky; (d) shows the comparison of the boundary. The real image boundary is abrupt and not composed of discrete pixels. There are no objects in the real world that show such ambiguous colors, so the model cannot correctly estimate its depth, which ultimately leads to a large loss.

[0063] The device for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency includes: a boundary blur mask module and an adaptive blur module, wherein:

[0064] The boundary fuzzy mask module is used to extract and mask the ambiguous pixels at the boundary of objects in the image to eliminate the adverse effects of anti-aliasing technology on the depth estimation of object boundaries and remove pixels that bring unreasonable photometric reprojection losses to the model in dense depth estimation tasks;

[0065] The adaptive blur module processes the masked input image so that the photometric loss function can more fairly evaluate the predicted depth in the high-frequency areas of the image, while revealing mismatched pixels in the high-frequency areas of the image.

[0066] Figure 4 It is shown that the adaptive blur module helps improve the unfairness of the photometric reprojection loss function when evaluating depth prediction in high-frequency areas of the image. Figure 4 In the figure, the first row is the training image with or without the adaptive blur module, (a) is the input image, (b) is the crop of the view (stretched), and c is the image processed by adaptive blur. The second row: d is the L1 loss of depth used by the self-supervised method, without adaptive blur and the supervised method. The middle figure e (∝: proportional to) shows that the maximum L1+L ssim The parallax of photometric loss is more accurate than the parallax of minimum photometric loss; the photometric loss of the true match is even greater than the loss of the incorrect match, while the self-supervised method enhanced by adaptive blur does not suffer from this misjudgment; Figure f on the right is a qualitative analysis of the relationship between network penalty and prediction accuracy. The positive proportional relationship presented by the supervised method is the most ideal. With adaptive blur, the loss function of the self-supervised method becomes more stable and closer to the supervised function. In addition, in the figure on the left, it presents a more "V" shape than the non-adaptive blur method, indicating a more obvious positive proportional relationship.

[0067] All indicators have not been post-processed (PP). The Data column indicates the input data for model training: S-stereoscopic image pair; M-monocular video; MS-stereoscopic video. The meanings of indicators from left to right are: absolute error, mean square error, root mean square error, logarithmic root mean square error, and prediction-to-true value ratio are less than 1.25 respectively. 1 , 1.25 2 , 1.25 3 As shown in Table 1, the model enhanced by the method of the present invention achieves stronger performance in almost all indicators.

[0068] Table 1 Comparison of the results of embedding the proposed method into a large number of self-supervised monocular depth estimation models in the current community

[0069]

[0070] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

Claims

1. A method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, characterized in that: The following steps are involved: S1. Extract and mask the ambiguous pixels of the object boundary in the image. Given the input target frame I t , first calculate the horizontal u and vertical v spatial frequencies of the pixel at i,j pixel by pixel As shown in formula (1-4): The mask μ is set to extract the pixels whose gradients in opposite directions have different signs, as shown in formula (5), where [·] is the Iverson bracket, Initial boundary blur mask A t Calculated by formula (6): Source frame I t+n , calculate its blur mask A according to the same method t+n ; Then the depth map D estimated by the depth network of the depth estimation model t And the camera pose T is obtained according to the pose network of the depth estimation model t+n→t , will I t Each pixel in is projected onto I t+n , calculate the sampler when generating the reconstructed frame Then the blur mask of the reconstructed frame is obtained by bilinear sampling Final blur mask for photometric reprojection loss After taking the maximum value of the blur mask of the target frame and the reconstructed image pixel by pixel, the threshold value δ is used to obtain the value, as shown in formula (7): S2. Use the adaptive blur module to process the input image obtained after the mask in step S1. First, identify the pixels located in the high-frequency area of ​​the input image and use formula (3) as the spatial frequency High frequency points are defined as points where the frequency is greater than the threshold λ: Then Do an average pooling with a stride of 1 and use the Iverson bracket again to determine whether the pixel is in the high frequency region: Where s is the kernel size of the average pooling, and pixel q belongs to the s×s neighborhood of p; Finally, the adaptive blurred image is calculated by the following formula: where w blur (p) is the weight to balance the original image and the Gaussian blurred image, is a Gaussian blurred image; as well as S3. Replace the original image with the image processed by steps S1 and S2 to participate in the training process of the existing monocular depth estimation model.

2. The method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency according to claim 1, characterized in that: In step S1, a boundary fuzzy mask module is used to extract and mask the pixels with ambiguous boundaries of objects in the image.

3. The method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency according to claim 1, characterized in that: In step S3, the image processed by steps S1 and S2 is used to participate in the calculation process of photometric loss, and the original image is used to participate in the training of the monocular depth estimation model, so that the monocular depth estimation model can achieve better depth estimation accuracy. The corrected photometric loss function is as follows:

4. A device for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency, used to implement the method for improving the performance of a self-supervised monocular depth estimation model based on spatial frequency according to any one of claims 1 to 3, characterized in that: include: Boundary fuzzy mask module and adaptive fuzzy module, where: The boundary fuzzy mask module is used to extract and mask ambiguous pixels at the boundary of objects in the image to eliminate the adverse effects of anti-aliasing technology on the depth estimation of the object boundary and remove pixels that bring unreasonable photometric reprojection loss to the model in the dense depth estimation task; The adaptive blur module is used to process the input image obtained after masking so that the photometric loss function can more fairly evaluate the predicted depth in the high-frequency area of ​​the image, while revealing the mismatched pixels in the high-frequency area of ​​the image.