A monocular infrared thermal image self-supervised depth estimation method based on edge constraint

Abstract

This invention discloses a self-supervised depth estimation method for monocular infrared thermal images based on edge constraints. It employs a network combining gated attention and receptive field convolutional neural networks to extract features and details from monocular infrared thermal images. Using image reconstruction techniques, the loss is calculated by comparing the difference between the reconstructed infrared thermal image and the original infrared thermal image. Simultaneously, an edge feature extraction network, EdgeNet, is used to add an edge loss, EdgeLoss, to the total loss to constrain the edge shape of objects in the predicted depth map. Experimental results show that, in low-light environments, the proposed method significantly reduces the occurrence of blurred object edge shapes in monocular infrared thermal image depth estimation tasks under insufficient ambient light conditions, and provides more accurate results for object edges in the predicted depth map.

Description

Technical Field

[0001] This invention belongs to the field of machine vision navigation and positioning, specifically relating to a self-supervised depth estimation method for monocular infrared thermal images based on edge constraints. Background Technology

[0002] In recent years, deep learning-based visual positioning and navigation technologies applied in robotics and autonomous vehicles have gradually attracted attention. Unsupervised methods, by constructing deep learning models and utilizing self-supervised signals such as disparity and image reconstruction, automatically learn geometric information in images to infer depth. This approach not only reduces data acquisition costs but also improves the model's adaptability to diverse scenarios, making it a current hot research area in depth estimation. Compared to multi-view depth estimation, monocular images lack disparity information, making accurate depth estimation difficult, especially in the absence of obvious cues. To compensate for this shortcoming, it is necessary to increase the computational cost of convolution and add attention mechanisms to selectively focus on important information.

[0003] Using thermal imaging cameras as data input instruments for visual positioning systems allows for the capture of visual information during periods of low sunlight or in adverse weather conditions, significantly improving the reliability of visual positioning capabilities for robots or other autonomous vehicles in low-light environments. However, infrared thermal images have inherent limitations; compared to RGB images, they have lower signal-to-noise ratios and contrast. Therefore, in depth estimation using monocular infrared thermal images, the edges of the predicted depth map are often more blurred than those of the predicted depth map from RGB images. Summary of the Invention

[0004] To address the aforementioned issues, this invention discloses a self-supervised depth estimation method for monocular infrared thermal images based on edge constraints. This method utilizes a gated attention-receptive field convolutional neural network, an edge extraction network, and edge loss to improve the accuracy of depth estimation at graphic edges in monocular infrared thermal images.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] A self-supervised depth estimation method for monocular infrared thermal images based on edge constraints includes the following steps:

[0007] (1) A receptive field gated attention convolutional neural network was used to extract semantic features of infrared thermal images;

[0008] (2) The Gaussian filter and the Sobel filter are connected in series and EdgeNet filters the target image and the reconstructed image respectively. The difference between them produces edge loss. Supervision improves the training network’s attention to the object edge.

[0009] (3) The structure of EdgeNet, a network for extracting edge information and segmenting and reconstructing image features.

[0010] Furthermore, the feature extraction method based on the receptive field gated attention convolutional neural network described in step (1) is expressed as follows:

[0011] O conv =W*X+b

[0012] O gate =σ(W gate *X+b gate )

[0013] O = O conv ⊙O gate

[0014] o final =RFAConv(o)

[0015] Where W is the convolution kernel, X is the input feature, b is the bias term, and W... gate It is a gated convolution kernel, b gate This is the gating bias term, * represents the convolution operation, σ is the activation function, ⊙ represents element-wise multiplication, and O conv For ordinary convolution output, O gate O is the output of the gating mechanism, O is the gating attention output, RFAConv is the receptive field convolutional neural network, and o is the output of the gating mechanism. final This is the final output of the features in the encoding process.

[0016] Specifically, the gated attention output is a gated feature map obtained by passing the gated convolution kernel and activation function. The output of the gated attention is the element-wise product of the ordinary convolution output and the gating mechanism output.

[0017] In this invention, the feature map is passed through two consecutive gated attention layers to improve the selective feature extraction effect of the attention mechanism, and the number of channels of the feature map is increased to 32 when it passes through the gated attention layer for the first time. Then, the feature map passes through a large convolutional layer and a four-layer receptive field convolutional neural network layer, with the number of output channels being 64, 64, 128, 256, and 512 respectively, and the size of the feature map decreases layer by layer. Residual connections are established after the large convolutional layer and each four-layer receptive field convolutional neural network layer. The tensors corresponding to the residual features are convolved together with the upsampled tensors along the second dimension during decoding. In the last layer of the decoding process, the disparity map is also selectively extracted and represented using two layers of gated attention.

[0018] Furthermore, step (2) is as follows:

[0019] To supervise the shape of object edges in the depth prediction map of infrared thermal images, this invention adds an edge loss, EdgeLoss, after image consistency loss and structural consistency loss.

[0020] Specifically, the process is as follows: the predicted depth map is a depth map generated from the target image through a deep network; the target image is a processed monocular infrared thermal image of the current frame; the ground truth edge map is generated by calculating the gradients of the target image through Gaussian and Sobel filters; the reconstructed image is a synthetic image where the depth map predicted by the deep network is warped into the frame of the processed source image according to the relative pose; the reconstructed edge map is generated by extracting features from the reconstructed image and segmenting it using EdgeNet. Then, useless features are removed using a mask, and finally, the squared difference between the tensors corresponding to the two feature maps is calculated. The result is the edge loss, EdgeLoss. EdgeLoss enhances the model's accuracy by forcing the model to generate more accurate depth predictions in edge regions.

[0021] Furthermore, the EdgeNet network construction method described in step (3) is as follows:

[0022] EdgeNet's overall structure resembles a U-shaped network, divided into encoding and decoding processes. During encoding, several similar downsampling processes occur. Feature maps pass through these downsampling layers first, then through a max-pooling layer, halving their length and width (essentially reducing their size to one-quarter), thus increasing the range of input information. The feature maps are then convolved with two small kernels, doubling the number of channels to capture more abstract and high-level features. During decoding, several similar upsampling processes occur. Each layer in the decoder undergoes upsampling bilinear interpolation layers, gradually restoring the spatial resolution of the feature maps to the resolution of the input image. The connection from the encoder to the decoder is a skip connection. Skip connections directly pass the feature maps of corresponding layers in the encoder to the corresponding layers in the decoder, concatenating them with the feature maps of the current decoder layer, and then convolving them with small kernels. The final output convolutional layer restores the number of channels to 1.

[0023] The number of layers in EdgeNet depends on the user's needs. If advanced and complex semantic features are not required, the number of downsampling layers and corresponding upsampling layers in EdgeNet can be reduced to save computational costs.

[0024] The beneficial effects of this invention are:

[0025] This invention presents a self-supervised depth estimation method for monocular infrared thermal images based on edge constraints. It employs a model combining gated attention and receptive field convolutional neural network structures to extract features and details from monocular infrared thermal images. Simultaneously, an edge loss, EdgeLoss, is incorporated into the total loss to constrain object edges in the predicted depth map and suppress edge blurring. The EdgeNet used in generating this edge loss effectively extracts edge features from the reconstructed image. Furthermore, this EdgeNet does not participate in the inference process, thus not increasing the push load. In low-light environments, the method proposed in this invention can significantly reduce the impact of insufficient ambient light on the accuracy of monocular infrared thermal image depth estimation and provides a more accurate representation of the edge structures in the predicted depth. Attached Figure Description

[0026] Figure 1 Edge constraint effects of different methods for depth estimation of monocular infrared thermal images: (a) Infrared thermal image (b) Depth prediction image using Shin et al.'s method (c) Prediction image with 5% edge loss added (d) Prediction image of the present invention.

[0027] Figure 2 This is a schematic diagram of the main process of the edge constraint method based on self-supervised depth estimation using monocular infrared thermal images.

[0028] Figure 3 This is a schematic diagram of the process for generating edge loss based on self-supervised depth estimation using monocular infrared thermal images.

[0029] Figure 4 A schematic diagram of the structure for EdgeNet to extract and reconstruct image edge features.

[0030] Figure 5 This is a schematic diagram of a gated attention-integrated receptive field convolutional neural network. Detailed Implementation

[0031] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0032] As shown in the figure, the self-supervised depth estimation method for monocular infrared thermal images based on edge constraints according to the present invention includes the following steps:

[0033] (1) A network structure combining gated attention and convolutional neural network is adopted, namely, receptive field gated attention convolutional neural network to extract infrared thermal image features;

[0034] (2) The target thermal image is filtered by a Gaussian filter and a Sobel filter in series to generate a real edge map; EdgeNet is used to filter the reconstructed image to generate a reconstructed edge map; the difference between the two edge feature maps is used as the edge loss EdgeLoss.

[0035] (3) The structure of EdgeNet, a network for extracting edge information and segmenting and reconstructing image features.

[0036] This invention uses the indoor portion of the ViViD dataset as both the training and testing sets to validate the proposed edge-constrained self-supervised depth estimation method for monocular infrared thermal images. The experimental platform consisted of Ubuntu 22.04, Python 3.7.6, torch 1.11.0, torchvision 0.12.0, torchaudio 0.11.0, and an NVIDIA GPU 12G with CUDA 11.8.

[0037] The ViViD dataset consists of 10 indoor sequences and 4 outdoor sequences under different lighting and motion conditions, including raw infrared thermal images, ground truth (GT) depth maps, and GT pose labels.

[0038] Table 1 is a statistical table showing the impact of adding edge losses with different weights on the depth estimation error and accuracy of monocular infrared thermal images.

[0039] Table 1 shows the accuracy and error of depth prediction for monocular infrared thermal images under different lighting conditions on the ViViD dataset, with edge losses of weights of 0.04, 0.05, 0.06, and 0.1.

[0040]

[0041] Experimental results show that when the added EdgeLoss weight is around 0.05, the model's accuracy reaches its maximum and the error reaches its minimum.

[0042] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A self-supervised depth estimation method for monocular infrared thermal images based on edge constraints, characterized in that: Includes the following steps: (1) A network structure combining gated attention and convolutional neural network is adopted, namely, receptive field gated attention convolutional neural network to extract infrared thermal image features; (2) The target thermal image is filtered by a Gaussian filter and a Sobel filter in series to generate a real edge map; EdgeNet is used to filter the reconstructed image to generate a reconstructed edge map; the difference between the two edge feature maps is used as the edge loss EdgeLoss. Specifically as follows: The monocular infrared thermal image of the current frame is processed to form the target image. The true edge map is generated by the gradients calculated from the target image through Gaussian and Sobel filters. The predicted depth map is a depth map generated from a target image through a deep network; The reconstructed image is a synthetic image that renders the predicted depth map into the frame of the source thermal image based on the relative pose of the target image and the source image. The reconstructed edge map is generated by extracting and segmenting the reconstructed image through EdgeNet. Useless features are removed by masking, and finally the squared difference between the corresponding tensors of the two images is calculated. The result is the edge loss, EdgeLoss. EdgeLoss enhances the accuracy of the model by forcing the model to generate more accurate depth predictions in edge regions. (3) The structure of EdgeNet, a network for extracting edge information and segmenting and reconstructing image features; The method for building the EdgeNet network is as follows: EdgeNet's overall structure is similar to a U-shaped network, consisting of encoding and decoding processes. During the encoding process, there are several similar downsampling processes. When the feature map passes through these downsampling layers, it first goes through a max pooling layer, which halves the length and width of the feature map, making the feature map about 1 / 4 of its original size, thereby increasing the range of input information. Then, the feature map is convolved with two small kernels, during which the number of channels is doubled, thereby capturing more abstract and high-level features. During the decoding process, there are several similar upsampling processes; each layer in the decoder goes through a bilinear interpolation layer for upsampling, so that the spatial resolution of the feature map is gradually restored to the resolution of the input image; the link from the encoder to the decoder is a skip connection; the skip connection directly passes the feature map of the corresponding layer in the encoder to the corresponding layer in the decoder, and concatenates it with the feature map of the current layer in the decoder, and then convolves it through a small convolution kernel; the final output convolutional layer restores the number of channels to 1.

2. The self-supervised depth estimation method for monocular infrared thermal images based on edge constraints according to claim 1, characterized in that: The feature extraction method based on the receptive field gated attention convolutional neural network described in step (1) is expressed as follows: ； ； ； ； in It is a convolution kernel. These are input features. It is a bias term. It is a gated convolution kernel. This is the gating bias term; * indicates the convolution operation. It is an activation function. Represents element-wise product. This is a normal convolution output. For gating mechanism output, For gating attention output, For receptive field convolutional neural networks, This is the final output of the features in the encoding process.

Images