A dense depth image computation method, system and electronic device

Abstract

This application relates to a dense depth image calculation method, system, and electronic device, belonging to the field of computer vision technology. The method includes: fusing visible light images and thermal infrared images to generate a multimodal fused image; processing a 3D point cloud to obtain a sparse depth image; inputting the sparse depth image and the multimodal fused image into a dense depth image calculation network; the dense depth image calculation network is a convolutional neural network based on an encoder-decoder structure, wherein the encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of the sparse depth image and the single multimodal fused image; the decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output a dense depth image. This method has the advantages of improving depth calculation accuracy in complex environments, adapting to harsh weather and occlusion conditions, and enhancing 3D perception capabilities.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a dense depth image calculation method, system, and electronic device. Background Technology

[0002] 3D inspection of power corridors is one of the core technologies for maintaining the normal operation of transmission lines and improving the intelligence level of the power grid. Due to cost and size limitations, the fixed monitoring equipment used for power corridor condition monitoring can usually only use low-line-count LiDAR. These LiDARs have low resolution and are difficult to acquire complete 3D point clouds, thus requiring algorithms to calculate dense 3D point clouds. One of the core technologies for dense 3D point cloud reconstruction is dense depth calculation. Given a depth image and corresponding camera intrinsic and extrinsic parameters, a 3D point cloud can be generated by back-projecting the depth image into 3D space. After spatial alignment of the visible light camera, thermal infrared camera, and LiDAR, projecting the LiDAR point cloud onto a 2D image can generate an initial depth map. Because the resolution of LiDAR point clouds is generally low, these initial depth maps are very sparse, especially lacking effective depth points for fine structures in the scene. Compared to LiDAR point clouds, visible light images and thermal infrared images have higher resolution and can more completely observe the fine structures in the scene.

[0003] Therefore, it is necessary to leverage the complementary advantages of multimodal data to compute dense depth images. Currently, dense depth estimation methods based on multi-sensor fusion are mostly applied in fields such as autonomous driving and augmented reality. These methods generally use visible light images as reference information and employ convolutional neural networks to diffuse the sparse depth projected by the LiDAR to surrounding pixels, thereby generating a dense depth map. However, these methods can only work under normal weather and sufficient lighting conditions. To support all-weather 3D perception, it is necessary to fuse thermal infrared images and visible light images to provide a reference for depth map densification. Image fusion technology aims to integrate complementary information from different source images into a single fused image, and use this multimodal fused image to comprehensively characterize the imaging scene and apply it to subsequent visual tasks. The fusion of infrared and visible light images can integrate the advantages of both types of images, taking into account rich texture details, ambient lighting, and other environmental information, while also providing some scene information under extreme conditions, severe weather, and partial occlusion. Existing technologies lack a complete solution that can effectively fuse multimodal image features and achieve high-precision dense depth computing. In particular, in special application scenarios such as power systems, there is an urgent need to develop depth computing technologies that can adapt to complex environmental conditions. Summary of the Invention

[0004] The purpose of this application is to provide a dense depth image calculation method, system and electronic device, which has the advantages of improving the depth calculation accuracy in complex environments, adapting to harsh weather and occlusion conditions, and enhancing three-dimensional perception capabilities.

[0005] This application provides a method for calculating dense depth images, the technical solution of which is as follows: A multimodal fused image is generated by fusing visible light images and thermal infrared images; The process generates a sparse depth image from the 3D point cloud. The sparse depth image and the multimodal fused image are input into a dense depth image computing network. The dense depth image computing network is a convolutional neural network based on an encoder-decoder structure. The encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of the sparse depth image and the single multimodal fused image. The decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output a dense depth image.

[0006] Furthermore, the process of fusing visible light images and thermal infrared images to generate a single multimodal fused image includes: inputting a set of calibrated visible light images and thermal infrared images into a visible light-thermal infrared image fusion network; the visible light-thermal infrared image fusion network includes a feature extraction module, an edge feature fusion module, and an image reconstruction module; The feature extraction module extracts depth features containing fine structures from the visible image and the thermal infrared image respectively, obtaining visible light features and thermal infrared features; the edge feature fusion module extracts high-level features from the visible light features and thermal infrared features, and fuses the complementary information between the visible light features and thermal infrared features, as well as the high-frequency information of the visible light features and thermal infrared features, outputting multi-layer features that simultaneously contain multi-scale semantic information and high-frequency edge details; the image reconstruction module stitches the multi-layer features together, and uses image reconstruction methods to generate and output a visible light-thermal infrared multimodal fusion image.

[0007] Furthermore, given the input of the edge feature fusion module EFFM Its output is:

[0008] in, Indicates a convolutional layer. The cascaded first One convolutional layer; symbol This represents the gradient operator, treating the convolution kernel as a manually configured convolution operation, using a CEDN convolutional neural network. This indicates element-wise addition; The edge feature fusion module EFFM represents the first... i Secondary input features; : Indicates the edge feature fusion module EFFM processing The resulting output features are then passed on to the next... i Characteristics of the +1 stage; Indicates input features Edge features obtained after gradient calculation.

[0009] Furthermore, visible light features and thermal infrared features are extracted through two branches respectively; the basic convolutional layer of the edge feature fusion module uses a 3×3 convolution and a leaky linear rectified unit as the activation function to extract shallow features; The edge feature fusion module is used to extract detailed features; The image reconstruction module consists of three cascaded 3×3 convolutional layers and one 1×1 convolutional layer, with activation functions LReLU and Tanh, respectively.

[0010] Furthermore, processing the 3D point cloud to obtain a sparse depth image involves projecting the 3D point cloud onto an image plane to obtain the sparse depth image.

[0011] Furthermore, the multimodal fused image and the sparse depth image are stitched together and fed into the encoder; The encoder includes multiple coding blocks of different sizes, each of which is composed of three 3×3 convolutions. To preserve detailed features during encoding, the encoder in this application employs a multi-scale feature fusion module (MSFF) to adaptively aggregate global context information. The MSFF module integrates features from three different scales using upprojection and channel concatenation. For each encoding block, the output feature maps of the other three scale encoding blocks are upsampled by 2x, 4x, and 8x respectively. The final output of each encoding block contains 8 channels. Upsampling is performed through channel concatenation, and finally, convolutional layers are used to connect and further transform the channels to obtain the residual depth map and the predicted depth map. The output of the MSFF module is shown below. The encoder outputs a residual depth map and a predicted depth map. Furthermore, considering that residual depth images often appear at depth discontinuities, which mostly overlap with image edges, image edges are extracted from the multimodal fused image. Here, the Canny operator or a convolutional neural network is used, and the calculated image edges are concatenated with the corresponding fused feature maps to generate the residual depth image. If an edge extraction convolutional neural network is used, the network needs to be trained in advance, and the corresponding weights should be fixed when training the dense depth image computation network. The decoder includes four Guided Up-sampling Blocks (GUBs). The inputs of each GUB are the downsampled multimodal fusion image, the residual depth image generated by the multi-scale features extracted by the MSFF module, and the depth image predicted by the previous scale. These three inputs are fused and processed through a series of steps to generate a fine depth image at the current scale.

[0012] This application also proposes a dense depth image calculation system, based on the above-mentioned dense depth image calculation method, including: The fusion unit is used to fuse visible light images and thermal infrared images to generate a multimodal fused image; The processing unit is used to process the 3D point cloud to obtain a sparse depth image. The prediction unit is used to input the sparse depth image and the multimodal fused image into the dense depth image computing network. The dense depth image computing network is a convolutional neural network based on an encoder-decoder structure. The encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of the sparse depth image and the single multimodal fused image. The decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output the dense depth image. This application also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described dense depth image calculation method.

[0013] This application also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described dense depth image calculation method.

[0014] This application also proposes a computer program product comprising computer instructions that instruct a computer to execute the above-described dense depth image calculation method.

[0015] As can be seen from the above, the dense depth image calculation method and system, electronic device, storage medium and computer program product provided in this application generate a multimodal fused image by fusing visible light and thermal infrared images, combine it with a sparse depth image generated by a 3D point cloud, use an encoder with a multi-scale residual pyramid structure to perform cross-modal feature fusion, and output a high-precision dense depth image through a decoder with an integrated guided upsampling module. This effectively solves the problem of missing depth information in complex environments and has the advantages of improving calculation accuracy, adapting to harsh environments and enhancing 3D perception capabilities. Attached Figure Description

[0016] Figure 1 A flowchart of a dense depth image calculation method provided in this application; Figure 2 A flowchart of dense depth calculation guided by multimodal fusion image provided for this application; Figure 3 A schematic diagram of the visible light-thermal infrared image fusion network structure provided in this application; Figure 4 A schematic diagram of the dense depth image computation network structure provided in this application; Figure 5 A schematic diagram of the guided upsampling module provided in this application; Figure 6 A schematic diagram of a dense depth image calculation system provided in this application. Detailed Implementation

[0017] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0018] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0019] In existing technologies, 3D detection of power corridors typically relies on low-line-count LiDAR to acquire 3D point cloud data. Due to equipment resolution limitations, the generated depth images suffer from sparsity, especially in complex structural regions where depth information is severely lacking. Traditional methods use visible light images to guide depth calculations, but under conditions of fog, rain, snow, or low light, the quality of visible light images deteriorates sharply, leading to the failure of depth estimation. Existing fusion algorithms are mostly designed for autonomous driving scenarios, failing to consider the all-weather monitoring needs of power corridors, and lack an effective fusion mechanism for cross-modal features, making it difficult to balance environmental adaptability with depth reconstruction accuracy.

[0020] To address the aforementioned issues, the inventors discovered that single sensors have perceptual limitations. Visible light and thermal infrared images possess complementary characteristics; the former preserves texture details, while the latter provides scene information in harsh environments. By fusing these two modalities to generate a unified representation, an all-weather reference can be provided for depth calculation. Simultaneously, LiDAR sparse depth maps contain geometric priors, but require diffusion based on image features. Traditional encoder structures cannot effectively integrate cross-modal features, leading to the loss of high-frequency details. Therefore, a multi-scale feature fusion mechanism needs to be designed, combined with guided upsampling techniques to gradually recover depth details.

[0021] like Figure 1 As shown, this application proposes a dense depth image calculation method, including: fusing visible light images and thermal infrared images to generate a multimodal fused image; processing three-dimensional point clouds to generate a sparse depth image; inputting the sparse depth image and the multimodal fused image into a convolutional neural network based on an encoder-decoder structure, wherein the encoder uses a multi-scale residual pyramid structure to encode and fuse cross-modal features, and the decoder uses a residual pyramid structure to integrate a guided upsampling module to predict and output the dense depth image.

[0022] The fusion of visible light and thermal infrared images refers to generating a single multimodal fused image through feature extraction, edge feature fusion, and image reconstruction steps. Specifically, this can be achieved using a fusion network comprising a feature extraction module, an edge feature fusion module, and an image reconstruction module. This process integrates the high-frequency details of visible light with the environmental robustness of thermal infrared. The multi-scale residual pyramid structure refers to using multiple coding blocks of different scales in the encoder. Each coding block consists of multiple convolutional layers, aggregating global contextual information through upsampling and channel cascading. For example, three 3×3 convolutional layers can be used to construct the coding block, achieving multi-level fusion of cross-modal features. The guided upsampling module refers to the joint processing of multimodal fused image features, residual depth images, and predicted depth maps during decoding. The upsampling process is optimized through channel attention mechanisms, such as using a Squeeze-and-Excitation network to adjust feature weights, ensuring that edge structures are preserved during depth reconstruction.

[0023] Among them, the Squeeze-and-Excitation (SE) network is a lightweight attention mechanism module focused on improving the feature learning ability of convolutional neural networks. By explicitly modeling the dependencies between channels, the network can adaptively focus on important feature channels and suppress useless or redundant features, thereby improving model performance.

[0024] Specifically, visible light and thermal infrared images are fused by a fusion network to generate a multimodal fused image containing complementary information, and a sparse depth map is generated by 3D point cloud projection as a geometric constraint. The two images are then stitched together and input into an encoder. A multi-scale residual pyramid structure extracts features through different receptive fields, fusing cross-modal information at multiple scales. The encoder outputs a residual depth map and a predicted depth map, combining edge features extracted from the multimodal fused image to enhance the representation of depth discontinuities. The decoder recovers depth details step-by-step through a guided upsampling module. At each upsampling stage, the spatial features of the multimodal fused image guide depth value diffusion, and the residual pyramid structure progressively corrects prediction errors, ultimately outputting a high-resolution, dense depth image.

[0025] Compared to existing technologies, traditional methods rely solely on visible light images to guide depth calculations, which are prone to failure in low light or inclement weather. This solution, however, fuses thermal infrared images to provide all-weather scene representation. Existing encoders employ single-scale convolutional layers, making it difficult to simultaneously capture global context and local details. This solution achieves multi-level fusion of cross-modal features through a multi-scale residual pyramid. Traditional decoders use bilinear interpolation upsampling, leading to edge blurring. This solution introduces a guided upsampling module, utilizing high-frequency features from the multimodal fused image to optimize the depth reconstruction process.

[0026] Through the above technical solution, this application solves the problem of structural loss when generating depth images using low-line-count lidar, and improves the robustness of depth calculation in complex environments by utilizing the complementary characteristics of multimodal images. The encoder's multi-scale feature fusion mechanism effectively integrates geometric priors and image features, while the decoder's guided upsampling technique ensures accurate recovery of depth details. This method can still generate accurate dense depth images under harsh conditions such as fog, rain, and snow, meeting the all-weather monitoring needs of power corridors.

[0027] This application further proposes a method for generating a single multimodal fused image by fusing visible light images and thermal infrared images, including: A set of calibrated visible light and thermal infrared images are input into a visible light-thermal infrared image fusion network, which consists of a feature extraction module, an edge feature fusion module, and an image reconstruction module. The feature extraction module extracts depth features containing fine structures from the visible light and thermal infrared images respectively, obtaining visible light features and thermal infrared features. The edge feature fusion module extracts high-level features from the visible light and thermal infrared features, fuses complementary and high-frequency information, and outputs multi-layer features containing multi-scale semantic information and high-frequency edge details. The image reconstruction module stitches the multi-layer features to generate a visible light-thermal infrared multimodal fused image.

[0028] The feature extraction module employs a dual-branch convolutional neural network to process visible light and thermal infrared images separately. Specifically, it can be implemented using multiple 3×3 convolutional layers connected in series with leaky linear rectified units (CRUs) to independently capture the fine texture and structural features of both modalities. The edge feature fusion module integrates complementary information from cross-modal high-level features. This can be achieved by extracting high-frequency edge features using gradient operators and superimposing them with convolutional features, for example, using a CEDN (Convolutional Encoder-Decoder Network) convolutional neural network for high-frequency information fusion. The image reconstruction module converts multi-scale fused features into a decoding structure for a multi-modal fused image. This can be implemented using cascaded 3×3 convolutional layers and the Tanh activation function to restore global illumination consistency while preserving details. Tanh (hyperbolic tangent) is a commonly used non-linear activation function in neural networks. CEDN is an encoder-decoder architecture based on convolutional neural networks, initially used for image segmentation and contour detection tasks.

[0029] Specifically, after spatial alignment, the visible light and thermal infrared images are input into a dual-branch feature extraction module. Modality-specific features are extracted through independent convolutional paths, avoiding information confusion caused by early fusion. The edge feature fusion module performs gradient feature extraction and cross-modal fusion on the two high-level features, enhancing the integrity of edge details through complementary high-frequency information. After multi-scale feature stitching, the image reconstruction module progressively upsamples the images, reconstructing a multimodal fused image with both semantic integrity and geometric accuracy while suppressing noise. This process, through a staged feature processing mechanism, effectively integrates visible light environmental information and thermal infrared thermal radiation features even under extreme lighting conditions.

[0030] Compared to existing technologies, traditional methods often suffer from blurred edges or loss of detail when directly fusing visible and infrared images under low-light conditions due to differences in features between modalities. This proposed solution addresses the issue of high-frequency information attenuation in cross-modal feature fusion by employing independent feature extraction and high-frequency information fusion mechanisms. This is achieved by preserving the advantageous features of each modality and enhancing edge alignment accuracy using gradient operators. Compared to single-path fusion networks, the dual-branch structure effectively avoids feature confusion, ensuring that the multimodal fused image simultaneously possesses the contour sharpness of thermal infrared images and the texture richness of visible light images.

[0031] Through the above technical solution, this application can generate multimodal fused images with clear edge contours and rich texture details in low-visibility scenes such as fog, haze, and nighttime. The fused image not only contains temperature distribution information of the thermal infrared mode but also retains the geometric structural features of the visible light mode, providing a reliable cross-modal reference for subsequent depth calculations. The multi-scale feature fusion mechanism effectively suppresses noise interference and ensures the edge alignment accuracy of depth discontinuities, thereby improving the reconstruction quality of dense depth images in complex environments.

[0032] This application further proposes an edge feature fusion module to extract high-level features from visible light features and thermal infrared features, and to fuse complementary information between visible light features and thermal infrared features; given the input of the edge feature fusion module, its output is achieved by adding gradient features to cascaded convolutional layers, specifically: the output of the edge feature fusion module is equal to the element-wise addition of the input features processed by the cascaded convolutional layers and the features processed by the gradient operator.

[0033] Among them, the gradient operator refers to the mathematical operation of extracting high-frequency components of the feature map through a preset convolutional kernel, specifically implemented using a manually set edge detection convolutional kernel, used to explicitly capture abrupt changes in depth discontinuities. A cascaded convolutional layer is a non-linear transformation structure composed of multiple consecutively stacked convolutional layers, which progressively extracts deep semantic features through cascading, enhancing feature representation capabilities. Element-wise addition refers to the operation of directly adding the values ​​of two feature maps at the same spatial location, used to fuse gradient features and deep semantic features, maintaining the sharpness of high-frequency edges while inheriting semantic abstraction capabilities. A convolutional neural network is a feature extraction model based on deep learning, using pre-trained convolutional neural network parameters to achieve high-frequency feature alignment, avoiding feature mismatch problems caused by manual filters.

[0034] Specifically, the edge feature fusion module first performs a cascaded convolution operation on the input features to extract deep features with multi-scale semantic information. Simultaneously, it performs convolution operations on the input features using gradient operators to extract gradient features containing high-frequency edge information. The deep features and gradient features are then added element-wise, ensuring that the fused features contain both the original semantic information and physically interpretable high-frequency components. During this process, the parameters of the pre-trained convolutional neural network are fixed to ensure spatial alignment of cross-modal features in the edge regions. This fusion mechanism effectively addresses the differences in texture details and thermal radiation characteristics between visible light and thermal infrared images, enabling the output multi-layer features to accurately represent the geometric structure of depth discontinuous regions.

[0035] Compared to existing technologies, traditional methods typically employ a single hand-designed filter or rely solely on learned features for fusion, making it difficult to simultaneously consider the physical characteristics of high-frequency edges and the abstract representation of deep semantics. For example, while Gaussian-Laplacian pyramid-based methods can extract multi-scale features, they cannot adaptively fuse cross-modal information; and while end-to-end convolutional networks possess learning capabilities, they lack explicit modeling of edge features. This proposed solution, through the synergistic effect of gradient operators and cascaded convolutions, achieves for the first time a complementary fusion of hand-designed and learned features in the frequency domain.

[0036] Through the above technical solution, this application can accurately extract and preserve the geometric edge details of the scene during multimodal image fusion, significantly improving the representation accuracy of the microstructures of power equipment such as transmission lines and insulator strings in depth image calculation. Especially in complex environments such as low light and haze, this technology can effectively avoid the problem of blurred depth estimation caused by the loss of edge information in traditional methods, providing a more reliable dense depth data foundation for the 3D detection of power corridors.

[0037] This application further proposes to extract visible light features and thermal infrared features through two branches respectively. The basic convolutional layer of the edge feature fusion module uses a 3×3 convolution and a leaky linear rectified unit as the activation function to extract shallow features. The edge feature fusion module is used to extract detailed features. The image reconstruction module consists of three cascaded 3×3 convolutional layers and one 1×1 convolutional layer, with LReLU and Tanh functions as activation functions respectively.

[0038] The 3×3 convolution refers to a two-dimensional convolution operation with a kernel size of 3×3, specifically implemented with a stride of 1 and padding of 1, used to extract spatial features within a local neighborhood. A leaky linear rectified unit (RCU) serves as the activation function, specifically implemented with a slope of 0.2 in the negative interval, used to preserve weak gradient information in the negative interval during feature extraction. The dual-branch structure means that visible light and thermal infrared images are processed through independent convolutional paths, specifically implemented using parallel convolutional layer groups, used to avoid mutual interference between different modalities in the shallow feature extraction stage. Three cascaded 3×3 convolutional layers refer to a cascaded stacked sequence of convolutional operations, specifically implemented by progressively increasing the number of channels layer by layer, used to gradually expand the receptive field and integrate multi-scale features during image reconstruction. A 1×1 convolutional layer refers to a two-dimensional convolution operation with a kernel size of 1×1, specifically implemented by compressing the number of channels to the target output dimension, used to adjust the channel dimension of the feature map to match the color space of the reconstructed image.

[0039] Specifically, visible light and thermal infrared images are processed by independent convolutional branches. Each branch extracts shallow features containing edges and textures through a 3×3 convolutional layer. Leaking linear rectified units retain weak responses in the negative range during activation, enhancing the ability to express features in low-contrast regions. The edge feature fusion module performs cross-modal interaction on the shallow features output from the two branches, capturing the detailed texture of the visible light image and the temperature contour features of the thermal infrared image through a high-frequency information extraction mechanism. In the image reconstruction stage, three cascaded 3×3 convolutional layers perform nonlinear transformations on the fused multi-scale features, gradually aggregating local details and global contextual information. Finally, a 1×1 convolutional layer maps the high-dimensional features to a three-channel multimodal fused image space. The LReLU activation function maintains the sparsity of feature propagation in the intermediate layers. LReLU (Leaky Rectified Linear Unit) is a variant of ReLU (Rectified Linear Unit), and the final Tanh function constrains the output value within the range [-1, 1], ensuring that the reconstructed image conforms to the pixel distribution pattern of a natural image.

[0040] Compared to existing technologies, traditional methods employ single-path feature extraction during image fusion, leading to interference between visible light and thermal infrared features at the shallow stage, and the smoothing suppression of high-frequency details during fusion. Existing edge feature fusion modules often rely on fixed operators to extract high-frequency information, making it difficult to adaptively adjust the contribution weights of different modal features. Conventional image reconstruction modules use single-scale convolution operations, failing to effectively integrate multi-level feature information. This scheme maintains modal feature independence through a dual-branch structure, dynamically balances the preservation of high-frequency details using a learnable edge fusion mechanism, and achieves progressive reconstruction of the feature space by combining multi-scale convolution sequences, thereby overcoming the shortcomings of traditional methods in detail preservation and feature integration.

[0041] Through the above technical solution, this application effectively solves the problem of high-frequency detail loss in the fusion process of visible light and thermal infrared images. It fully extracts shallow features from different modalities through an independent dual-branch structure and enhances the representation of fine scene structures using an edge feature fusion module. The multi-scale convolutional reconstruction mechanism improves the spatial continuity of the multimodal fused image, enabling the output multimodal fused image to simultaneously retain visible light texture details and thermal infrared contour information, providing high-quality cross-modal reference data for subsequent dense depth calculations.

[0042] This application further proposes to project a 3D point cloud onto an image plane to obtain a sparse depth image.

[0043] Among them, 3D point cloud projection refers to the process of mapping 3D spatial coordinates to a 2D image plane through coordinate system transformation. Specifically, it can be achieved using a camera intrinsic and extrinsic parameter matrix, which includes focal length, principal point coordinates, distortion coefficients, and the position and attitude information of the camera relative to the 3D point cloud coordinate system.

[0044] The image plane refers to a two-dimensional plane that is spatially aligned with the visible light image and the thermal infrared image. Specifically, it can be achieved using a calibrated camera imaging model. By substituting the coordinates of the three-dimensional point cloud into the camera projection equation, a depth distribution map with the same resolution and spatial correspondence as the multimodal image is generated.

[0045] Specifically, an initial depth map is generated by projecting each point in the 3D point cloud onto a 2D image plane using a camera imaging model. This process utilizes the spatial coordinate information of the 3D point cloud, combined with pre-calibrated camera intrinsic and extrinsic parameters, to convert the coordinates of each point in the 3D point cloud into 2D image pixel coordinates, and then fills in the corresponding pixel positions based on the depth value. Because the point cloud acquired by the LiDAR has a low density, the depth map generated by projection exhibits sparse characteristics, but retains the depth measurement accuracy of the original point cloud. During projection, the depth values ​​are quantized to the same resolution as the visible light image and the thermal infrared image, ensuring the consistency of spatial dimension during subsequent multimodal feature fusion. Compared to existing technologies, traditional methods typically employ interpolation or filtering algorithms to preprocess sparse depth maps to generate pseudo-dense depth maps. However, such methods can introduce false depth information or lead to blurred edges. This proposed solution directly preserves the depth measurements of the original point cloud through geometric projection, avoiding information distortion caused by human intervention. Simultaneously, rigorous camera calibration parameters ensure precise spatial alignment between the projection results and the multimodal images, providing an accurate sparse depth data foundation for subsequent cross-modal feature fusion.

[0046] Through the above technical solution, this application solves the problem of efficient conversion from 3D point cloud to sparse depth image. While preserving the accuracy of the original depth information, it achieves spatial alignment with visible light and thermal infrared images, providing reliable multimodal input data for subsequent dense depth computing networks and effectively improving the accuracy of cross-modal feature fusion in the process of depth image densification.

[0047] This application further proposes a dense depth image calculation method, in which the encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of sparse depth images and single multimodal fused images, and the decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output a dense depth image.

[0048] The multi-scale residual pyramid structure refers to a feature extraction network composed of multiple coding blocks at different scales. Each coding block consists of three 3×3 convolutional layers, used to extract cross-modal features layer by layer, capturing local details and global contextual information at different scales. The multi-scale feature fusion module (MSFF) integrates features from three different scales through upprojection and channel concatenation, adaptively aggregating global contextual information, enhancing the feature representation of depth discontinuities. The guided upsampling module (GUB) is an upsampling unit that combines downsampled multimodal fused images, residual depth images, and historical predicted depth maps for feature fusion. It integrates an SE network, improving the accuracy of fine structure recovery during depth map reconstruction through channel attention mechanisms. Image edge feature concatenation involves stitching the edge features extracted from the multimodal fused image with the fused feature map output by the encoder, strengthening the consistency between depth image edges and visual edges.

[0049] Specifically, the encoder concatenates the multimodal fused image with a sparse depth image and inputs it into a multi-scale residual pyramid structure. Each coding block extracts cross-modal features through convolutional layers. The MSFF module upsamples and concatenates features at different scales to generate a residual depth map and a predicted depth map containing global information. Edge features of the multimodal fused image are extracted using the Canny operator or a pre-trained convolutional neural network and then concatenated with the encoded features to constrain the generation of residual depth. The decoder employs four guided upsampling modules. Each module receives the downsampled multimodal fused image, the residual depth map, and the prediction result from the previous scale. It then weights the channel features through an SE network and gradually recovers a high-resolution depth map by fusing multi-source information.

[0050] Compared to existing technologies, traditional methods typically employ a single-scale encoding structure, making it difficult to effectively fuse cross-modal features. Furthermore, the lack of multi-source information guidance during depth map upsampling leads to blurred edges. This scheme achieves cross-scale feature aggregation through a multi-scale residual pyramid and an MSFF module, combined with an edge feature connection mechanism, significantly improving the detail preservation capability in depth discontinuities. The guided upsampling module incorporates an SE network and an iterative prediction mechanism, resolving the edge distortion problem caused by traditional bilinear interpolation upsampling.

[0051] Through the above technical solution, this application can effectively fuse high-frequency details and sparse depth information from visible light and thermal infrared images under complex lighting or severe weather conditions to generate dense depth images with clear edges and complete structure. This method overcomes the feature loss problem of traditional single-modal depth estimation methods in extreme environments, while avoiding information conflicts in the cross-modal feature fusion process, providing a high-precision depth data foundation for the three-dimensional detection of power corridors.

[0052] As a specific solution, this application further proposes a dense depth image calculation method, which adopts a dense depth image calculation method guided by visible light-thermal infrared multimodal fusion image. First, the visible light image and the thermal infrared image are fused into a single image, which is then used as input along with a sparse depth image projected from a laser point cloud. This input is used to construct a single-channel lightweight encoder-decoder structure for a depth image denser convolutional neural network. Considering factors such as computational complexity and fine structure perception capability, this application proposes to use a feature fusion encoder based on a multi-scale residual pyramid for feature encoding and fusion, and a residual pyramid decoder based on a guided upsampling module to predict the final depth image. Through the above methods, the efficiency of computation on edge devices and the accuracy of fine structure depth are ensured.

[0053] First, this application constructs an end-to-end multi-mode image-guided dense depth image computation network. Drawing on image-guided dense depth computation methods, image fusion technology is used to fuse visible light and thermal infrared images into a single image. Then, a convolutional neural network guides the densification of the projected depth of the laser point cloud, thereby enabling the algorithm to support complex weather scenes. In dense depth computation, a lightweight multi-scale residual pyramid and a guided upsampling module are employed to improve the model's ability to perceive fine structures in power corridors and ensure the algorithm can run on edge devices. After generating the dense depth image of the scene, the dense depth image is back-projected into 3D space based on sensor intrinsic parameters to obtain a dense 3D point cloud. Delaunay triangulation algorithms are then used to generate a mesh model composed of triangular facets. The technical roadmap of this application is as follows: Figure 2 As shown.

[0054] 1) Visible-thermal-infrared image fusion network: Given a set of calibrated visible and thermal infrared images, visible-thermal-infrared image fusion is achieved through feature extraction and image reconstruction. First, an Edge Feature Fusion Module (EFFM) is used to fully fuse complementary information between images of different modalities. Specifically, a feature extraction module is used to extract features from the visible images... and thermal infrared images Extracting deep features containing fine structures. This step is represented as ,in and These represent visible light features and thermal infrared features, respectively. EFFM ensures the network's ability to represent fine structures by extracting high-level features. Given the input to EFFM... Its output is:

[0055] in, Indicates a convolutional layer. The cascaded first One convolutional layer; symbol This represents the gradient operator, treating the convolution kernel as a manually configured convolution operation, using a CEDN convolutional neural network. This indicates element-wise addition; The edge feature fusion module EFFM represents the first... i Secondary input features; : Indicates the edge feature fusion module EFFM processing The resulting output features are then passed on to the next... i Characteristics of the +1 stage; Indicates input features Edge features obtained after gradient calculation.

[0056] Next, the multimodal fused image is generated through the image reconstruction module. First, the visible light features and thermal infrared features generated in the previous step are stitched together, represented as... ,in This represents a stitching operation along the channel dimension. The multimodal fused image is then processed using image reconstruction methods. Generation. The overall network structure is as follows: Figure 3 As shown, the feature extraction module extracts visible light features and thermal infrared features through two branches. The basic convolutional layer uses a... Convolutional layers and Leaky Rectified Linear Units (LReLU) are used as activation functions to extract shallow features. Subsequent EFFM is used to extract detailed features. The image reconstruction module uses three concatenated layers... Convolutional layer and one The system consists of convolutional layers with LReLU and Tanh activation functions, and the final output is a visible-thermal infrared multimodal fusion image.

[0057] 2) Dense Depth Image Computation Network; Considering computational complexity, this application proposes to use a single-channel encoder-decoder network structure to infer dense depth images from multimodal fused images and sparse depth images. The network structure is as follows: Figure 4 As shown, the multimodal fused image and the sparse depth image are first stitched together and then fed into the encoder.

[0058] The encoder includes three coding blocks of different scales, each consisting of three... Convolutional Structure. To preserve detailed features during encoding, this application employs a Multi-Scale Feature Fusion (MSFF) module to adaptively aggregate global contextual information. The MSFF module integrates features from three different scales using upprojection and channel concatenation. For each encoding block, the output feature maps of the other three scale encoding blocks are upsampled by 2x, 4x, and 8x respectively, resulting in a final output of 8 channels for each encoding block. This upsampling is performed through channel concatenation, and then concatenated and further transformed by convolutional layers to obtain the output of the MSFF module. The encoder output consists of a residual depth map and a predicted depth map. Furthermore, considering that residual depth images often appear at depth discontinuities, which mostly overlap with image edges, this application extracts image edges from the multimodal fused image using the Canny operator or a convolutional neural network. The calculated image edges are then concatenated with the corresponding fused feature maps to generate the residual depth image. If an edge extraction convolutional neural network is used, it needs to be trained in advance, and the corresponding weights must be fixed during the training of the dense depth image computation network.

[0059] The decoder comprises four Guided Up-sampling Blocks (GUBs). Each GUB takes as input a downsampled multimodal fused image, a residual depth image generated from multi-scale features extracted by the MSFF module, and a depth map predicted from the previous scale. These three inputs undergo a series of fusion and processing steps to generate a refined depth map at the current scale. The detailed structure of each GUB is shown below. Figure 5 As shown, the SE network represents the Squeeze-and-Excitation network. The use of the guided upsampling module can maintain the accuracy of fine structure during depth map upsampling.

[0060] 3) Loss function design and network training; The loss function of the image fusion network includes grayscale intensity loss, which evaluates the overall intensity of the multimodal fused image. To evaluate the texture loss in terms of texture detail Evaluate the depth error loss of the upper-level task. The image fusion loss can be calculated as follows:

[0061] in, The calculation is performed by measuring the grayscale difference between the multimodal fused image and the input image at the pixel level. This is calculated by measuring the gradient difference between each pixel of the multimodal fused image and the input image. The absolute error between the depth calculated by dense depth and the true depth value is calculated.

[0062] The loss function of dense depth computing networks includes depth error loss. Depth gradient loss Normal loss and image structure similarity evaluation loss The dense depth calculation loss can be calculated as follows:

[0063] in, Consistent with the depth error loss in image fusion network loss, Calculate the pixel-by-pixel depth gradient error between the predicted depth map and the ground truth depth map. Calculate the normal error pixel by pixel. Calculate the overall perceptual similarity of the depth images. The losses described above are consistent with commonly used depth estimation methods.

[0064] The network training in this embodiment is divided into three stages. The first stage trains an initial image fusion network based on grayscale intensity loss and texture loss. The second stage fixes the parameters of the image fusion network and trains a dense depth computing network. The third stage jointly trains the image fusion network and the dense depth computing network.

[0065] like Figure 6 As shown, this application further proposes a dense depth image calculation system, including: The fusion unit 100 is used to fuse visible light images and thermal infrared images to generate a multimodal fused image; Processing unit 200 is used to process the 3D point cloud to obtain a sparse depth image; The prediction unit 300 is used to input the sparse depth image and the multimodal fused image into a dense depth image computing network. The dense depth image computing network is a convolutional neural network based on an encoder-decoder structure. The encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of the sparse depth image and the single multimodal fused image. The decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output a dense depth image.

[0066] The multi-scale residual pyramid structure refers to the use of stacked convolutional layers with different receptive fields in the encoder to form a pyramid-shaped feature extraction path. Specifically, it can be implemented by paralleling three dilated convolutions with different dilation rates to capture multi-scale contextual information of cross-modal features. The guided upsampling module introduces edge features as upsampling guidance signals during the decoding stage. Specifically, it can be implemented by combining SE attention mechanism with bilinear interpolation to ensure the integrity of edge details in the depth map. Specifically, the fusion unit 100 extracts high-frequency texture features from the visible light image and low-frequency thermal radiation features from the thermal infrared image using a convolutional neural network, and performs cross-modal feature alignment at the edge feature fusion layer. The processing unit 200 projects the lidar point cloud onto the image plane through coordinate transformation, generating a sparse depth map spatially aligned with the multimodal fused image. The encoder of the prediction unit 300 uses multi-scale residual blocks to process the multimodal fused image and the sparse depth map in parallel, achieving intermodal feature complementarity through a cross-channel attention mechanism. This application further proposes a decoder comprising four guided upsampling modules. The inputs to each guided upsampling module are the downsampled multimodal fused image, the residual depth image generated from multi-scale features extracted by the multi-scale feature fusion module, and the depth image predicted from the previous scale. These three inputs are fused and processed through a series of steps to generate a fine-grained depth image at the current scale. Here, the SE network stands for Squeeze-and-Excitation network. The use of guided upsampling modules can maintain the accuracy of fine structure during depth map upsampling.

[0067] The guided upsampling module is a functional unit that achieves progressive depth map reconstruction through multi-source feature fusion. Specifically, it can use the downsampled multimodal fused image, the residual depth image, and the predicted depth map from the previous scale as input, and achieve feature interaction through convolutional layers and attention mechanisms. This module solves the problem of detail loss in depth discontinuities by integrating multimodal perception features and inter-layer spatial correlation information.

[0068] The residual depth image refers to an intermediate feature map that reflects the correction information for depth discontinuities at edges. Specifically, it can be generated from the depth residual features extracted by the multi-scale feature fusion module. This image carries correction signals for regions with abrupt depth changes, which are used to guide the recovery of edge details during the upsampling process.

[0069] The Squeeze-and-Excitation network is a neural network structure with a channel attention mechanism. Specifically, it captures inter-channel dependencies through global average pooling and dynamically adjusts the feature channel weights. This network enhances the response of feature channels sensitive to depth edges and improves the preservation of high-frequency details.

[0070] Specifically, each guided upsampling module receives input data from three dimensions: the downsampled multimodal fusion image provides multimodal perception features, including visible light texture and thermal infrared radiation characteristics; the residual depth image carries depth edge correction information extracted by the multi-scale feature fusion module; and the predicted depth map from the previous scale conveys spatial correlation features. After feature alignment of the three sets of inputs through convolution operations, a channel attention mechanism is used to dynamically weight the fused features, focusing on enhancing feature channels related to depth edges. In the four-level cascaded guided upsampling process, each module sequentially upsamples the low-resolution depth map by a factor of 2, while simultaneously performing detail correction based on the multimodal fusion image features at the current scale. This progressive reconstruction mechanism enables the depth map to maintain the geometric integrity of key parts of power equipment, such as conductor connection points and insulator outlines, even under conditions of rain, fog, or low light.

[0071] Compared to existing technologies, traditional depth map densification methods rely on interpolation and upsampling of a single modality image, which is susceptible to noise interference and edge blurring under adverse weather conditions. Our proposed solution, however, constructs a physically meaningful multi-source guidance mechanism by fusing multimodal features from visible light and thermal infrared, combined with correction information carried by the residual depth image. Furthermore, a channel attention mechanism is introduced to replace the fixed-weight feature fusion method, enabling the network to adaptively enhance high-frequency detail features and overcome the edge smoothing effect caused by bilinear interpolation.

[0072] Through the above technical solution, this application can accurately recover the depth abrupt edges in power corridor detection scenarios under complex environments such as nighttime and rain / fog, effectively solving the depth diffusion error caused by traditional methods in sparse areas of lidar point clouds. For the 3D reconstruction of millimeter-level components such as transmission line fittings and insulator strings, this method can maintain sub-pixel-level accuracy in areas of depth discontinuity, providing reliable dense 3D data support for intelligent inspection of power equipment status.

[0073] This application further proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a dense depth image calculation method.

[0074] Among them, memory refers to the physical storage unit used to store computer programs and process data. Specifically, it can be implemented using solid-state drives or flash memory chips. Its function is to solidify the algorithm flow of multimodal image fusion and depth calculation, and ensure the repeatability of the calculation process.

[0075] The processor refers to the computing unit that executes the algorithm logic. Specifically, it can be implemented using a multi-core central processing unit or a graphics processing unit. Its role is to accelerate the cross-modal feature encoding and guide the upsampling process through parallel computing, so as to meet the needs of real-time processing of multimodal data.

[0076] The computer program refers to the set of instructions for implementing multimodal image fusion and depth calculation methods. Specifically, it includes algorithm modules such as a visible light-thermal infrared image fusion network, a multi-scale residual pyramid encoder, and a guided upsampling decoder. Its function is to correlate the sparse depth of the three-dimensional point cloud with multimodal image information across modal features to generate a dense depth image containing fine structure.

[0077] Specifically, when the processor executes the computer program, it first extracts high-frequency edge features from the visible light image and semantic features from the thermal infrared image through a visible light-thermal infrared image fusion network, generating a multimodal fused image that retains multi-scale information. Then, the sparse depth image generated by projecting the 3D point cloud is stitched together with the multimodal fused image and input into a multi-scale residual pyramid encoder for cross-modal feature fusion. The encoder aggregates global context information through a multi-scale feature fusion module and combines it with image edge features to generate a residual depth map. The decoder, through a guided upsampling module, performs multi-scale feature fusion on the downsampled multimodal fused image, the residual depth map, and the predicted depth map, gradually restoring the fine structure of the depth image. During this process, the memory continuously stores intermediate feature maps and calculation results, and the processor accelerates feature matching and residual prediction of the multimodal data through parallel computing.

[0078] Compared to existing technologies, traditional electronic devices typically rely on a single visible light image for depth diffusion when processing low-resolution LiDAR data, resulting in an inability to effectively fuse complementary information from thermal infrared images in adverse weather conditions. This solution, through the collaborative design of hardware architecture and algorithm programs, embeds multimodal fusion network parameters in memory and implements parallel computation of cross-modal feature encoding and residual prediction in the processor. This enables electronic devices to effectively address the problem of missing fine structures in sparse depth images by simultaneously utilizing the texture details of visible light images and the semantic information of thermal infrared images in complex scenarios such as rain, fog, and nighttime.

[0079] Through the above technical solutions, this application achieves the ability to densely reconstruct three-dimensional point clouds of power corridors under all-weather conditions. By using multimodal image fusion and cross-scale feature encoding, the depth details of key components such as transmission lines and insulators in the scene are preserved. At the same time, the parallel computing characteristics of electronic devices are used to improve the efficiency of depth image generation, thus meeting the needs of real-time monitoring of power facilities.

[0080] This application further proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a method for dense depth image calculation.

[0081] Computer-readable storage media refers to the physical carrier used to store computer program instructions. Specifically, it can be implemented using solid-state drives, flash memory chips, or optical discs. Its function is to transform the multimodal image fusion and depth computing process into persistent binary code.

[0082] The computer program refers to the executable instruction sequence that implements the dense depth image calculation method. Specifically, it includes the forward propagation logic of the visible light-thermal infrared image fusion network, the feature processing steps of the encoder-decoder structure, and the loss function calculation module. Its function is to encapsulate the algorithm flow of image fusion and depth prediction into a reusable program unit.

[0083] The processor refers to the computing unit that executes computer program instructions. Specifically, it can be implemented using a central processing unit or a graphics processing unit. Its function is to drive the multi-scale residual pyramid structure and guide the upsampling module to complete cross-modal feature fusion and depth map reconstruction based on the program code in the storage medium.

[0084] Specifically, during execution, the program instructions in the storage medium first control the processor to load visible light and thermal infrared images. A feature extraction module extracts depth features containing high-frequency edge information, and an edge feature fusion module integrates multimodal complementary information to generate a multimodal fused image. Subsequently, the sparse depth map generated by projecting the 3D point cloud and the multimodal fused image are input into the encoder. Cross-modal feature encoding is performed in a multi-scale residual pyramid structure, and global context information is aggregated through upsampling channel cascading to generate a residual depth map. The decoder dynamically fuses the residual depth map with image edge features based on the guided upsampling module, progressively restoring spatial resolution and outputting a dense depth image. During execution, the program instructions synchronously call the joint loss function module to optimize depth prediction accuracy through gradient error and normal error constraints.

[0085] Compared to existing technologies, traditional methods rely on specific hardware environments to achieve multimodal data fusion and deep computing, and their algorithm flow is fixed in dedicated equipment, making cross-platform deployment difficult. This solution, however, transforms the algorithm flow into standardized program code through computer-readable storage media, enabling core operations such as multi-scale residual coding and edge-guided upsampling to be executed on general-purpose processors. This avoids dependence on high-cost dedicated computing hardware and ensures operational consistency across different devices through standardized encapsulation of program instructions.

[0086] Through the above technical solutions, this application achieves cross-platform deployment capability for multimodal depth calculation methods, enabling the dense depth prediction process, which integrates the advantages of visible light high-frequency details and thermal infrared all-weather imaging, to run stably on various computing devices. The program instructions solidify the dynamic fusion mechanism of image edge features and residual depth maps, effectively solving the problem of missing depth information in low-line-count LiDAR scenarios and ensuring the accuracy and robustness of 3D point cloud reconstruction in the complex environment of power corridors. The programmatic encapsulation of the joint loss function in the storage medium further optimizes the geometric consistency between the depth prediction results and the real scene, providing reliable 3D data support for power transmission line condition monitoring around the clock.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] This application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, readable storage media, optical storage, etc.) containing computer-usable program code.

[0089] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort should fall within the scope of protection of this application.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation methods of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of this application.

Claims

1. A method for calculating dense depth images, characterized in that, include: A multimodal fused image is generated by fusing visible light images and thermal infrared images; 3D point cloud is processed to obtain a sparse depth image; The sparse depth image and the multimodal fused image are jointly input into a dense depth image computing network. The dense depth image computing network is a convolutional neural network based on an encoder-decoder structure. The encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of the sparse depth image and the single multimodal fused image. The decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output the dense depth image. The encoder includes multiple coding blocks of different scales, each coding block consisting of three... Convolutional structures are used; the encoder employs a multi-scale feature fusion module to adaptively aggregate global contextual information. The multi-scale feature fusion module integrates features from multiple different scales by using up projection and channel cascading. For each coding block, the output feature map is upsampled through channel concatenation. Finally, it is transformed by a convolutional layer to obtain the residual depth map and the predicted depth map. The image edges are extracted from the multimodal fused image using the Canny operator or a convolutional neural network. The calculated image edges are then connected to the corresponding fused feature maps to generate the residual depth image. The decoder includes four guided upsampling modules. The input of each guided upsampling module is the downsampled multimodal fusion image, the residual depth image generated by the multi-scale features extracted by the MSFF module, and the depth image predicted by the previous scale. Through fusion and processing, a fine depth image at the current scale is generated.

2. The dense depth image calculation method according to claim 1, characterized in that, The process of fusing visible light images and thermal infrared images to generate a multimodal fused image includes: A set of calibrated visible light images and thermal infrared images are input into a visible light-thermal infrared image fusion network; the visible light-thermal infrared image fusion network includes a feature extraction module, an edge feature fusion module, and an image reconstruction module; The feature extraction module extracts depth features containing fine structures from visible images and thermal infrared images, obtaining visible light features and thermal infrared features respectively; The edge feature fusion module extracts high-level features from visible light and thermal infrared features, and fuses complementary information between visible light and thermal infrared features; The image reconstruction module stitches together multi-layer features and uses image reconstruction methods to generate and output a multimodal fusion image of visible light and thermal infrared.

3. The dense depth image calculation method according to claim 2, characterized in that, The edge feature fusion module extracts high-level features from visible light and thermal infrared features, and fuses complementary information between visible light and thermal infrared features, including: Given the input of the edge feature fusion module EFFM Its output is: in, Indicates a convolutional layer. The cascaded first One convolutional layer; symbol This represents the gradient operator, treating the convolution kernel as a manually configured convolution operation, using a CEDN convolutional neural network. This indicates element-wise addition; The edge feature fusion module EFFM represents the first... i Secondary input features; : Indicates the edge feature fusion module EFFM processing The resulting output features are then passed on to the next... i Characteristics of the +1 stage; Indicates input features Edge features obtained after gradient calculation.

4. The dense depth image calculation method according to claim 2, characterized in that, The edge feature fusion module extracts visible light features and thermal infrared features through two branches respectively; The basic convolutional layer of the edge feature fusion module passes through a... Convolutional and leaky linear rectified units are used as activation functions to extract shallow features; The edge feature fusion module is used to extract detailed features; The image reconstruction module uses three cascaded... Convolutional layer and one The convolutional layers are composed of LReLU and Tanh activation functions, respectively.

5. The dense depth image calculation method according to claim 1, characterized in that, The process of processing a 3D point cloud to obtain a sparse depth image involves projecting the 3D point cloud onto an image plane to obtain a sparse depth image.

6. A dense depth image calculation system, based on the dense depth image calculation method according to any one of claims 1 to 5, characterized in that, include: The fusion unit is used to fuse visible light images and thermal infrared images to generate a multimodal fused image; The processing unit is used to process the 3D point cloud to obtain a sparse depth image. The prediction unit is used to input the sparse depth image and the multimodal fused image into the dense depth image computing network. The dense depth image computing network is a convolutional neural network based on an encoder-decoder structure. The encoder uses a multi-scale residual pyramid structure to encode and fuse the cross-modal features of the sparse depth image and the single multimodal fused image. The decoder uses a residual pyramid structure and integrates a guided upsampling module to predict the fusion result and output the dense depth image.

7. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the dense depth image calculation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the dense depth image calculation method according to any one of claims 1 to 5.

9. A computer program product, the computer program product comprising computer instructions, characterized in that, The computer instructions instruct the computer to execute the dense depth image calculation method according to any one of claims 1 to 5.

Images