A low-light target detection method and device fusing depth information

Abstract

The application discloses a low-light target detection method and device fusing depth information, and relates to the technical field of deep learning computer vision recognition. The method comprises the following steps: collecting visible light and thermal infrared images, and obtaining visible light and thermal infrared layered feature maps with multiple levels of different resolutions; based on the visible light and thermal infrared layered feature maps with multiple levels of different resolutions, a preset multi-task network architecture is combined to obtain a depth map of a target to be recognized; and parameters of the multi-task network architecture are adaptively optimized according to the difference between the depth map of the target to be recognized and a pseudo-depth true value map. The application significantly improves the target detection robustness in a harsh environment by constructing a double-branch network of a pseudo-twin architecture, combining joint enhancement of a frequency domain and a spatial domain, and introducing depth information as auxiliary supervision.

Description

Technical Field

[0001] This application relates to the field of deep learning computer vision recognition technology, specifically to a low-light target detection method and apparatus that integrates depth information. Background Technology

[0002] In recent years, target detection in low-light environments has played an increasingly important role. Commonly used visible light target detection methods suffer from low accuracy and poor real-time performance under low-light conditions. With the rise of deep learning and multimodal perception technologies, target detection based on multi-source images such as visible light and infrared has gradually become a research hotspot.

[0003] While existing object detection research has made some progress in general scenarios, two prominent problems remain. First, cross-scale detection accuracy is insufficient. Due to factors such as diverse shooting perspectives, targets vary greatly in scale within images. Existing models have limited ability to represent multi-scale features, easily leading to over-focusing on large target features and loss of small target features, resulting in a significant decrease in cross-scale recognition accuracy. Second, recognition performance is unstable under complex lighting conditions. In low-light environments, image signals are prone to insufficient brightness, reduced contrast, and blurred edges. Traditional visible light-based detection algorithms struggle to extract effective features, causing unstable recognition results, target localization misalignment, or increased false detection rates.

[0004] Given the limitations of existing target detection methods in low-light environments, how to achieve effective feature fusion and accurate detection of multi-scale targets, and maintain stable recognition performance in complex environments such as low light, has become a key technical problem that this application urgently needs to solve. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this application aims to provide a low-light target detection method and apparatus that integrates depth information, overcoming the deficiencies of existing technologies. By constructing a pseudo-twin architecture dual-branch network, combining joint enhancement of the frequency and spatial domains, and introducing depth information as auxiliary supervision, the robustness of target detection in harsh environments is significantly improved.

[0006] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a low-light target detection method that integrates depth information, the method comprising the following steps: Visible light and thermal infrared images were acquired, and multi-level layered feature maps of visible light and thermal infrared at different resolutions were obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a pre-set multi-task network architecture, a depth map of the target to be identified is obtained. Based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map, the parameters of the multi-task network architecture are adaptively optimized.

[0007] Based on the above technical solution, visible light and thermal infrared images are acquired, and multi-level visible light and thermal infrared layered feature maps with different resolutions are obtained, including the following steps: Acquire visible light and thermal infrared images, perform multi-scale precise annotation, and add dimension annotations; Visible light and thermal infrared images are corrected to obtain enhanced, clear images; The enhanced sharpened image is inferred using a pre-trained depth estimation model to generate a corresponding pseudo-depth map and a corresponding pseudo-depth ground truth label. A harsh environment adaptability dataset is constructed by integrating enhanced clear images, size annotations, pseudo-depth maps, and pseudo-depth ground truth labels.

[0008] Based on the above technical solution, visible light and thermal infrared images are acquired, and multi-level visible light and thermal infrared layered feature maps with different resolutions are obtained, including the following steps: The enhanced, clear image is transmitted to a dual-branch enhanced SwinTransformer backbone network based on a pseudo-Twin architecture to obtain multi-level visible light and thermal infrared layered feature maps with different resolutions; wherein, The pseudo-twin architecture-based dual-branch enhanced Swin Transformer backbone network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters.

[0009] Based on the above technical solution, a depth map of the target to be identified is obtained by combining multi-level visible light and thermal infrared layered feature maps of different resolutions with a pre-set multi-task network architecture, including the following steps: Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target detection regression head with a pre-set multi-task network architecture, the bounding box position of the target to be identified is obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target category prediction head with a pre-set multi-task network architecture, the category probability information of the target to be identified is obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, and combined with the pseudo-depth map prediction branch of the pre-set multi-task network architecture, the depth map of the target to be identified is obtained.

[0010] Based on the above technical solution, the parameters of the multi-task network architecture are adaptively optimized according to the differences between the depth map and the pseudo-depth ground truth map of the target to be identified, including the following steps: A depth-guided loss based on logarithmic transformation is used to calculate the difference between the depth map of the target to be identified and the pseudo-depth ground truth map, and the parameters of the multi-task network architecture are adaptively optimized.

[0011] Secondly, this application provides a low-light target detection device that integrates depth information, the device comprising: The dataset construction module is used to acquire visible light and thermal infrared images, generate pseudo-depth ground truth maps with added size annotations, and integrate them to build a harsh environment adaptability dataset. A layered feature map construction module is used to obtain multi-level visible light and thermal infrared layered feature maps of different resolutions based on the harsh environment adaptability dataset. The target detection module is used to obtain the depth map of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a preset multi-task network architecture. An architecture optimization module is used to adaptively optimize the parameters of the multi-task network architecture based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map.

[0012] Based on the above technical solution, the dataset construction module is also used to acquire visible light and thermal infrared images, perform multi-scale precise annotation, and add size annotations; The dataset construction module is also used to perform visible light and thermal infrared image correction on visible light and thermal infrared images to obtain enhanced clear images; The dataset construction module is also used to infer the enhanced clear image using a pre-trained depth estimation model, generate a corresponding pseudo-depth map, and generate a corresponding pseudo-depth ground truth label. The dataset construction module is also used to integrate and construct a harsh environment adaptability dataset based on enhanced clear images, size annotations, pseudo-depth maps, and pseudo-depth ground truth labels.

[0013] Based on the above technical solution, the hierarchical feature map construction module is further used to transmit the enhanced clear image to a dual-branch enhanced Swin Transformer backbone network based on a pseudo-Twin architecture to obtain multi-level visible light and thermal infrared hierarchical feature maps of different resolutions; wherein, The pseudo-twin architecture-based dual-branch enhanced Swin Transformer backbone network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters.

[0014] Based on the above technical solution, the target detection module is also used to obtain the bounding box position of the target to be identified by combining multi-level visible light and thermal infrared layered feature maps of different resolutions with a target detection regression head with a preset multi-task network architecture. The target detection module is also used to obtain the category probability information of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target category prediction head with a preset multi-task network architecture. The target detection module is also used to obtain the depth map of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with the pseudo-depth map prediction branch of the preset multi-task network architecture.

[0015] Based on the above technical solution, the architecture optimization module is further used to calculate the difference between the depth map of the target to be identified and the pseudo-depth ground truth map using a depth-guided loss based on logarithmic transformation, and to adaptively optimize the parameters of the multi-task network architecture. Compared with the prior art, the advantages of this application are: This application overcomes the shortcomings of the prior art by constructing a pseudo-twin architecture dual-branch network, combining frequency and spatial domain enhancements, and introducing depth information as auxiliary supervision, which significantly improves the robustness of target detection in harsh environments. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the algorithm principle of the low-light target detection method that integrates depth information according to an embodiment of this application. Figure 2 This is a diagram illustrating the overall network architecture of the low-light target detection method that integrates depth information according to an embodiment of this application. Figure 3 This is a diagram of the MWLFM module architecture of the low-light target detection method that integrates depth information according to an embodiment of this application. Figure 4 This is a diagram of the Enhanced-WTB module architecture of the low-light target detection method that integrates depth information according to an embodiment of this application. Figure 5 This is a diagram of the CM-CFC module architecture of the low-light target detection method that integrates depth information according to an embodiment of this application; Figure 6 This is an architecture diagram of the target detection branch and pseudo-depth prediction branch of the low-light target detection method that integrates depth information according to an embodiment of this application. Figure 7 This is a structural framework diagram of a low-light target detection device that integrates depth information according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0020] This application provides a low-light target detection method and apparatus that integrates depth information, overcoming the shortcomings of the prior art. By constructing a pseudo-twin architecture dual-branch network, combining frequency domain and spatial domain enhancement, and introducing depth information as auxiliary supervision, the robustness of target detection in harsh environments is significantly improved.

[0021] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0022] Firstly, see [the following] Figures 1-6 As shown in the figure, this application provides a low-light target detection method that integrates depth information. The method includes the following steps: S1. Acquire visible light and thermal infrared images, add size annotations, generate corresponding pseudo-depth ground truth maps, and integrate them to construct a harsh environment adaptability dataset; S2. Based on the visible light and thermal infrared images, obtain multi-level visible light and thermal infrared layered feature maps with different resolutions; S3. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a preset multi-task network architecture, the bounding box position, category probability information and depth map of the target to be identified are obtained. S4. Based on the difference between the depth map of the target to be identified and the pseudo-depth ground truth map, the parameters of the multi-task network architecture are adaptively optimized.

[0023] In this embodiment, the shortcomings of the prior art are overcome by constructing a pseudo-twin architecture dual-branch network, combining frequency domain and spatial domain enhancement, and introducing depth information as auxiliary supervision, which significantly improves the robustness of target detection in harsh environments.

[0024] Furthermore, visible light and thermal infrared images are acquired, size annotations are added, corresponding pseudo-depth ground truth maps are generated, and these are integrated to construct a harsh environment adaptability dataset, including the following steps: Acquire visible light and thermal infrared images, perform multi-scale precise annotation, and add dimension annotations; Visible light and thermal infrared images are corrected to obtain enhanced, clear images; The enhanced sharpened image is inferred using a pre-trained depth estimation model to generate a corresponding pseudo-depth map and a corresponding pseudo-depth ground truth label. A harsh environment adaptability dataset is constructed by integrating enhanced clear images, size annotations, pseudo-depth maps, and pseudo-depth ground truth labels.

[0025] Furthermore, based on the visible light and thermal infrared images, multi-level visible light and thermal infrared layered feature maps with different resolutions are obtained, including the following steps: The enhanced, clear image is transmitted to a dual-branch enhanced SwinTransformer backbone network based on a pseudo-Twin architecture to obtain multi-level visible light and thermal infrared layered feature maps with different resolutions; wherein, The pseudo-twin architecture-based dual-branch enhanced Swin Transformer backbone network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters.

[0026] Furthermore, based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a pre-defined multi-task network architecture, the bounding box location, category probability information, and depth map of the target to be identified are obtained, including the following steps: Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target detection regression head with a pre-set multi-task network architecture, the bounding box position of the target to be identified is obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target category prediction head with a pre-set multi-task network architecture, the category probability information of the target to be identified is obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, and combined with the pseudo-depth map prediction branch of the pre-set multi-task network architecture, the depth map of the target to be identified is obtained.

[0027] Furthermore, based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map, the parameters of the multi-task network architecture are adaptively optimized, including the following steps: A depth-guided loss based on logarithmic transformation is used to calculate the difference between the depth map of the target to be identified and the pseudo-depth ground truth map, and the parameters of the multi-task network architecture are adaptively optimized.

[0028] It should be noted that, based on the technical solution of the embodiments of this application, the specific implementation process is as follows: Step 1: Construction of Cross-Modal Dataset and Generation of Robust Pseudo-Depth Ground Truth Map: Acquire synchronous visible light and thermal infrared channel images of the monitored area, and perform multi-scale accurate annotation on all targets in the images; perform HSV color gamut transformation and Atmospheric Scattering Model (ASM) correction on images under low-light scenes to obtain enhanced clear images; use a pre-trained depth estimation model to infer the enhanced clear images and generate corresponding pseudo-depth maps, which are used as pseudo-depth ground truth labels; finally, integrate the original images, target annotation information, and the pseudo-depth ground truth labels to construct a harsh environment adaptability dataset, which serves as a depth-guided loss supervision signal for subsequent training stages; Step 2: Construction and Joint Feature Extraction of Enhanced Cross-Modal Swin Transformer Backbone Network: A dual-branch enhanced Swin Transformer backbone network based on a pseudo-Twin architecture is constructed. This network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters. The visible light image and thermal infrared image processed in Step 1 are input into the corresponding branches. During the hierarchical feature encoding process using the Swin Transformer module, a frequency domain-spatial domain feature joint enhancement module is embedded. The global amplitude adjustment capability in the frequency domain and the local texture recovery capability in the spatial domain of the enhancement module are used to perform feature-level joint correction and coupling degradation decoupling of feature noise in low-light scenes. Finally, multi-level visible light and thermal infrared layered feature maps with high robustness and spatial alignment at different resolutions are output. Step 3: Construct a multi-task network architecture: Construct an object detection regression head, an object category prediction head, and a pseudo-depth map prediction branch, which are used to output the precise bounding box location, detailed category probability, and multi-scale depth information of the target to be identified, respectively, to realize multi-task joint prediction of object detection and depth perception; Step 4, Depth-guided loss calculation and progressive parameter optimization: The depth-guided loss based on logarithmic transformation is used to calculate the difference between the depth map output by the pseudo-depth map prediction branch in Step 3 and the pseudo-depth ground truth map obtained in Step 1; the target detection localization and classification loss jointly optimize the network parameters, and the LoRA (Low-Rank Adaptation) fine-tuning module is used to perform low-rank updates on the key attention mechanism parameters in the enhanced backbone network and feature aggregation module, thereby realizing accurate guidance and domain-adaptive optimization of depth information for cross-modal feature fusion in harsh environments.

[0029] Specifically, in step 1, all low-light images need to be corrected using the ASM atmospheric model before undergoing HSV brightness enhancement: Step 1: Predict transmittance in the image using the ASM atmospheric correction method with Dark Channel Prior (DCP). With atmospheric light value The image's RGB channels are then enhanced and repaired according to a physical correction model, as shown in the following formula: In the formula Indicates the lower bound for division by zero error. express The original channel amplitude of the pixel. express The enhanced channel amplitude of a pixel; The second step is to convert the enhanced image to the HSV color gamut and apply Gamma correction to the V channel to increase brightness. The correction formula is given below: In the formula This represents the corrected brightness value; R, G, and B represent the red, green, and blue channel signals in the original image channels, respectively. This indicates the correction coefficient; the lower the coefficient, the brighter the image after correction.

[0030] Specifically, in step 1, the process of generating a corresponding pseudo-depth ground truth map based on the enhanced image includes: A convolutional neural network based on an encoder-decoder architecture was selected as the depth estimation model. This model has been pre-trained and its parameters initialized on a general large-scale scene dataset. The enhanced image after HSV color gamut transformation and atmospheric scattering model ASM correction is input into the encoder module of the depth estimation model, and multi-scale high-dimensional semantic features of the image are extracted through multi-layer convolution operation; The high-dimensional semantic features are upsampled and recovered using the decoder module, and shallow texture features and deep semantic features are fused using a skip connection to perform pixel-level depth regression prediction. The predicted relative depth map is output and normalized to generate a pseudo-depth ground truth map that corresponds one-to-one with the pixel coordinates of the original image.

[0031] Specifically, in step 2, the dual-branch enhanced Swin Transformer backbone network organizes visible light and thermal infrared information at various scales through the FPN feature pyramid, and constructs a frequency-spatial domain feature joint enhancement module through Discrete Wavelet Transformation (DWT). The specific fusion process includes: Obtain the alignment feature maps of each level output by the Swin Transformer module in the visible light branch and the thermal infrared branch; The low-frequency branch performs weighted fusion of visible light and infrared features at the same scale. The high-frequency branch corrects the high-frequency information of visible light through a gating mechanism; The high- and low-frequency fusion part achieves cross-modal fusion by adding a cross-attention module; finally, the inverse discrete wavelet transform (IDWT) is used to reconstruct each frequency component after fusion back into the spatial domain, and outputs a combined feature image of multi-scale cross-modal frequency domain fusion.

[0032] Specifically, the visible light branch and thermal infrared branch in the dual-branch enhanced Swin Transformer backbone network both adopt a hierarchical pyramid structure, including: Image Patch Partition and Embedding: The input visible light or thermal infrared image is segmented into non-overlapping image patches, and each image patch is mapped to a high-dimensional feature vector; Cascaded Swin Transformer Blocks: Utilize window-based multi-head self-attention (W-MSA) to compute pixel associations within a local window, and utilize moving window multi-head self-attention (SW-MSA) to establish cross-window information interaction between adjacent windows, so as to capture long-distance spatial context dependencies while maintaining linear computational complexity. Patch Merging: In the deeper stages of the network, the feature maps are downsampled to gradually reduce the resolution and increase the channel dimension, generating multi-level hierarchical feature maps with step sizes of 4, 8, 16, and 32. These multi-level hierarchical feature maps serve as the input basis for subsequent discrete wavelet transform and frequency domain fusion.

[0033] Specifically, the steps for performing discrete wavelet transform on the visible light feature maps and thermal infrared feature maps at each level are as follows: Two-dimensional discrete wavelet decomposition is performed on the input feature map using Haar wavelets as basis functions; Output a low-frequency approximation component (LL) and three high-frequency detail components, namely the horizontal detail (LH), vertical detail (HL), and diagonal detail (HH). Among them, the low-frequency approximation component preserves the overall geometric contours of the ship and the dam as well as the information on the illumination distribution, while the high-frequency detail component preserves the texture information of the ship's edges, water ripples, and atmospheric scattering noise.

[0034] The resolution of the image after wavelet transform is halved compared to the original image. The specific calculation formula is as follows: ; The formula represents the original image signal. , These represent low-pass and high-pass filters, respectively, which extract low-frequency and high-frequency features from the image. , These represent the low-frequency and high-frequency image features obtained after filtering, respectively.

[0035] LL, LH, HL, and HH represent two-dimensional images obtained by performing high-frequency and low-frequency filtering in the x and y directions, respectively. The calculation formulas are as follows: ; ; ; ; In the formula, LL represents low-frequency filtering in both the x and y directions of the image; LH represents low-frequency filtering in the x direction and high-pass filtering in the y direction; the calculation process of HL and HH is similar.

[0036] Specifically, the frequency domain-spatial domain feature joint enhancement module adopts a dual-stream parallel interaction architecture: In the low-frequency branch, an infrared-guided space-frequency fusion module is deployed. Feature amplitude and phase are extracted through fast Fourier transform. The amplitude information of the infrared low-frequency component is used to weighted modulate the amplitude of the visible light low-frequency component, so as to maintain structural consistency while correcting the color attenuation of visible light. In the high-frequency branch, an infrared-enhanced Transformer module is deployed to fuse infrared high-frequency edge features as a gating signal with visible light high-frequency texture features, thereby enhancing the global detail modeling capability through a self-attention mechanism. A cross-modal cross-frequency modulator (CM-CFC) is constructed between the two branches to achieve feature interaction. Visible low-frequency features and infrared full-band aggregated features are spliced ​​together along the channel dimension to construct a super reference feature library containing multimodal context. Then, attention is applied to the reference feature library and the high-frequency and low-frequency features of visible light to explore the correlation between the regions with missing visible light textures and the regions with clear infrared structures, so as to achieve precise guidance of infrared structural information for the restoration of visible light details.

[0037] Specifically, in the low-frequency branch, a modality-weighted low-frequency fusion module (MWLFM) is constructed, which combines visible and infrared cross-modal weighting. The infrared and visible low-frequency components are first input into a multi-scale convolutional spatial domain unit (SDU), and multi-scale convolutional kernels are used to extract spatial context features with multiple receptive fields. The extracted spatial features are then subjected to a Fast Fourier Transform (FFT) to decouple them into an amplitude component representing illumination contrast and a phase component representing geometric structure. An infrared amplitude guidance mechanism is constructed, which uses an infrared amplitude generation gain mask to perform weighted modulation of the visible light amplitude, and then recombines the modulated visible light amplitude with the original visible light phase using Euler's formula. Finally, the frequency domain features are mapped back to the spatial domain using inverse fast Fourier transform (IFFT), and the residuals are added to the original spatial features output by the spatial domain units, resulting in low-frequency enhanced features with preserved output structure.

[0038] Specifically, in the high-frequency branch, the deployment of the infrared-enhanced wide transformer module (Enhanced-WTB) involves the following processes: We obtain the visible light high-frequency features and infrared high-frequency features after wavelet transform, and use the self-attention mechanism in the wide Transformer module to perform global context modeling of the visible light high-frequency features. A feature-gated fusion mechanism is constructed, which uses infrared high-frequency features as structural priors and concatenates them with visible light high-frequency features in the channel dimension. An adaptive spatial weight mask is generated through a 1×1 convolutional layer and a sigmoid activation function. ; Using the mask Pixel-level weighted fusion of visible light and infrared features is performed, and the calculation formula is as follows: In the formula This indicates element-wise multiplication. , These represent high-frequency depth maps of visible light and infrared images, respectively. Through the gated fusion mechanism, while preserving the effective texture details of visible light, sharp edge information from infrared images is adaptively introduced to output a structure-enhanced cross-modal high-frequency feature map.

[0039] Specifically, in step 2, the cross-modal cross-frequency modulator (CM-CFC) is configured as a bridge connecting the high-frequency branch and the low-frequency branch, and its input includes visible light high-frequency characteristics. Low-frequency characteristics of visible light Infrared aggregation features are achieved by summing and splicing high-frequency and low-frequency infrared signals. The output is the high-frequency features after the interaction. Low-frequency characteristics ; The specific processing procedure of this regulator is as follows: First, a cross-modal shared reference feature library is constructed, and then visible light low-frequency features are combined through channel splicing operations. splicing with infrared aggregation features It is then mapped to a key vector containing multimodal information using 1×1 convolution. AND value vector ; Subsequently, a bidirectional query mechanism was constructed: in high-frequency interaction paths, visible light high-frequency features were... Mapped to query vector Calculate its relationship with The correlation matrix and from Extract structural texture details and generate hybrid high-frequency features. And return to the high-frequency branch summation; in the low-frequency interaction path, the low-frequency features of visible light are... Mapped to query vector Calculate its relationship with The correlation matrix and from Infrared structural guidance information is extracted to generate hybrid low-frequency features. Returning to the low-frequency branch allows for continuous correction of the infrared structural information across the entire visible light spectrum during feature propagation.

[0040] The calculation formula for the cross-attention mechanism is as follows: In the formula, Q and K represent the query and reference vectors, respectively, corresponding to the visible light high and low frequency queries and feature reference libraries mentioned above. The matrix dimension is used for normalization operations; This represents the feature information carried by the high and low frequency characteristics of visible light.

[0041] Specifically, in step 3, a multi-scale feature pyramid network (FPN) is constructed to predict the multi-level features output by the backbone network, connecting the target detection branch and the pseudo-depth prediction branch respectively: The target detection branch adopts a multi-scale prediction architecture: independent target category prediction heads and target box regression heads are derived from the three different scale feature maps (P3, P4, P5) output by FPN, which are used to detect small-scale, medium-scale and large-scale ship targets respectively. The pseudo-depth prediction branch serves as an auxiliary supervision module. It adopts a multi-scale feature aggregation structure, which unifies and aggregates upsampled features to align the feature resolutions of the three scales. It then fuses multi-scale semantic and texture information through a 3×3 convolutional layer and maps the aggregated depth features into a single-channel depth map through a 1×1 convolutional layer, and upsamples them again to the original image resolution.

[0042] Furthermore, the loss function design of the target detection branch specifically includes two parts: classification loss and regression loss. 1) Target category loss: The target category prediction head uses the Softmax function as the activation function, normalizes the output category confidence into a probability distribution, and calculates the cross-entropy loss between the predicted probability and the true category label to maximize the prediction probability of the correct category. 2) Target bounding box regression loss: The target bounding box regression head is based on an anchor-free architecture design, which regresses the distance offset from the center point to the four sides (left, top, right, and bottom) of the ground truth bounding box; and uses the CIoU loss function to directly optimize the intersection-union ratio between the predicted box and the ground truth box.

[0043] Furthermore, in step 4, the process of calculating the depth-guided loss function using the depth map output by the predicted branch and the pseudo-depth ground truth map, and then fine-tuning the network parameters using LoRA, specifically includes: A scale-invariant refurbishment loss function (SIRL) is constructed. Logarithmic transformation is performed on the predicted depth map output in step 3 and the pseudo depth ground truth map generated in step 1. The scale-invariant error of the two in the logarithmic domain is calculated. A label refurbishment strategy is introduced, and soft pseudo labels are constructed using the weighted moving average of historical prediction values ​​to smooth the ground truth noise. A low-rank adaptive (LoRA) fine-tuning strategy is adopted, freezing most of the pre-trained weights of the two-branch Swing Transformer backbone network and injecting trainable low-rank decomposition matrices only into the query and value projection layers of the multi-head self-attention module (MSA). During the training backpropagation phase, the deep guided loss function is jointly weighted and optimized with the classification loss and regression loss of the object detection task, and only the parameters of the low-rank decomposition matrix and the multi-task prediction head are updated.

[0044] Furthermore, the SIRL loss function between the predicted depth map and the pseudo-depth map labels is calculated using scale-invariant renewal loss. This is achieved by averaging the predicted values ​​from different training iterations of the model to mitigate potential noise or errors in the pseudo-depth map labels. The calculation formula is as follows: ; In the formula It is the depth value predicted by the network; It is the pseudo-tag depth value. It is the confidence factor for the pseudo-label, and it is a hyperparameter of the model settings. It represents the total number of pixels in the depth map.

[0045] The implementation process is as follows: Figure 1 As shown, the specific steps include the following: 1) Cross-modal dataset construction and robust pseudo-depth ground truth map generation. This step aims to provide high-quality input data and supervision signals for network training. First, synchronous visible light (RGB) and thermal infrared (IR) channel images of the region are acquired, and all targets in the images are accurately labeled at multiple scales. Targeted image enhancement is then performed for low-light conditions: The first step involves using the Dark Channel Prior (DCP) ASM atmospheric correction method to predict transmittance and atmospheric light values ​​in the image, and then performing image enhancement on the RGB three channels of the image according to the physical correction model, as shown in the following formula: In the formula Indicates the lower bound for division by zero error. This represents the original channel amplitude of the pixel. This represents the enhanced channel amplitude of a pixel.

[0046] The second step is to convert the enhanced image to the HSV color gamut and apply Gamma correction to the V channel to increase brightness, where the correction formula is: In the formula This indicates the corrected brightness value. This indicates the correction coefficient; the lower the coefficient, the brighter the image after correction.

[0047] The third step involves selecting the Metric3D pre-trained depth estimation model based on an encoder-decoder architecture, inputting the enhanced and clear image into the model, extracting features through multi-layer convolution and performing upsampling recovery, outputting the predicted relative depth map, and normalizing it as the pseudo-depth ground truth label to construct a harsh environment adaptability dataset.

[0048] 2) Construction and joint feature extraction of the enhanced cross-modal Swin Transformer backbone network. A dual-branch enhanced Swin Transformer backbone network based on a pseudo-Twin architecture is constructed. (See [link to documentation]). Figure 2The network comprises a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters. Both branches employ a hierarchical pyramid structure, including image segmentation and embedding layers, cascaded Swin Transformer modules (utilizing W-MSA and SW-MSA mechanisms), and feature map patch merging layers, generating multi-level hierarchical feature maps with strides of 4, 8, 16, and 32. During feature extraction, a frequency-spatial domain feature joint enhancement module is embedded. First, the feature map is decomposed into a low-frequency approximation component (LL) and high-frequency detail components in three directions (LH, HL, HH) using Discrete Wavelet Transform (DWT). The specific fusion process is divided into three branches: Low-frequency branch (MWLFM): See module architecture Figure 3 A low-frequency fusion module for cross-modal weighting of visible and infrared light is constructed. The low-frequency components of infrared and visible light are input into a spatial domain unit (SDU) composed of a 1×1 and 3×3 combined convolutional network to extract spatial features. These features are then transformed to the frequency domain using a Fast Fourier Transform (FFT), decoupling them into amplitude and phase. The spatial domain unit (SDU) employs a resolution-preserving dual-stream parallel multi-scale feature extraction structure, and its specific processing steps are as follows: First, a 1×1 convolutional layer is used to perform channel dimensionality reduction and feature integration on the input feature map. Then, the features are split into two parallel processing branches to capture spatial information at different scales. Secondly, in the upper branch, a 3×3 convolutional layer with padding set to 1 is deployed to extract fine local spatial texture features, which are then processed by the GELU activation function. In the lower branch, two cascaded 3×3 convolutional layers with padding of 1 are deployed to expand the effective receptive field by stacking large convolutional kernels to capture wide-area background context information, and are also processed by the GELU activation function; All the above convolution operations use a stride of 1, which strictly ensures that the spatial resolution of each branch feature map remains unchanged before and after processing, thereby achieving spatial alignment of multi-scale features; Finally, the feature maps output from the upper and lower branches are concatenated along the channel dimension, and the concatenated multi-scale features are input into the 1×1 convolutional layer at the end for cross-channel information fusion and dimension adjustment, outputting spatial embedding features rich in local texture and global context.

[0049] Frequency domain decoupling and feature mapping: First, the infrared low-frequency features and visible light low-frequency features after SDU processing are transformed to the frequency domain by FFT and decoupled into amplitude components and phase components.

[0050] Infrared amplitude-guided cascaded modulation: First-stage modulation (infrared guiding visible light): The infrared amplitude component is passed through a 1×1 convolutional layer and a Sigmoid activation function to generate a first-stage spatial gain mask. This mask is then multiplied element-wise with the visible light amplitude features mapped by the 1×1 convolution to suppress background noise in the visible light using infrared structural priors. Second-stage modulation (visible light feedback to infrared): The output of the first-stage modulation is again passed through a 1×1 convolution and a Sigmoid activation function to generate second-stage attention weights containing cross-modal complementary information.

[0051] Amplitude reconstruction: The second-level attention weights are multiplied element-wise with the original infrared amplitude signal to obtain the final enhanced amplitude. This step ensures that the final amplitude distribution is primarily dominated by the robust infrared illumination structure, while incorporating visible light texture feedback.

[0052] Phase alignment and Euler coupling: The original phase component of visible light is subjected to 1×1 convolution feature mapping, while maintaining its spatial geometric properties; then, the Euler formula is used to couple the reconstructed enhanced amplitude with the mapped visible light phase into complex features.

[0053] Inverse Transform and Residual Fusion: Perform an Inverse Fast Fourier Transform (IFFT) on the coupled complex features to restore them to the spatial domain, and add the restored results to the original visible light SDU output features with pixel-level residuals to output the final low-frequency enhanced features.

[0054] High-frequency branch (Enhanced-WTB): See module architecture. Figure 4 An infrared-enhanced wide Transformer module is deployed, designed to address the problem of high-frequency texture blurring in visible light. It achieves precise repair by introducing infrared high-frequency features as structural priors. The specific processing flow includes the following three stages: Phase 1: Global Modeling of Visible Light High-Frequency Features (WTB Basic Processing). The visible light high-frequency features after wavelet transform are input into the Transformer basic processing unit. First, query (Q), key (K), and value (V) vectors are generated through layer normalization (Norm) and linear projection layers. The long-range dependencies of high-frequency textures are captured using a self-attention mechanism. Then, visible light high-frequency features with global context information are output through a multilayer perceptron (MLP) and residual connections.

[0055] Phase Two: Adaptive Feature Gating and Complementary Fusion (Feature Gating Unit). The visible light features and infrared high-frequency features output from the WTB are concatenated along the channel dimension. The concatenated features are processed in parallel via two paths: Mask generation path: Compress channels using a 1×1 convolutional layer and generate an adaptive spatial weight mask using a Sigmoid activation function. This mask indicates the validity of the visible light texture.

[0056] Feature extraction path: Local details of fused features are extracted using 3×3 convolutional blocks containing the GELU activation function.

[0057] Then, complementary weighted fusion is performed: on the one hand, using a mask. The extracted local detail features are weighted to preserve the effective texture for visible light; on the other hand, an inverse mask is used. The original infrared high-frequency features are weighted, and sharp infrared edges are forcibly injected into the blurred visible light region.

[0058] The third stage: Feature smoothing and remapping. The weighted fused features are input into a post-processing module consisting of two cascaded 3×3 convolutional layers and a GELU activation function to eliminate boundary artifacts caused by the fusion of features from different modalities, and to perform nonlinear feature mapping, finally outputting a structure-enhanced cross-modal high-frequency feature map.

[0059] Convergence Branch (CM-CFC): See module architecture Figure 5 To eliminate the degradation of high and low frequency characteristics of visible light under harsh environments, a CM-CFC module is constructed as a bridge connecting the high-frequency and low-frequency branches. This module achieves cross-modal bidirectional interaction of full-band information by constructing a shared infrared feature space. The specific processing flow is as follows: Constructing a cross-modal shared parameter library: First, high-frequency features (including HL, LH, HH) and low-frequency features (LL) of the infrared branch are extracted. Channel stitching is used to aggregate the high-frequency texture and low-frequency structural information of the infrared spectrum. Subsequently, the aggregated features are sequentially passed through a 1× convolutional layer and a normalization layer for feature fusion and distribution adjustment, and then mapped to a key vector (K) and a value vector (V) through two independent linear layers. This process constructs a super reference library containing robust information across the entire infrared frequency band.

[0060] High-frequency interaction path: High-frequency visible light features are input into the interaction path, and then sequentially processed through a 1×1 convolution, normalization (Norm), and a linear layer to map them into a high-frequency query vector. ).calculate With shared key vectors ( The dot product correlation of ) is used to generate an attention map via Softmax, and based on this, the shared value vector ( A weighted search is performed. The retrieved infrared structural details are fused back into the original visible light high-frequency features through residual connections to generate hybrid high-frequency features, thereby using clear infrared edges to repair blurred visible light textures.

[0061] Low-frequency interaction path: Similarly, the low-frequency features of visible light are mapped to a low-frequency query vector through the same convolution, normalization, and linear transformation. This vector also queries the shared infrared parameter library mentioned above, using infrared-stable illumination structure priors to correct the low-frequency components of visible light. The corrected features are then fused through residual connections to generate hybrid low-frequency features.

[0062] 3) Construct a multi-task network architecture: Based on the three feature maps of different scales (P3, P4, P5, with resolutions of H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32 respectively) output by the FPN feature pyramid, construct a multi-task parallel processing architecture that includes object detection and pseudo-depth prediction. The object detection branch employs a multi-scale decoupled prediction architecture, deploying structurally identical detection methods on the P3, P4, and P5 feature layers. Each detection head internally utilizes a decoupled classification and regression design, specifically including two parallel branches: Classification branch: The input features are processed through four stacked "3×3 convolution-group normalization-ReLU" modules to extract semantic features. Then, the class confidence is output through 1×1 convolution and Softmax function, and the cross-entropy loss is used for optimization.

[0063] Regression branch: Similarly, geometric position features are extracted through four stacked "3×3 convolution-group normalization-ReLU" modules. Based on the Anchor-Free architecture, the distance offset from the center point to the four sides of the ground truth bounding box is directly regressed through 1×1 convolution and ReLU activation, and optimized using CIoU loss.

[0064] Pseudo-depth prediction branch: As an auxiliary supervision module, it adopts a multi-scale feature aggregation structure to balance global semantics and local details. Feature aggregation: First, the small-scale P5 and medium-scale P4 feature maps are upsampled to align their spatial resolution with the large-scale P3 feature map (H / 8×W / 8); then, the three are concatenated along the channel dimension. Depth decoding: The concatenated features are sequentially fused through a "3×3 convolution-normalization-ReLU" module, and then upsampled to restore the original image resolution. Output and supervision: Finally, a single-channel pixel-level depth prediction map is output through the Sigmoid activation function, and the scale-invariant renovation loss SIRL is calculated with the pseudo-depth map label to achieve depth supervision of the backbone network's 3D perception capability.

[0065] Moreover, the pseudo-depth prediction branch is configured to be enabled only as an auxiliary supervision module during the model training and fine-tuning phases; during the model inference or deployment phases, this branch is removed from the computation graph or frozen and does not participate in the final forward inference computation, thereby ensuring that the inference speed is not affected.

[0066] 4) Depth-Guided Loss Calculation and Progressive Parameter Optimization. To leverage depth information to guide feature fusion, Scale-Invariant Renewal Loss (SIRL) is used to calculate the difference between the predicted depth map and the pseudo-depth ground truth map generated in step 1. The SIRL loss formula is as follows: ;in To predict depth for the network, For pseudo-label depth, The confidence factor is used. This loss function introduces a label re-adjustment strategy, utilizing a weighted moving average of historical predictions to mitigate the noise impact of false labels. During training, a low-rank adaptive (LoRA) fine-tuning mechanism is introduced. Most of the pre-trained weights of the two-branch Swing Transformer backbone are frozen, and trainable low-rank decomposition matrices are injected only into the query and value projection layers of the multi-head self-attention module (MSA). The classification and regression losses for the detection task, as well as the depth-guided loss, are jointly optimized, updating only the LoRA parameters and prediction head parameters.

[0067] In practice, the above process can be implemented automatically using computer software technology. By applying this invention, the location and type of ships and pedestrians can be quickly and accurately identified in surveillance images under low light conditions.

[0068] Secondly, see Figure 7 As shown, this application provides a low-light target detection device that integrates depth information. The device includes: The dataset construction module is used to acquire visible light and thermal infrared images, add size annotations, generate corresponding pseudo-depth ground truth maps, and integrate them to build a harsh environment adaptability dataset. A layered feature map construction module is used to obtain multi-level layered feature maps of visible light and thermal infrared at different resolutions based on the visible light and thermal infrared images. The target detection module is used to obtain the bounding box position, category probability information and depth map of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a preset multi-task network architecture. An architecture optimization module is used to adaptively optimize the parameters of the multi-task network architecture based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map.

[0069] In this embodiment, the shortcomings of the prior art are overcome by constructing a pseudo-twin architecture dual-branch network, combining frequency domain and spatial domain enhancement, and introducing depth information as auxiliary supervision, which significantly improves the robustness of target detection in harsh environments.

[0070] Furthermore, the dataset construction module is also used to acquire visible light and thermal infrared images, perform multi-scale precise annotation, and add size annotations; The dataset construction module is also used to perform visible light and thermal infrared image correction on visible light and thermal infrared images to obtain enhanced clear images; The dataset construction module is also used to infer the enhanced clear image using a pre-trained depth estimation model, generate a corresponding pseudo-depth map, and generate a corresponding pseudo-depth ground truth label. The dataset construction module is also used to integrate and construct a harsh environment adaptability dataset based on enhanced clear images, size annotations, pseudo-depth maps, and pseudo-depth ground truth labels.

[0071] Furthermore, the hierarchical feature map construction module is also used to transmit the enhanced clear image to a dual-branch enhanced Swin Transformer backbone network based on a pseudo-Twin architecture to obtain multi-level visible light and thermal infrared hierarchical feature maps of different resolutions; wherein, The pseudo-twin architecture-based dual-branch enhanced Swin Transformer backbone network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters.

[0072] Furthermore, the target detection module is also used to obtain the bounding box position of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target detection regression head with a preset multi-task network architecture. The target detection module is also used to obtain the category probability information of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target category prediction head with a preset multi-task network architecture. The target detection module is also used to obtain the depth map of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with the pseudo-depth map prediction branch of the preset multi-task network architecture.

[0073] Furthermore, the architecture optimization module is also used to calculate the difference between the depth map of the target to be identified and the pseudo-depth ground truth map by using a depth-guided loss based on logarithmic transformation, and to adaptively optimize the parameters of the multi-task network architecture.

[0074] In summary, the low-light target detection device with fused depth information provided in this application embodiment is the same as the low-light target detection method with fused depth information provided in the first aspect in terms of technical problems, technical solutions, technical effects and technical principles, so it will not be described in detail here.

[0075] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0076] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A low-light target detection method incorporating depth information, characterized in that, The method includes the following steps: Visible light and thermal infrared images were acquired, and multi-level layered feature maps of visible light and thermal infrared at different resolutions were obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a pre-set multi-task network architecture, a depth map of the target to be identified is obtained. Based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map, the parameters of the multi-task network architecture are adaptively optimized.

2. The low-light target detection method with fused depth information as described in claim 1, characterized in that, Acquire visible light and thermal infrared images, and obtain multi-level visible light and thermal infrared layered feature maps with different resolutions, including the following steps: Acquire visible light and thermal infrared images, perform multi-scale precise annotation, and add dimension annotations; Visible light and thermal infrared images are corrected to obtain enhanced, clear images; The enhanced sharpened image is inferred using a pre-trained depth estimation model to generate a corresponding pseudo-depth map and a corresponding pseudo-depth ground truth label. A harsh environment adaptability dataset is constructed by integrating enhanced clear images, size annotations, pseudo-depth maps, and pseudo-depth ground truth labels.

3. The low-light target detection method with fused depth information as described in claim 2, characterized in that, Acquire visible light and thermal infrared images, and obtain multi-level visible light and thermal infrared layered feature maps with different resolutions, including the following steps: The enhanced, clear image is transmitted to a dual-branch enhanced Swin Transformer backbone network based on a pseudo-Twin architecture to obtain multi-level visible light and thermal infrared layered feature maps with different resolutions; wherein, The pseudo-twin architecture-based dual-branch enhanced Swin Transformer backbone network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters.

4. The low-light target detection method that integrates depth information as described in claim 1, characterized in that, Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, and combined with a pre-set multi-task network architecture, a depth map of the target to be identified is obtained, including the following steps: Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target detection regression head with a pre-set multi-task network architecture, the bounding box position of the target to be identified is obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target category prediction head with a pre-set multi-task network architecture, the category probability information of the target to be identified is obtained. Based on multi-level visible light and thermal infrared layered feature maps of different resolutions, and combined with the pseudo-depth map prediction branch of the pre-set multi-task network architecture, the depth map of the target to be identified is obtained.

5. The low-light target detection method with fused depth information as described in claim 1, characterized in that, Based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map, the parameters of the multi-task network architecture are adaptively optimized, including the following steps: A depth-guided loss based on logarithmic transformation is used to calculate the difference between the depth map of the target to be identified and the pseudo-depth ground truth map, and the parameters of the multi-task network architecture are adaptively optimized.

6. A low-light target detection device that integrates depth information, characterized in that, The device includes: The dataset construction module is used to acquire visible light and thermal infrared images, generate pseudo-depth ground truth maps with added size annotations, and integrate them to build a harsh environment adaptability dataset. A layered feature map construction module is used to obtain multi-level visible light and thermal infrared layered feature maps of different resolutions based on the harsh environment adaptability dataset. The target detection module is used to obtain the depth map of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a preset multi-task network architecture. An architecture optimization module is used to adaptively optimize the parameters of the multi-task network architecture based on the differences between the depth map of the target to be identified and the pseudo-depth ground truth map.

7. The low-light target detection device with fused depth information as described in claim 6, characterized in that: The dataset construction module is also used to acquire visible light and thermal infrared images, perform multi-scale precise annotation, and add size annotations. The dataset construction module is also used to perform visible light and thermal infrared image correction on visible light and thermal infrared images to obtain enhanced clear images; The dataset construction module is also used to infer the enhanced clear image using a pre-trained depth estimation model, generate a corresponding pseudo-depth map, and generate a corresponding pseudo-depth ground truth label. The dataset construction module is also used to integrate and construct a harsh environment adaptability dataset based on enhanced clear images, size annotations, pseudo-depth maps, and pseudo-depth ground truth labels.

8. The low-light target detection device with fused depth information as described in claim 7, characterized in that: The hierarchical feature map construction module is further used to transmit the enhanced clear image to a dual-branch enhanced Swin Transformer backbone network based on a pseudo-Twin architecture to obtain multi-level visible light and thermal infrared hierarchical feature maps of different resolutions; wherein... The pseudo-twin architecture-based dual-branch enhanced Swin Transformer backbone network includes a visible light branch and a thermal infrared branch with parallel structures but independent weight parameters.

9. The low-light target detection device with fused depth information as described in claim 6, characterized in that: The target detection module is also used to obtain the bounding box position of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target detection regression head with a preset multi-task network architecture. The target detection module is also used to obtain the category probability information of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with a target category prediction head with a preset multi-task network architecture. The target detection module is also used to obtain the depth map of the target to be identified based on multi-level visible light and thermal infrared layered feature maps of different resolutions, combined with the pseudo-depth map prediction branch of the preset multi-task network architecture.

10. The low-light target detection device with fused depth information as described in claim 6, characterized in that: The architecture optimization module is also used to calculate the difference between the depth map of the target to be identified and the pseudo-depth ground truth map by using a depth-guided loss based on logarithmic transformation, and to adaptively optimize the parameters of the multi-task network architecture.

Images