Infrared and polarization image fusion detection method and device

Abstract

The application provides an infrared and polarization image fusion detection method and device. It belongs to the technical field of image recognition. The method comprises the following steps: inputting infrared and polarization images into a low-rank sparse decomposition backbone network based on algorithm unfolding, decoupling low-rank feature components representing natural background and sparse feature components representing high-frequency abrupt material from the feature level; inputting the decoupled multi-modal feature sequence into a cross-modal polarization state space model for dynamic feature modulation and fusion; extracting extreme points in the polarization sparse features as spatial priors to generate an initial target query vector, which is sent to a converter detection head; and finally adopting wavelet frequency domain decoupling to optimize shared parameters in back propagation. Through low-rank sparse decoupling and frequency domain optimization at the physical level, the application completely overcomes multi-task conflicts and greatly improves the accuracy of camouflage target detection.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method and apparatus for detecting infrared and polarization image fusion. Background Technology

[0002] Joint research on multimodal image fusion and target detection is crucial for improving the perception capabilities in complex environments. This technology exhibits unique advantages, particularly in applications combining infrared and polarization imaging. Infrared images can effectively highlight targets in low-light or adverse weather conditions by capturing the thermal radiation characteristics of objects; while polarization images carry information about the microscopic geometry and material properties of object surfaces, enhancing the distinction between targets and backgrounds and highlighting edge details. Therefore, deep fusion of infrared intensity information and polarization features is of paramount importance for improving the robustness and accuracy of target detection in tasks such as autonomous driving, drone reconnaissance, and nighttime surveillance.

[0003] Currently, mainstream multimodal fusion detection methods are mainly divided into two categories. The first category is a two-stage step-by-step approach. This method first uses traditional algorithms or deep learning networks to fuse infrared and polarization images into a single visual image, and then inputs the fused result into an independent object detector for recognition. The drawback of this method is that the optimization objectives of the fusion and detection stages are inconsistent: the fusion network is usually trained based on pixel-level metrics, pursuing visual fidelity, but these metrics may not be the best match for the semantic features required by the downstream detection task, resulting in good fusion results but not necessarily optimal detection performance.

[0004] The second category is end-to-end joint learning schemes, which optimize fusion and detection as a whole to obtain detection-friendly feature representations. However, existing object detection heads have randomness in initializing bounding boxes, resulting in slow detector convergence and a high risk of missed detections. Furthermore, there are severe gradient conflicts between the fusion and detection tasks in the shared backbone network. These conflicts lead to gradient interference during the shared parameter updates of the backbone network, thus limiting the overall optimization upper limit and generalization performance of the model.

[0005] In summary, how to achieve deep fusion of infrared features and polarization state features, while mitigating gradient conflicts between multiple tasks during training, thereby improving the performance and robustness of the fusion detection model, has become an urgent technical problem to be solved. Summary of the Invention

[0006] This application provides an infrared and polarization image fusion detection method and apparatus to address the shortcomings of existing technologies, such as poor infrared and polarization image fusion detection performance, difficulty in capturing global long-distance physical dependencies, and serious gradient conflicts in end-to-end joint training.

[0007] This application provides a method for infrared and polarization image fusion detection, including the following steps: The infrared intensity image, polarization degree image, and polarization angle image are input into the backbone feature extraction network of the detection model to obtain the corresponding output infrared features, polarization degree features, and polarization angle features. The infrared features, polarization degree features, and polarization angle features are then flattened into a one-dimensional multimodal feature sequence. The multimodal feature sequence is input into the cross-modal polarization state space model of the detection model, and the state transition equation of the infrared intensity feature sequence is dynamically modulated by the polarization state feature sequence to obtain the fused feature. The polarization state feature sequence includes the polarization degree feature sequence and the polarization angle feature sequence. Local extreme value coordinates are extracted from the polarization degree features as spatial priors and combined with position encoding to generate an initial target query vector. The fused features and the initial target query vector are input into the detection head based on the converter architecture in the detection model to obtain the target detection result output by the detection head; During the backpropagation phase of the detection model training, the fusion loss gradient and the detection loss gradient are separated and reconstructed based on the wavelet frequency domain gradient decoupling mechanism. The shared parameters of the backbone feature extraction network are updated based on the reconstructed conflict-free gradient. The fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion, and the detection loss gradient is the loss gradient of the detection head for target detection.

[0008] According to the infrared and polarization image fusion detection method provided in this application, the backbone feature extraction network contains a deep neural network layer based on an approximate gradient descent algorithm. The specific feature extraction process includes: The input infrared intensity image, polarization degree image, and polarization angle image are mapped into a feature matrix; In the forward propagation of a differentiable neural network, a parameterized singular value threshold shrinkage operator is used to solve the kernel norm minimization of the feature matrix, and the output is a low-rank background feature representing a smooth thermal radiation or natural polarization background. The L1 norm of the feature matrix is ​​minimized by a parameterized soft threshold shrinkage operator, and the sparse anomalous features representing the edges and polarization distortion abrupt points of the artificial material are output. After assigning learnable modal weights to the low-rank background features and the sparse anomaly features respectively, the features are concatenated to obtain the final infrared features, polarization degree features and polarization angle features.

[0009] According to the infrared and polarization image fusion detection method provided in this application, the feature fusion process of the cross-modal polarization state space model includes: During the discretization state transition process of the state-space model, the hidden states of the polarization degree feature sequence and the polarization angle feature sequence are linearly projected and then injected as input dependency coefficients into the state equation of the infrared intensity feature sequence. The state transition matrix and input matrix of the infrared intensity feature sequence are dynamically adjusted by the input dependency coefficient, and the fused features are determined based on the adjusted state transition matrix and input matrix.

[0010] According to the infrared and polarization image fusion detection method provided in this application, the step of extracting local extreme value coordinates from the polarization degree features as spatial priors and combining them with position encoding to generate an initial target query vector includes: Threshold filtering and local maximum pooling operations are applied to the polarization degree feature to extract the coordinates of polarization hotspot pixels with response values ​​higher than a preset threshold. The coordinates of polarization hotspot pixels represent the physical polarization distortion region between the artificial material and the natural background. The extracted polarization hotspot pixel coordinates are mapped into an initial target query vector using two-dimensional sinusoidal position encoding.

[0011] According to the infrared and polarization image fusion detection method provided in this application, the step of performing frequency band separation and reconstruction of the fusion loss gradient and the detection loss gradient based on the wavelet frequency domain gradient decoupling mechanism includes: Obtain the detection gradient map obtained by backpropagation of the detection loss function, and the fusion gradient map obtained by backpropagation of the fusion loss function, respectively. Two-dimensional discrete wavelet transform is performed on the detected gradient map and the fused gradient map respectively, decomposing them into low-frequency approximate sub-bands and high-frequency detail sub-bands in the horizontal, vertical and diagonal directions; In the low-frequency approximation sub-band, the components of the detection gradient map are given higher update weights; in the high-frequency detail sub-band, the components of the fused gradient map are given higher update weights. The weighted low-frequency approximate subband and high-frequency detail subband are reconstructed using inverse two-dimensional discrete wavelet transform to obtain the decoupled global update gradient, thereby optimizing the backbone feature extraction network.

[0012] According to the infrared and polarization image fusion detection method provided in this application, the detection model is trained based on a joint loss function, which includes detection loss and fusion loss; the fusion loss includes pixel reconstruction loss, structural similarity loss and Stokes parameter physical consistency constraint. The Stokes parameter physical consistency constraint is used to force the polarization degree and polarization angle calculated by the fused feature inversion to maintain mathematical consistency with the polarization degree and polarization angle of the input polarization degree image and polarization angle image in terms of physical optical properties.

[0013] This application also provides an infrared and polarization image fusion detection device, including the following modules: The feature extraction module is used to: input the infrared intensity image, polarization degree image and polarization angle image into the backbone feature extraction network of the detection model, obtain the corresponding output infrared features, polarization degree features and polarization angle features, and flatten the infrared features, polarization degree features and polarization angle features into a one-dimensional multimodal feature sequence; The feature fusion module is used to: input the multimodal feature sequence into the cross-modal polarization state space model of the detection model, and dynamically modulate the state transition equation of the infrared intensity feature sequence using the polarization state feature sequence to obtain the fused features. The polarization state feature sequence includes a polarization degree feature sequence and a polarization angle feature sequence. The query generation module is used to: extract the local extreme value coordinates in the polarization degree features as spatial priors, and combine them with position encoding to generate an initial target query vector; The target detection module is used to: input the fused features and the initial target query vector into the detection head based on the converter architecture in the detection model, and obtain the target detection result output by the detection head; The parameter optimization module is used to: perform frequency band separation and reconstruction of the fusion loss gradient and the detection loss gradient based on the wavelet frequency domain gradient decoupling mechanism during the backpropagation stage of the detection model training; update the shared parameters of the backbone network based on the reconstructed conflict-free gradient; wherein the fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion; and the detection loss gradient is the loss gradient of the detection head for target detection.

[0014] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the infrared and polarization image fusion detection method as described above.

[0015] This application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the infrared and polarization image fusion detection method as described above.

[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the infrared and polarization image fusion detection method as described above.

[0017] The infrared and polarization image fusion detection method and apparatus provided in this application input infrared intensity feature sequences and polarization state feature sequences into a cross-modal polarization state space model. The state transition equation of the infrared intensity feature sequence is dynamically modulated using the polarization state feature sequence. The polarization state reflects the physical properties of the target surface, and its changes are related to the target's material, orientation, and microstructure. These properties also affect the local characteristics and spatial distribution of its thermal radiation. By modulating the infrared feature state transition through polarization state, this physical correlation is essentially encoded into the feature evolution process, so that the propagation and updating of infrared features are guided by polarization physical information in real time. This allows the fused features to simultaneously contain the target's radiation intensity differences and fine surface physical properties, thereby generating fused features that more deeply characterize the target's essence and are more robust to complex background interference, providing a reliable feature data foundation for subsequent high-precision detection. Furthermore, in polarization images, significant geometric features such as target edges and corners often exhibit local extrema. This characteristic is utilized to extract local extremum coordinates from polarization features as spatial priors, and an initial target query vector is generated accordingly. This provides the detection head with initial candidates rich in target location cues, enabling the detection head to focus its attention on potential target areas more quickly and accurately, effectively accelerating the model convergence process, thereby improving overall detection efficiency and positioning accuracy. Furthermore, since different tasks have varying sensitivities to different frequency features—fusion tasks are sensitive to low-frequency components, while detection tasks are sensitive to high-frequency components—a wavelet frequency domain gradient decoupling mechanism is used to purposefully separate and selectively reconstruct the fusion loss gradient and detection loss gradient in the frequency domain during the backpropagation stage. This preserves the low-frequency components beneficial to the fusion task and the high-frequency components crucial for the detection task, while suppressing or eliminating potentially conflicting frequency components. Based on the reconstructed gradient with significantly reduced conflict, the shared parameters of the backbone feature extraction network are updated, ensuring that the backbone feature extraction network can simultaneously learn general feature representations beneficial to feature fusion and discriminative features beneficial to accurate target localization and classification. This alleviates the suboptimal performance problem caused by inconsistent gradient directions in traditional multi-task learning, thereby improving the overall detection accuracy and generalization ability of the model. In summary, this application systematically improves the performance of infrared and polarization image fusion detection through dynamic feature fusion, polarization feature prior guidance, and wavelet frequency domain gradient decoupling, enhancing image detection accuracy and robustness. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts of the infrared and polarization image fusion detection method provided in this application; Figure 2 This is a schematic diagram of the backbone feature extraction network provided in this application; Figure 3 This is a schematic diagram of the structure of the regional sensing fusion subnet provided in this application; Figure 4 This is a schematic diagram of the structure of the detection head provided in this application; Figure 5 This is a schematic diagram of the hierarchical gradient alignment mechanism provided in this application; Figure 6 This is the second flowchart of the infrared and polarization image fusion detection method provided in this application; Figure 7 This is a schematic diagram of the infrared and polarization image fusion detection device provided by the present invention; Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following is combined with Figures 1 to 8 This application describes the infrared and polarization image fusion detection method and apparatus.

[0022] Figure 1 This is one of the flowcharts of the infrared and polarization image fusion detection method provided in this application, such as... Figure 1 As shown, the method includes the following: S110, the infrared intensity image, polarization degree image and polarization angle image are input into the backbone feature extraction network of the detection model to obtain the corresponding output infrared features, polarization degree features and polarization angle features, and the infrared features, polarization degree features and polarization angle features are flattened into a one-dimensional multimodal feature sequence; S120, the multimodal feature sequence is input into the cross-modal polarization state space model of the detection model, and the state transition equation of the infrared intensity feature sequence is dynamically modulated by the polarization state feature sequence to obtain the fused feature. The polarization state feature sequence includes the polarization degree feature sequence and the polarization angle feature sequence. S130, extract the local extreme value coordinates in the polarization degree feature as spatial prior, and combine them with the position encoding to generate an initial target query vector; S140, the fused features and the initial target query vector are input into the detection head based on the converter architecture in the detection model to obtain the target detection result output by the detection head; S150, during the backpropagation stage of the detection model training, the fusion loss gradient and the detection loss gradient are separated and reconstructed based on the wavelet frequency domain gradient decoupling mechanism. The shared parameters of the backbone feature extraction network are updated based on the reconstructed conflict-free gradient. The fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion, and the detection loss gradient is the loss gradient of the detection head for target detection.

[0023] It should be noted that the execution subject of the infrared and polarization image fusion detection method provided in this application embodiment can be a server, computer equipment, such as mobile phone, tablet computer, laptop computer, handheld computer, vehicle electronic equipment, wearable device, ultra-mobile personal computer (UMPC), netbook or personal digital assistant (PDA), etc.

[0024] In S110, the infrared intensity image is a grayscale image generated by capturing the thermal radiation energy emitted by the target object through an infrared sensor. It reflects the temperature distribution of the scene and is mathematically represented by low-rank components composed of a large area of ​​thermal background and sparse anomalies composed of high-temperature heat sources. The degree of polarization (DoLP) image is a physical quantity that describes the polarization state of light waves. It can effectively suppress non-metallic reflections and enhance the contrast between artificial objects and natural backgrounds. Its physical characteristics are manifested as the sparse high-frequency response of artificial targets relative to natural low-polarization backgrounds. The angle of polarization (AoP) image is a physical quantity that describes the vibration direction of light waves and is highly sensitive to the surface normal direction and geometry of the object.

[0025] In this embodiment, the detection model receives three input data streams: an infrared intensity image, a polarization degree image, and a polarization angle image. The backbone feature extraction network does not employ a traditional pure convolutional stacking structure, but rather a deep low-rank sparse decomposition network based on algorithmic unfolding. Specifically, this network models the extraction process of infrared and polarization features as a constrained matrix factorization problem, utilizing the differentiability of deep learning to unfold the traditional Alternating Direction Method of Multipliers (ADMM) or proximal gradient descent algorithm into neural network layers: First, for the polarization angle image, it is mapped to dual-channel features through sine-cosine encoding to eliminate numerical abrupt changes caused by the periodicity of the angle.

[0026] Subsequently, the images of each modality are input into the low-rank background extraction branch and the sparse target extraction branch: In the low-rank background extraction branch, the learnable Singular Value Thresholding (SVT) operator is used to extract low-rank features that reflect the global radiation floor and smooth background of the scene by performing singular value decomposition on the feature matrix and thresholding low contribution components; In the sparse target extraction branch, the learnable Soft Thresholding operator is used to extract sparse features that reflect the abnormal distribution of camouflaged target edges, corners and materials.

[0027] Finally, the network performs inter-modal interaction fusion on the low-rank and sparse features after decoupling from each mode, and outputs high-dimensional infrared features, polarization degree features, and polarization angle features that maintain consistent spatial resolution. Furthermore, the high-dimensional feature map is flattened through a cross-scanning mechanism to construct a multimodal feature sequence capable of characterizing long-distance physical dependencies.

[0028] In S120, the Cross-Polarization Mamba (CP-Mamba) state-space model utilizes polarization state feature sequences as dynamic modulation signals. By changing the evolution path of the state-space equations, it guides the infrared intensity feature sequences to achieve physical-level feature aggregation globally. Unlike traditional static region masks, this application achieves dynamic allocation of fusion weights for different material regions across the entire image through real-time adjustment of the state transition matrix based on polarization features, thereby outputting a high-fidelity fused feature map containing rich semantic edges and thermal radiation details.

[0029] In this embodiment, the CP-Mamba module receives multimodal sequence data output from the backbone network. Internally, this module maintains a hidden state vector that evolves with spatial location. Specifically, the system extracts the physical response intensity of the polarization state sequence (containing polarization degree and polarization angle information) at the current scan position and transforms it into modulation coefficients of the state transition matrix and input control matrix through a linear mapping.

[0030] Under this mechanism, when the sequence scans to a potential target region with significant polarization distortion (i.e., a high-response region of sparse anomalous features), the dynamically generated modulation coefficients enhance the sensitivity of the state equation to capturing current infrared features, enabling the network to automatically focus on the physical boundaries and texture details of the target. Conversely, in natural background regions with lower polarization response (i.e., low-rank background regions), the state equation tends to evolve more smoothly, making the network more inclined to preserve the global thermal radiation structure in the infrared intensity image. Finally, after unidirectional or bidirectional cross-scanning processing by the CP-Mamba module, the hidden state sequence is mapped back to a two-dimensional spatial scale, and the final fused feature map is generated through feature recombination. Compared to traditional region cue masks, this state-space model-based fusion strategy possesses global long-range dependency modeling capabilities, can more naturally handle feature alignment at material boundaries, and effectively avoids fusion distortion caused by inaccurate initial bounding boxes.

[0031] In S130, local maxima are detected in the polarization feature map. These points usually correspond to the edges or corners of man-made targets. The coordinates of the extreme points are used as spatial priors of the possible existence of the target. Combined with position encoding, the extreme coordinates are converted into an initial target query vector, which improves the detection convergence speed and accuracy.

[0032] In S140, the detection head based on the Transformer architecture uses a high-fidelity fused feature map as the global context and an initial target query vector generated based on polarization extrema as a physical prior anchor point, performing parallel decoding through a multi-layer cross-attention mechanism. Since the initial target query vector has already locked the high-response region of material anomalies during the initialization phase, the decoding process can quickly and accurately aggregate the semantic information and fine boundaries of the target from the fused features.

[0033] In S150, during the backpropagation stage, wavelet transforms are performed on the fusion loss gradient and the detection loss gradient, respectively, to decompose them into low-frequency, mid-frequency, and high-frequency components. Conflicting components with opposite gradient directions in the high-frequency band are identified, the shared gradients of low-frequency and mid-frequency are preserved, and the high-frequency conflicting gradients are orthogonalized or selectively suppressed to reconstruct conflict-free gradients. The reconstructed gradients are used to update the shared parameters of the backbone network, thereby achieving collaborative optimization of feature fusion and target detection.

[0034] The infrared and polarization image fusion detection method provided in this application inputs the infrared intensity feature sequence and the polarization state feature sequence into a cross-modal polarization state space model. The polarization state feature sequence is used to dynamically modulate the state transition equation of the infrared intensity feature sequence. The polarization state reflects the physical properties of the target surface, and its changes are related to the target's material, orientation, and microstructure. These properties also affect the local characteristics and spatial distribution of its thermal radiation. By modulating the infrared feature state transition through polarization state, this physical correlation is essentially encoded into the feature evolution process, so that the propagation and updating of infrared features are guided by polarization physical information in real time. This allows the fused features to simultaneously contain the target's radiation intensity differences and fine surface physical properties, thereby generating fused features that more deeply characterize the target's essence and are more robust to complex background interference, providing a reliable feature data foundation for subsequent high-precision detection. Furthermore, in polarization images, significant geometric features such as target edges and corners often exhibit local extrema. This characteristic is utilized to extract local extremum coordinates from polarization features as spatial priors, and an initial target query vector is generated accordingly. This provides the detection head with initial candidates rich in target location cues, enabling the detection head to focus its attention on potential target areas more quickly and accurately, effectively accelerating the model convergence process, thereby improving overall detection efficiency and positioning accuracy. Furthermore, since different tasks have varying sensitivities to different frequency features—fusion tasks are sensitive to low-frequency components, while detection tasks are sensitive to high-frequency components—a wavelet frequency domain gradient decoupling mechanism is used to purposefully separate and selectively reconstruct the fusion loss gradient and detection loss gradient in the frequency domain during the backpropagation stage. This preserves the low-frequency components beneficial to the fusion task and the high-frequency components crucial for the detection task, while suppressing or eliminating potentially conflicting frequency components. Based on the reconstructed gradient with significantly reduced conflict, the shared parameters of the backbone feature extraction network are updated, ensuring that the backbone feature extraction network can simultaneously learn general feature representations beneficial to feature fusion and discriminative features beneficial to accurate target localization and classification. This alleviates the suboptimal performance problem caused by inconsistent gradient directions in traditional multi-task learning, thereby improving the overall detection accuracy and generalization ability of the model. In summary, this application systematically improves the performance of infrared and polarization image fusion detection through dynamic feature fusion, polarization feature prior guidance, and wavelet frequency domain gradient decoupling, enhancing image detection accuracy and robustness.

[0035] In an optional embodiment, the backbone feature extraction network contains a deep neural network layer based on an approximate gradient descent algorithm, and the specific feature extraction process includes: The input infrared intensity image, polarization degree image, and polarization angle image are mapped into a feature matrix; In the forward propagation of a differentiable neural network, a parameterized singular value threshold shrinkage operator is used to solve the kernel norm minimization of the feature matrix, and the output is a low-rank background feature representing a smooth thermal radiation or natural polarization background. The L1 norm of the feature matrix is ​​minimized by a parameterized soft threshold shrinkage operator, and the sparse anomalous features representing the edges and polarization distortion abrupt points of the artificial material are output. After assigning learnable modal weights to the low-rank background features and the sparse anomaly features respectively, the features are concatenated to obtain the final infrared features, polarization degree features and polarization angle features.

[0036] In this embodiment of the application, the infrared intensity image, the polarization degree image, and the polarization angle image are respectively input into the independent low-rank sparse decomposition branches in the backbone network, and mapped into a high-dimensional feature matrix through the algorithm expansion layer; In the forward propagation of a differentiable neural network, a parameterized singular value thresholding contraction operator is used to minimize the kernel norm of the high-dimensional feature matrix, outputting low-rank background features; a parameterized soft thresholding contraction operator is used to minimize the L1 norm of the high-dimensional feature matrix, outputting sparse anomaly features. The low-rank background features and sparse anomaly features after decoupling from each modality are weighted and recombined. The recombined two-dimensional spatial features are flattened through a cross-scanning mechanism to obtain the corresponding one-dimensional multimodal feature sequence.

[0037] Here, the independent low-rank sparse decomposition branches refer to neural network paths designed separately for three different modes: infrared intensity, polarization degree, and polarization angle, based on mathematical-physical prior decoupling. These branches do not share weights in the initial stage, aiming to utilize the physical differences between infrared thermal radiation and the polarization characteristics of artificial materials to achieve mathematical separation of the scene background and target edges at the very beginning of feature extraction, thus avoiding aliasing of signals with different physical properties.

[0038] Here, the parameterized singular value thresholding shrinkage operator is implemented by unfolding the traditional iterative optimization algorithm into a deep network layer. It aims to extract low-rank background features with strong spatial correlation and gentle energy distribution in the image, such as large-area thermal backgrounds or natural environmental substrates, by performing singular value decomposition on the feature matrix and thresholding low-contribution components.

[0039] Here, the parameterized soft thresholding shrinkage operator extracts sparse anomalous features with high-frequency response and sparse distribution in the image by performing nonlinear soft thresholding mapping, such as the physical boundaries of man-made targets, extreme points of heat sources, or material polarization distortion points.

[0040] Here, the cross-scanning mechanism refers to the use of a multi-directional sequence sampling strategy to transform a decoupled feature map with a two-dimensional spatial topology into a one-dimensional multimodal feature sequence with full-map dependencies, thereby breaking the limitation of the local receptive field of the convolutional kernel and providing a data foundation for subsequent cross-modal state space models to capture long-distance physical dependencies.

[0041] Figure 2 This is a schematic diagram of the backbone feature extraction network provided in this application. For example... Figure 2 As shown, firstly, an infrared intensity image (IR), a degree of polarization (DoLP) image, and an angle of polarization (AoP) image are received at the input end. The AoP image is then coded using sine-cosine encoding, mapping it to dual-channel sine and cosine features to eliminate periodic jumps in polarization angle information and improve the stability of the feature representation.

[0042] Furthermore, in the feature extraction part, this application introduces a low-rank and sparse decomposition (LRSD) branch in the algorithm's unfolded architecture. Unlike traditional convolutional networks that blindly extract semantics, this application utilizes the physical differences between infrared thermal radiation background and polarization distortion of artificial targets to model feature extraction as a process of solving for the low-rank and sparse components of the feature matrix. Specifically, the backbone feature extraction network receives the encoded three-modal data in parallel and feeds them into low-rank sparse decomposition paths with identical structures but independent parameters. Within each path, a parameterized singular value thresholding contraction operator is used to perform kernel norm minimization constraints, extracting low-rank background features with gentle energy distribution and strong spatial correlation to characterize the natural background and thermal radiation substrate in the scene. Simultaneously, a parameterized soft thresholding contraction operator is used to perform L1 norm minimization constraints to capture high-frequency response and discretely distributed sparse anomaly features, which are used to accurately locate the physical edges and material abrupt change points of camouflaged targets.

[0043] Furthermore, in the deep feature fusion section, the aforementioned low-rank background features and sparse anomaly features are weighted and converged through learnable modal weights. To overcome the limitations of the local receptive field of traditional convolutional kernels, the system introduces a cross-scanning mechanism. This mechanism, through a multi-directional sequence sampling strategy, transforms the decoupled feature map with a two-dimensional spatial topology into a one-dimensional multimodal feature sequence with global dependencies. Through this branch decoupling-sequence mapping structure, the network achieves physical-level separation between "background" and "target" at the shallow level and establishes long-distance physical property associations between modalities at the deep level, providing highly discriminative feature representations for subsequent cross-modal state space fusion.

[0044] In some embodiments, both the parameterized singular value threshold shrinkage operator and the soft threshold shrinkage operator are implemented using algorithm expansion techniques, that is, mapping mathematical iteration steps to tunable parameter layers of a neural network. This enables the model to automatically optimize the decomposition threshold through backpropagation while maintaining physical interpretability, so as to adapt to the detection needs under different lighting and weather conditions.

[0045] In other embodiments, the decoupled feature sequence passes through a dynamic amplitude alignment layer before entering subsequent modules. This layer aims to eliminate the magnitude difference between the infrared thermal intensity value and the polarization feature value, ensuring that the subsequent state-space model Mamba can maintain numerical stability during dynamic modulation, thereby synergistically enhancing the saliency and discriminative power of the target region.

[0046] The infrared and polarization image fusion detection method provided in this application achieves explicit decoupling of physical properties at the front end through a low-rank sparse decomposition backbone network based on algorithm expansion. The low-rank branch extracts global background radiation, and the sparse branch extracts high-response material edges, providing a clean and physically meaningful input sequence for subsequent feature fusion. Through a cross-modal polarization state space model, dynamic feature complementarity under long-distance dependence is achieved. The hidden state of the polarization sequence is used to modulate the state evolution path of the infrared sequence in real time, ensuring that the fused features maintain the continuity of thermal radiation while possessing extremely high edge contrast and material discrimination, thus maximizing the balance between fusion quality and detection task benefits.

[0047] In an optional embodiment, the feature fusion process of the cross-modal polarization state space model includes: During the discretization state transition process of the state-space model, the hidden states of the polarization degree feature sequence and the polarization angle feature sequence are linearly projected and then injected as input dependency coefficients into the state equation of the infrared intensity feature sequence. The state transition matrix and input matrix of the infrared intensity feature sequence are dynamically adjusted by the input dependency coefficient, and the fused features are determined based on the adjusted state transition matrix and input matrix.

[0048] In this embodiment of the application, in the sequence evolution layer, the input infrared intensity feature sequence is defined as the evolution path, and the polarization degree feature sequence and polarization angle feature sequence are defined as the external modulation path.

[0049] During the state space scanning process, the hidden state of the external modulation path at the current scanning position is extracted, and a dynamic modulation matrix related to the spatial position is generated through nonlinear projection mapping. The dynamic modulation matrix includes a dynamic transition matrix and a dynamic input matrix, which are used to adjust the update step size and memory strength of the infrared features in the induced path in real time.

[0050] Using the dynamic modulation matrix, the evolution path is guided to increase the state update weight when scanning to the high polarization response region (i.e. the edge of the artificial material), while maintaining stable state evolution when scanning to the low polarization low-rank background region, thereby realizing physical-level adaptive feature aggregation and outputting the aggregated multimodal feature sequence.

[0051] The aggregated multimodal feature sequence is recombined by outputting the observation matrix, and the sequence is restored to a two-dimensional spatial scale fused feature by using the reverse cross-scanning mechanism and then output.

[0052] In this embodiment, a cross-modal polarization state-space model replaces the traditional cascaded architecture based on region mask feedback. This model achieves deep isomorphism between infrared thermal radiation distribution and polarization texture characteristics through a physical quantity-driven state transition mechanism.

[0053] Figure 3 This is a schematic diagram of the structure of the cross-modal polarization state-space model provided in this application, as shown below. Figure 3 As shown, the model abandons the cross-modal attention mechanism, whose computational complexity increases with the square of the resolution, and instead adopts a multi-directional scanning state space architecture with linear complexity.

[0054] In terms of physical co-modulation, the model employs a coarse-to-fine cross-scale scanning strategy, utilizing the sparse anomalies inherent in the polarization state sequence as a valve signal to dynamically determine the retention ratio of infrared radiation information in the feature space. For example, when detecting weak heat source targets against a complex background, the polarization branch identifies material polarization distortion, generating a high-response latent state activation. This forces the state transition matrix to perform feature enhancement operations at that location, thereby improving the target boundary sharpness by orders of magnitude during the fusion stage.

[0055] In terms of global consistency modeling, CP-Mamba possesses an infinitely long context modeling capability, enabling it to capture long-distance physical constraints between heat source distribution and polarization reflection field across the entire image. This mechanism allows the fused feature map to not only accurately preserve local details but also automatically suppress artifacts caused by environmental noise or sensor mismatch. Ultimately, the fused features reconstructed through the inverse scanning operator not only possess thermal radiation features consistent with visual perception but also include highly discriminative material fingerprints that support downstream detection tasks, providing physically interpretable and robust feature inputs for subsequent detection stages.

[0056] In the specific implementation process, for the physical hidden state extraction layer, the polarization degree and polarization angle features are first mapped to the hidden space through the linear projection layer to extract the extreme response component that characterizes the material specificity; during the sequence scanning process, this component is subjected to position-by-position Hadamard product operation with the transfer matrix of infrared features to achieve real-time masking or enhancement of the infrared thermal radiation background.

[0057] For multi-scale dynamic feature aggregation, CP-Mamba internally contains multiple SSM (State Space Model) operators with different scan steps, capturing pixel-level fine textures and region-level material distributions respectively. This multi-scale parallel architecture ensures that the fused features are highly aligned semantically with complex camouflage scenes.

[0058] Furthermore, this application introduces an output residual normalization layer, which residually connects the CP-Mamba modulated deep features with the low-rank sparse decoupled features of the original input. On the one hand, this structure ensures effective gradient propagation by introducing a direct connection, solving the convergence difficulty problem of deep state-space models in the early stage of training; on the other hand, by superimposing the original physical components with the deep modulated features, the robustness of the fusion result under extreme, low signal-to-noise ratio conditions is further enhanced, ensuring that the output fused feature map has both clear visual boundaries and clear target category saliency.

[0059] The infrared and polarization image fusion detection method provided in this application, through a linear scanning mechanism of a cross-modal polarization state space model, first establishes long-distance physical dependencies at the global sequence scale, providing a complete set of data for infrared thermal radiation distribution and polarization texture features. Figure 1 Consistent alignment guidance is then employed. Subsequently, the state transition process is dynamically modulated using the hidden states of the polarization sequence, precisely capturing the physical complementarity between different modalities by altering the feature evolution path. This mechanism significantly improves the spatial consistency of the fused features, enabling the network to adaptively align and reconstruct the infrared thermal target subject with the edge of the polarization material in the feature space. This effectively avoids feature misalignment and edge artifacts caused by the local receptive field of traditional convolution, making the fused representation not only more physically consistent but also more accurate in terms of target saliency.

[0060] Furthermore, by dynamically modulating the polarization latent states during the fusion process, the spontaneous enhancement of the weights of potential target regions and the adaptive suppression of background redundancy are achieved, thereby significantly improving the signal-to-noise ratio of the target region in the fused feature map and reducing background false alarm interference. Simultaneously, through output residual connections and nonlinear mapping layers, the network can deeply learn the high-order nonlinear combination relationship between the decoupled low-rank background and sparse material details, ensuring that the original effective information extracted from the backbone network is not lost during multi-layer state evolution. This results in output fused features that possess both clear and sharp physical edges and highly abstract target semantics conducive to classification.

[0061] In an optional embodiment, the step of extracting local extremum coordinates from the polarization degree features as spatial priors and combining them with position encoding to generate an initial target query vector includes: Threshold filtering and local maximum pooling operations are applied to the polarization degree feature to extract the coordinates of polarization hotspot pixels with response values ​​higher than a preset threshold. The coordinates of polarization hotspot pixels represent the physical polarization distortion region between the artificial material and the natural background. The extracted polarization hotspot pixel coordinates are mapped into an initial target query vector using two-dimensional sinusoidal position encoding.

[0062] Here, polarization hotspots refer to areas in a polarization image where the pixel value is significantly higher than the surrounding background or exceeds a certain threshold. These areas typically correspond to the boundary between man-made objects and the natural environment or highly reflective surfaces, because artificial smooth surfaces produce stronger linearly polarized light.

[0063] In this embodiment, an empirical threshold is set, or a threshold is calculated using an adaptive algorithm. All pixels in the feature map with values ​​less than the threshold are set to zero, thereby quickly filtering out a large amount of natural background. On the filtered feature map, a sliding window is used to find local maxima. If the value of a pixel is greater than the values ​​of all other pixels in its neighborhood, the coordinates of that pixel are retained. These coordinates are polarization hotspots, which may be the center or a significant edge of the target. Further, the extracted set of coordinates is used as input, and a two-dimensional sinusoidal position encoding function is used to map each two-dimensional coordinate into a d-dimensional vector. The generated sequence is the initial target query vector.

[0064] The infrared and polarization image fusion detection method provided in this application removes most of the background area through threshold filtering and extracts hotspot pixels through local maximum pooling. The generated query vector directly points to the high-probability target area, thereby reducing the search space for subsequent target detection and significantly reducing the false detection rate of the background. In addition, a two-dimensional sinusoidal position encoding is used to map continuous coordinates to a high-dimensional orthogonal space, preserving the periodicity features and relative distance information of the coordinates. The query vector generated in this way enables the detection head to perceive the relative layout of the target in the image through the frequency components in the encoding, thereby enabling the model to learn the geometric distribution pattern of the target more quickly and improve the accuracy of bounding box regression.

[0065] Furthermore, the high-fidelity fused features are input into the detection head, and the sparse anomalous features (high polarization response points) extracted from the backbone network are used to generate an initial target query vector. The initial target query vector serves as an anchor point with physical prior knowledge, guiding the detection head to perform directional search in the feature space. The initial target query vector is input into the Transformer decoding module of the detection head. Using a cross-attention mechanism, the boundary texture and high-level semantic features of the target are dynamically aggregated in the fused feature map. During the decoding process, the extreme value information in the polarization features is used as a spatial position guide to accurately constrain and refine the prediction of the target bounding box. Combining the updated query vector output by the decoding module, target category classification and bounding box coordinate regression are performed through parallel prediction branches, directly outputting the final target detection result. This embodiment replaces the time-consuming diffusion iteration process with a parallel decoding architecture, significantly improving target localization accuracy while greatly reducing system inference latency.

[0066] In this embodiment, a Polarization-Activated Query Detection Head (PAQ-DETR) is employed to achieve fully parallel decoding of infrared polarization fusion features and high-precision target boundary localization. This detection head utilizes a physically prior-guided target query mechanism, taking extreme value information in polarization features as spatial anchors, transforming the traditional blind search process into a directional feature aggregation process, thus balancing global semantic perception and local boundary representation.

[0067] Here, the target query generation module does not use randomly initialized candidate boxes, but instead directly extracts local maximum coordinates from the sparse anomaly features (high polarization response points) output by the backbone network. Based on the physical properties of polarization, the polarization degree of the surface of man-made targets (such as camouflage nets and metal vehicles) exhibits a significant distortion response. By capturing these physical response extrema and combining them with position encoding, an initial target query vector with clear spatial priors is generated. This approach allows the detector to lock onto potential target regions in the early stages of decoding, greatly alleviating the search pressure against complex infrared backgrounds.

[0068] Here, the converter decoding module uses a multi-layered cross-attention operator to treat the initial target query vector as a "search instruction" and extracts target features in parallel from the global fusion feature map output by the cross-modal state space model. In each decoding layer, the initial query vector not only aggregates high-level semantic information but also utilizes high-frequency details in the polarization feature map to finely adjust the target boundary. Compared to traditional diffusion iterative denoising processes, the decoding mechanism of this application supports the simultaneous output of the class probabilities and regression coordinates of all targets in a single forward propagation, completely eliminating the inference latency caused by iterative calculations and achieving true end-to-end real-time detection.

[0069] Here, cross-scan position encoding maps the spatial coordinates of polarization extrema points to a high-dimensional vector space, providing the detection model with a global positional sense that transcends the local receptive field. Through the self-attention mechanism of the converter architecture, the detection head can automatically evaluate the spatial correlation between different extreme response points, thereby effectively suppressing redundant false alarm information and enhancing the discriminative power of real target features.

[0070] Figure 4 This is a schematic diagram of the detection head provided in this application. Figure 4 As shown, firstly, in the physical response activation phase, the detection head utilizes the sparse anomaly features (polarization feature map) output by the backbone network to locate high-polarization response points through local extremum retrieval. These response points directly reflect the physical attribute differences between the camouflaged target and the natural background, thus providing the system with accurate initial search anchor points. Secondly, in the prior query encoding phase, the coordinates of the aforementioned response points, combined with position encoding, are mapped into an initial target query vector. This vector, as a query instruction with physical priors, guides the subsequent converter architecture to perform directional feature aggregation in the feature space. Finally, in the parallel decoding phase, the detection head uses a multi-layer cross-attention mechanism to associate the initial query vector with the global fusion features output by the state-space model. This allows the detection results to utilize both global thermal radiation semantics and accurately fit local polarization edges, thereby outputting high-precision target bounding boxes and categories in a single forward propagation.

[0071] In the specific implementation process, the fused features are input to the detection head. The detection head first extracts local maxima pixels from the polarization feature map as physical prior coordinates, and transforms these coordinates into an initial target query vector in high-dimensional space through a positional encoding layer. Subsequently, the fused features are used as the key and value, and the initial target query vector is used as the query, both input into a decoder based on a converter architecture. Inside the decoder, through multi-layer cross-attention calculation, the initial query vector automatically collapses towards the target region in the fused feature map, capturing the target's geometric contour and semantic category. Compared to traditional iterative denoising processes, the decoding mechanism of this application supports parallel processing of all target query vectors. Finally, the decoded and updated feature vector is fed into a feedforward neural network and a classifier, directly outputting the target's specific category, confidence level, and predicted bounding box coordinates. The final target detection result not only includes the target's precise location but also possesses semantic consistency guided by physical priors.

[0072] The infrared and polarization image fusion detection method provided in this application employs a polarization activation query detection head that uses polarization extrema as physical anchors, transforming the traditional bounding box regression problem into a feature sampling problem based on physical priors. During feature aggregation, polarization prior information pulls previously blurred candidate regions in the infrared image toward the real physical edges, enabling the detection box to converge rapidly and closely adhere to the target's real material boundary. Particularly when the temperature difference between the target and background is small but the material polarization difference is significant, this application achieves detection sensitivity far exceeding that of a super-connected iterative architecture. Furthermore, based on the global modeling capability of the converter architecture, the detection head can automatically establish spatial relationship constraints between different targets, effectively reducing duplicate detections and missed detections. Combined with parallel output characteristics, it achieves extremely low system inference latency while ensuring optimal detection accuracy and positioning precision.

[0073] In an optional embodiment, the frequency band separation and reconstruction of the fusion loss gradient and the detection loss gradient based on the wavelet frequency domain gradient decoupling mechanism includes: Obtain the detection gradient map obtained by backpropagation of the detection loss function, and the fusion gradient map obtained by backpropagation of the fusion loss function, respectively. Two-dimensional discrete wavelet transform is performed on the detected gradient map and the fused gradient map respectively, decomposing them into low-frequency approximate sub-bands and high-frequency detail sub-bands in the horizontal, vertical and diagonal directions; In the low-frequency approximation sub-band, the components of the detection gradient map are given higher update weights; in the high-frequency detail sub-band, the components of the fused gradient map are given higher update weights. The weighted low-frequency approximate subband and high-frequency detail subband are reconstructed using inverse two-dimensional discrete wavelet transform to obtain the decoupled global update gradient, thereby optimizing the backbone feature extraction network.

[0074] In this embodiment of the application, the detection gradient matrix returned by the detection task and the fusion gradient matrix returned by the fusion task are obtained respectively. Two-dimensional discrete wavelet transform is performed on the detection gradient matrix and the fused gradient matrix respectively, decomposing them into low-frequency approximate subbands (LL) representing low-frequency semantic information, and high-frequency detail subbands (LH, HL, HH) representing high-frequency texture details in the horizontal, vertical, and diagonal directions. Weight decoupling is performed based on multi-task frequency domain preference: the update ratio of the detection gradient is enhanced in the low-frequency approximate sub-band, and the update ratio of the fusion gradient is enhanced in the high-frequency detail sub-band, thereby eliminating gradient conflicts between tasks in the shared parameter space. The decoupled and weighted gradients of each sub-band are aggregated, and the conflict-free global update gradient is reconstructed by inverse two-dimensional discrete wavelet transform. The parameters of each module in the detection model are then trained for consistency optimization.

[0075] In this embodiment, a wavelet-domain gradient decoupling (WDGD) mechanism is used to address the gradient conflict problem in the end-to-end training process of fusion and detection tasks from the fundamental source of the physical signal frequency band. This mechanism utilizes the frequency domain separation characteristics to achieve decoupling optimization between the high-frequency polarization gradient representing material details and the low-frequency detection gradient representing macroscopic semantics, thereby eliminating model training instability caused by performance competition between tasks.

[0076] Here, the low-frequency approximate gradient component is the LL subband obtained by performing a two-dimensional discrete wavelet transform on the gradient matrix. It retains low-frequency semantic information such as the overall outline, category attributes, and spatial location of the target, and is the core energy guiding the detector to perform qualitative and quantitative target identification. The high-frequency detail gradient components are the LH, HL, and HH subbands after wavelet transform, which capture drastically changing pixel edges, micro-polarization textures, and material abrupt changes in the image. The mathematical essence of wavelet frequency domain decoupling lies in using the orthogonality of basis functions to map the task gradients that were originally mixed in the spatial domain to a frequency band space that does not interfere with each other: giving the detection loss a higher update weight in the low-frequency subband and the fusion loss a higher update weight in the high-frequency subband, ensuring that the network can accurately recognize the overall target and delicately depict material details when updating parameters.

[0077] Figure 5 This is a schematic diagram of the wavelet frequency domain gradient decoupling mechanism provided in this application. Figure 5 As shown, firstly, in the gradient frequency domain decomposition stage, WDGD decomposes the gradient matrices back-propagated from the fusion and detection tasks into four frequency sub-bands respectively. Secondly, in the frequency domain weight adaptive allocation stage, based on the physical characteristics of infrared polarization detection, the detection task is highly sensitive to the target contour (low frequency), while the fusion task is highly sensitive to the material polarization state (high frequency). Accordingly, the system dynamically adjusts the gradient gain of different sub-bands to suppress fusion noise in the low-frequency space and enhance the polarization detail response in the high-frequency space. Finally, in the inverse transform gradient reconstruction stage, the processed frequency domain gradient is restored to the spatial domain through inverse wavelet transform, constructing a globally consistent, conflict-free updated gradient, achieving efficient evolution of the backbone network parameters.

[0078] In the specific implementation, during backpropagation, the fusion gradient matrix corresponding to the fusion loss function and the detection gradient matrix corresponding to the detection loss function are calculated separately. Single-layer or multi-layer two-dimensional discrete wavelet transforms are then performed on both matrices to extract low-frequency approximate subbands and multiple high-frequency detail subbands. Significant gain coefficients are assigned to the low-frequency subbands of the detection gradient and the high-frequency subbands of the fusion gradient, respectively, and the reconstructed gradient matrices are then fused. This mechanism ensures that gradient updates are independent in the frequency band dimension. Updates of shallow parameters can focus on capturing polarization physical edges, while updates of deep parameters can be highly focused on the target semantic logic.

[0079] The infrared and polarization image fusion detection method provided in this application embodiment has a joint multi-task loss function that includes fusion decoding loss and detection binary matching loss.

[0080] The fusion decoding loss includes infrared thermal radiation pixel reconstruction loss, multi-scale structural similarity loss, and Stokes parameter physical consistency constraint. The pixel reconstruction loss reflects the direct difference between the fused features and the source image in pixel intensity values ​​(S0 component); the structural similarity loss forces the model to retain texture details in the polarization image by calculating local contrast and structural correlation.

[0081] Here, the Stokes parameter physical consistency constraint is based on the Stokes vector equations of polarization optical imaging. In practice, the predicted Stokes parameters are solved through a parameterized inversion branch using fused features, and the predicted degree of polarization and polarization angle are calculated accordingly. A physical closed-loop comparison is performed between the inverted predicted polarization map and the original input polarization map. The resulting physical consistency error forces the feature representation learned by the network to strictly follow the polarization laws of real optics, rather than simply mathematical fitting. This gives the feature representation explicit physical interpretability and effectively avoids the generation of "illusionary" features that do not conform to physical common sense during deep training.

[0082] The detection binary matching loss is specifically designed for detection heads based on a converter architecture. It utilizes the Hungarian Algorithm to establish a unique one-to-one correspondence between the predicted set and the true target set, specifically including classification cost and bounding box regression cost, such as a combination of GIoU loss and L1 loss.

[0083] This application's embodiments balance the demands of different tasks for image features across different frequency bands by using wavelet frequency domain gradient decoupling. This allows the network to evolve independently along the core physical dimensions of each task when updating parameters, achieving true synergistic progress in semantic recognition and detail reconstruction. Through Stokes physical consistency regularization, the network is forced to learn robust features that conform to optical laws, greatly improving the model's generalization ability and reliability when dealing with targets of unknown materials. The final trained detection model contains rich physical material fingerprints and possesses extremely high semantic saliency, achieving a qualitative leap in infrared polarization detection performance.

[0084] In an optional embodiment, the detection model is trained based on a joint loss function, which includes a detection loss and a fusion loss; the fusion loss includes pixel reconstruction loss, structural similarity loss, and Stokes parameter physical consistency constraint. The Stokes parameter physical consistency constraint is used to force the polarization degree and polarization angle calculated by the fused feature inversion to maintain mathematical consistency with the polarization degree and polarization angle of the input polarization degree image and polarization angle image in terms of physical optical properties.

[0085] Here, pixel reconstruction loss is used to measure the difference in pixel intensity between the fused image and the source image, ensuring that the fused image retains the original brightness information of the source image and avoids overall darkening or brightening.

[0086] Here, structural similarity loss measures image similarity from three dimensions: brightness, contrast, and structure. It forces the fused images to retain high-frequency structural information such as edges and textures of the source images, thus preventing image blurring.

[0087] Here, the Stokes parameters describe a set of four parameter vectors S=[S0,S1,S2,S3] that represent the polarization state of the light wave. ， Where S0 represents the total light intensity, S1 and S2 describe the linear polarization components, and S3 describes the circular polarization components. The Stokes parameter physical consistency constraint requires that the degree of polarization and polarization angle derived from the fused features after inversion calculation must be mathematically consistent with the actual input physical quantities to prevent the network from generating illusory features that violate optical principles.

[0088] The infrared and polarization image fusion detection method provided in this application locks in structural information through structural similarity loss, preventing the network from excessively distorting the image structure in pursuit of detection scores, thereby improving the model's robustness in complex scenes; by forcing the fused features to follow the Stokes geometry in the latent space through Stokes parameters, the fused features not only contain visual information but also retain accurate material properties, ensuring the interpretability of the features.

[0089] Figure 6 This is the second flowchart illustrating the infrared and polarization image fusion detection method provided in this application. Figure 6As shown, the end-to-end infrared polarization image fusion detection network provided in this application simultaneously introduces infrared intensity image and polarization state information at the input end. By performing sine-cosine encoding on the polarization angle, a physically stable polarization feature expression is obtained. In the feature extraction stage, a low-rank sparse decomposition backbone network based on algorithm expansion is adopted to explicitly separate the image from the physical source into low-rank components representing the environmental background and sparse components representing the material edge, realizing the initial decoupling of the underlying physical properties. In the fusion stage, a cross-modal polarization state space model is introduced. Using a long-distance dependent state space scanning mechanism, the state evolution of the infrared intensity feature sequence is dynamically modulated by the hidden state of the polarization sequence, realizing global feature penetration and high-quality fusion at the physical level. In the detection stage, a polarization activation query detection head is adopted. The polarization sparse response extrema are used as physical anchors to generate an initial target query vector. Parallel decoding through a converter architecture is used to achieve high-precision positioning and semantic recognition of the target boundary. Finally, in the optimization training stage, a wavelet frequency domain gradient decoupling mechanism is proposed. Discrete wavelet transform is used to separate multi-task conflict gradients in the frequency domain to ensure that the model can collaboratively learn high-frequency material details and low-frequency target semantics. Through this integrated design, this application effectively solves the core problems of existing methods, such as feature space aliasing, insufficient global physical dependency capture, and end-to-end training gradient conflict. It generates infrared polarization fusion detection results that combine thermal infrared sensitivity, polarization material clarity, high detection accuracy and low inference latency, significantly improving the system's robustness in detecting camouflaged targets in complex scenarios.

[0090] The infrared and polarization image fusion detection device provided in the embodiments of this application is described below. The infrared and polarization image fusion detection device described below and the infrared and polarization image fusion detection method described above can be referred to and correspond to each other.

[0091] Figure 7 This is a schematic diagram of the infrared and polarization image fusion detection device provided by the present invention, as shown below. Figure 7 As shown, the infrared and polarization image fusion detection device may include, but is not limited to: The feature extraction module 710 is used to: input the infrared intensity image, polarization degree image and polarization angle image into the backbone feature extraction network of the detection model, obtain the corresponding output infrared features, polarization degree features and polarization angle features, and flatten the infrared features, polarization degree features and polarization angle features into a one-dimensional multimodal feature sequence; The feature fusion module 720 is used to: input the multimodal feature sequence into the cross-modal polarization state space model of the detection model, and dynamically modulate the state transition equation of the infrared intensity feature sequence using the polarization state feature sequence to obtain the fused features. The polarization state feature sequence includes a polarization degree feature sequence and a polarization angle feature sequence. The query generation module 730 is used to: extract the local extreme value coordinates in the polarization degree feature as spatial priors, and combine them with the position encoding to generate an initial target query vector; The target detection module 740 is used to: input the fused features and the initial target query vector into the detection head based on the converter architecture in the detection model, and obtain the target detection result output by the detection head; The parameter optimization module 750 is used to: perform frequency band separation and reconstruction of the fusion loss gradient and the detection loss gradient based on the wavelet frequency domain gradient decoupling mechanism during the backpropagation stage of the detection model training; update the shared parameters of the backbone network based on the reconstructed conflict-free gradient; wherein the fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion; and the detection loss gradient is the loss gradient of the detection head for target detection.

[0092] It should be noted that the infrared and polarization image fusion detection device provided in this embodiment of the invention can execute the infrared and polarization image fusion detection method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0093] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an infrared and polarization image fusion detection method, which includes: The infrared intensity image, polarization degree image, and polarization angle image are input into the backbone feature extraction network of the detection model to obtain the corresponding output infrared features, polarization degree features, and polarization angle features. The infrared features, polarization degree features, and polarization angle features are then flattened into a one-dimensional multimodal feature sequence. The multimodal feature sequence is input into the cross-modal polarization state space model of the detection model, and the state transition equation of the infrared intensity feature sequence is dynamically modulated by the polarization state feature sequence to obtain the fused feature. The polarization state feature sequence includes the polarization degree feature sequence and the polarization angle feature sequence. Local extreme value coordinates are extracted from the polarization degree features as spatial priors and combined with position encoding to generate an initial target query vector. The fused features and the initial target query vector are input into the detection head based on the converter architecture in the detection model to obtain the target detection result output by the detection head; During the backpropagation phase of the detection model training, the fusion loss gradient and the detection loss gradient are separated and reconstructed based on the wavelet frequency domain gradient decoupling mechanism. The shared parameters of the backbone feature extraction network are updated based on the reconstructed conflict-free gradient. The fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion, and the detection loss gradient is the loss gradient of the detection head for target detection.

[0094] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the infrared and polarization image fusion detection method provided by the above methods, the method including: The infrared intensity image, polarization degree image, and polarization angle image are input into the backbone feature extraction network of the detection model to obtain the corresponding output infrared features, polarization degree features, and polarization angle features. The infrared features, polarization degree features, and polarization angle features are then flattened into a one-dimensional multimodal feature sequence. The multimodal feature sequence is input into the cross-modal polarization state space model of the detection model, and the state transition equation of the infrared intensity feature sequence is dynamically modulated by the polarization state feature sequence to obtain the fused feature. The polarization state feature sequence includes the polarization degree feature sequence and the polarization angle feature sequence. Local extreme value coordinates are extracted from the polarization degree features as spatial priors and combined with position encoding to generate an initial target query vector. The fused features and the initial target query vector are input into the detection head based on the converter architecture in the detection model to obtain the target detection result output by the detection head; During the backpropagation phase of the detection model training, the fusion loss gradient and the detection loss gradient are separated and reconstructed based on the wavelet frequency domain gradient decoupling mechanism. The shared parameters of the backbone feature extraction network are updated based on the reconstructed conflict-free gradient. The fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion, and the detection loss gradient is the loss gradient of the detection head for target detection.

[0096] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the infrared and polarization image fusion detection method provided by the methods described above, the method comprising: The infrared intensity image, polarization degree image, and polarization angle image are input into the backbone feature extraction network of the detection model to obtain the corresponding output infrared features, polarization degree features, and polarization angle features. The infrared features, polarization degree features, and polarization angle features are then flattened into a one-dimensional multimodal feature sequence. The multimodal feature sequence is input into the cross-modal polarization state space model of the detection model, and the state transition equation of the infrared intensity feature sequence is dynamically modulated by the polarization state feature sequence to obtain the fused feature. The polarization state feature sequence includes the polarization degree feature sequence and the polarization angle feature sequence. Local extreme value coordinates are extracted from the polarization degree features as spatial priors and combined with position encoding to generate an initial target query vector. The fused features and the initial target query vector are input into the detection head based on the converter architecture in the detection model to obtain the target detection result output by the detection head; During the backpropagation phase of the detection model training, the fusion loss gradient and the detection loss gradient are separated and reconstructed based on the wavelet frequency domain gradient decoupling mechanism. The shared parameters of the backbone feature extraction network are updated based on the reconstructed conflict-free gradient. The fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion, and the detection loss gradient is the loss gradient of the detection head for target detection.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for infrared and polarization image fusion detection, characterized in that, include: The infrared intensity image, polarization degree image, and polarization angle image are input into the backbone feature extraction network of the detection model to obtain the corresponding output infrared features, polarization degree features, and polarization angle features. The infrared features, polarization degree features, and polarization angle features are then flattened into a one-dimensional multimodal feature sequence. The multimodal feature sequence is input into the cross-modal polarization state space model of the detection model, and the state transition equation of the infrared intensity feature sequence is dynamically modulated by the polarization state feature sequence to obtain the fused feature. The polarization state feature sequence includes the polarization degree feature sequence and the polarization angle feature sequence. Local extreme value coordinates are extracted from the polarization degree features as spatial priors and combined with position encoding to generate an initial target query vector. The fused features and the initial target query vector are input into the detection head based on the converter architecture in the detection model to obtain the target detection result output by the detection head; During the backpropagation phase of the detection model training, the fusion loss gradient and the detection loss gradient are separated and reconstructed based on the wavelet frequency domain gradient decoupling mechanism. The shared parameters of the backbone feature extraction network are updated based on the reconstructed conflict-free gradient. The fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion, and the detection loss gradient is the loss gradient of the detection head for target detection.

2. The infrared and polarization image fusion detection method according to claim 1, characterized in that, The backbone feature extraction network contains deep neural network layers based on an approximate gradient descent algorithm. The specific feature extraction process includes: The input infrared intensity image, polarization degree image, and polarization angle image are mapped into a feature matrix; In the forward propagation of a differentiable neural network, a parameterized singular value threshold shrinkage operator is used to solve the kernel norm minimization of the feature matrix, and the output is a low-rank background feature representing a smooth thermal radiation or natural polarization background. The L1 norm of the feature matrix is ​​minimized by a parameterized soft threshold shrinkage operator, and the sparse anomalous features representing the edges and polarization distortion abrupt points of the artificial material are output. After assigning learnable modal weights to the low-rank background features and the sparse anomaly features respectively, the features are concatenated to obtain the final infrared features, polarization degree features and polarization angle features.

3. The infrared and polarization image fusion detection method according to claim 1, characterized in that, The feature fusion process of the cross-modal polarization state space model includes: During the discretization state transition process of the state-space model, the hidden states of the polarization degree feature sequence and the polarization angle feature sequence are linearly projected and then injected as input dependency coefficients into the state equation of the infrared intensity feature sequence. The state transition matrix and input matrix of the infrared intensity feature sequence are dynamically adjusted by the input dependency coefficient, and the fused features are determined based on the adjusted state transition matrix and input matrix.

4. The infrared and polarization image fusion detection method according to claim 1, characterized in that, The step of extracting local extremum coordinates from the polarization degree features as spatial priors and combining them with position encoding to generate an initial target query vector includes: Threshold filtering and local maximum pooling operations are applied to the polarization degree feature to extract the coordinates of polarization hotspot pixels with response values ​​higher than a preset threshold. The coordinates of polarization hotspot pixels represent the physical polarization distortion region between the artificial material and the natural background. The extracted polarization hotspot pixel coordinates are mapped into an initial target query vector using two-dimensional sinusoidal position encoding.

5. The infrared and polarization image fusion detection method according to claim 1, characterized in that, The frequency band separation and reconstruction of the fusion loss gradient and detection loss gradient based on the wavelet frequency domain gradient decoupling mechanism includes: Obtain the detection gradient map obtained by backpropagation of the detection loss function, and the fusion gradient map obtained by backpropagation of the fusion loss function, respectively. Two-dimensional discrete wavelet transform is performed on the detected gradient map and the fused gradient map respectively, decomposing them into low-frequency approximate sub-bands and high-frequency detail sub-bands in the horizontal, vertical and diagonal directions; In the low-frequency approximation sub-band, the components of the detection gradient map are given higher update weights; in the high-frequency detail sub-band, the components of the fused gradient map are given higher update weights. The weighted low-frequency approximate subband and high-frequency detail subband are reconstructed using inverse two-dimensional discrete wavelet transform to obtain the decoupled global update gradient, thereby optimizing the backbone feature extraction network.

6. The infrared and polarization image fusion detection method according to claim 1, characterized in that, The detection model is trained based on a joint loss function, which includes detection loss and fusion loss; the fusion loss includes pixel reconstruction loss, structural similarity loss and Stokes parameter physical consistency constraint. The Stokes parameter physical consistency constraint is used to force the polarization degree and polarization angle calculated by the fused feature inversion to maintain mathematical consistency with the polarization degree and polarization angle of the input polarization degree image and polarization angle image in terms of physical optical properties.

7. An infrared and polarization image fusion detection device, characterized in that, include: The feature extraction module is used to: input the infrared intensity image, polarization degree image and polarization angle image into the backbone feature extraction network of the detection model, obtain the corresponding output infrared features, polarization degree features and polarization angle features, and flatten the infrared features, polarization degree features and polarization angle features into a one-dimensional multimodal feature sequence; The feature fusion module is used to: input the multimodal feature sequence into the cross-modal polarization state space model of the detection model, and dynamically modulate the state transition equation of the infrared intensity feature sequence using the polarization state feature sequence to obtain the fused features. The polarization state feature sequence includes a polarization degree feature sequence and a polarization angle feature sequence. The query generation module is used to: extract the local extreme value coordinates in the polarization degree features as spatial priors, and combine them with position encoding to generate an initial target query vector; The target detection module is used to: input the fused features and the initial target query vector into the detection head based on the converter architecture in the detection model, and obtain the target detection result output by the detection head; The parameter optimization module is used to: perform frequency band separation and reconstruction of the fusion loss gradient and the detection loss gradient based on the wavelet frequency domain gradient decoupling mechanism during the backpropagation stage of the detection model training; update the shared parameters of the backbone network based on the reconstructed conflict-free gradient; wherein the fusion loss gradient is the loss gradient of the cross-modal polarization state space model for feature fusion; and the detection loss gradient is the loss gradient of the detection head for target detection.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the infrared and polarization image fusion detection method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the infrared and polarization image fusion detection method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the infrared and polarization image fusion detection method as described in any one of claims 1 to 6.

Images