A multi-modal image fusion method and system based on RGB images and polarization images

Abstract

The application relates to the technical field of computer imaging and image processing, and discloses a multi-modal image fusion method and system based on RGB images and polarization images, which comprises the following steps: acquiring RGB images and polarization images synchronously collected under the same scene, and performing demosaicing reconstruction, polarization parameter calculation and spatial registration processing on the polarization images; performing multi-scale feature coding on the RGB images and the polarization images respectively to obtain corresponding multi-layer feature representations; according to feature level differences, low-layer features are fused in a cross-modal attention interaction mode, high-layer features are fused in a feature enhancement mode, and fusion features are generated; and the fusion features are decoded and reconstructed to obtain a fusion output image. By adopting differentiated fusion strategies at different feature levels, the application enhances structure and edge information while maintaining the naturalness of image colors, and improves the imaging robustness under complex light and strong reflection environments.

Description

Technical Field

[0001] This invention relates to the field of computational imaging and image processing technology, and in particular to a multimodal image fusion method and system based on RGB images and polarization images, which can be applied to fields such as intelligent monitoring, autonomous driving, remote sensing, medical imaging and visual perception of complex environments. Background Technology

[0002] With the widespread application of artificial intelligence and multimodal perception technologies in fields such as intelligent transportation, security monitoring, remote sensing, and medical imaging, image-based environmental perception capabilities have become a crucial foundation for various intelligent systems. Among these, RGB images, due to their intuitive content, natural colors, and mature equipment, have long been widely used for tasks such as target recognition, scene understanding, and behavior analysis. However, RGB imaging relies on the reflectivity of object surfaces to incident light, making it highly susceptible to factors such as brightness fluctuations, strong reflection interference, low-light noise, and atmospheric scattering under complex lighting conditions. This can lead to decreased image contrast, weakened details and textures, unclear edges, and even targets being submerged in the background, thus affecting overall perception quality and the stability of subsequent tasks.

[0003] In contrast, polarization imaging is a novel imaging method that utilizes the polarization characteristics of light waves to acquire information. When light is reflected from an object's surface or scattered in a medium, its polarization state typically undergoes characteristic changes. Different materials, roughness, or surface structures cause differences in polarization information. Therefore, polarization images have unique advantages in revealing the surface structure, normal information, material properties, and scattering characteristics of objects. Especially in highly reflective scenes such as water surfaces, glass surfaces, and metal surfaces, polarization imaging can effectively enhance edge and detail structures, overcoming to some extent the difficulty in recognizing features in traditional RGB imaging.

[0004] However, using polarization images alone for visual perception also has significant limitations. On the one hand, polarization imaging does not record complete spectral and color information, and its visual representation lacks the intuitiveness of natural scenes, which is not conducive to human observation and some algorithmic processing. On the other hand, in weakly polarized regions or complex scattering backgrounds, polarization features may also be affected by noise, thereby reducing stability. Therefore, effectively fusing RGB images with polarization images, enabling the two modalities to complement each other within the same framework, has become an important technical direction for improving imaging robustness.

[0005] Existing multimodal fusion methods mainly fall into two categories. One category comprises fusion strategies based on traditional signal processing methods, such as wavelet decomposition, pyramid decomposition, sparse representation, and weighted fusion. These methods typically rely on manually designed fusion rules, making it difficult to characterize complex cross-modal relationships and limiting their adaptability to different scenarios. The other category consists of deep learning-based fusion methods, which automatically learn modal features through convolutional networks or attention mechanisms and complete the fusion at the feature layer. While these methods have achieved some improvement in image representation capabilities, they still generally suffer from the following problems: First, most methods primarily focus on the fusion of grayscale or infrared images with polarization information, lacking joint modeling of color information and polarization physical features; second, existing models often employ a uniform fusion approach across different levels, ignoring the differences in expression and importance between shallow texture and deep semantic features, easily leading to feature redundancy or conflicts; third, the utilization of polarization physical meaning remains insufficient, lacking stable structural consistency constraints, which can easily cause structural breaks or optical information distortion.

[0006] Therefore, how to fully explore the advantages of polarization imaging in terms of reflection, scattering and material representation while ensuring natural colors and clear structures, and how to achieve cross-modal deep fusion in a more reasonable way, has become an urgent problem to be solved in the current technical field. Summary of the Invention

[0007] This invention provides a multimodal image fusion method and system based on RGB images and polarization images. It addresses the problems of insufficient structural detail expression, limited color reproduction, and weak anti-interference ability of existing imaging technologies under complex lighting conditions, strong reflection environments, and weak texture target scenes. By introducing polarization physical characteristics, hierarchical feature fusion strategies, and joint reconstruction constraint mechanisms, it achieves complementary enhancement of natural imaging and polarization imaging, thereby improving the structural clarity, edge stability, and perceived naturalness of the fused image.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a multimodal image fusion method based on RGB images and polarization images, comprising the following steps:

[0009] S110, acquire RGB images and polarization images synchronously acquired in the same scene, wherein the polarization image is acquired by a polarization imaging device containing multiple filtering units with different polarization directions;

[0010] S120, the polarization image is subjected to demosaic reconstruction processing to restore the complete image corresponding to each polarization direction, and polarization parameters are calculated based on the reconstructed image corresponding to each polarization direction to generate a polarization feature map containing at least one of polarization intensity feature, polarization degree feature and polarization angle feature; at the same time, the RGB image and the polarization feature map are spatially registered to achieve pixel-level or sub-pixel-level alignment.

[0011] S130, perform multi-scale feature encoding processing on the RGB image and the polarization feature map respectively, and obtain corresponding first modal features and second modal features by extracting features at different scales, wherein the first modal features and the second modal features each include at least low-level texture features and high-level structure features;

[0012] S140, according to the different feature levels, the first modal features and the second modal features are fused in layers, wherein: the low-level texture features are fused using a cross-modal attention interaction method to establish the correlation between the first modal features and the second modal features; the high-level structural features are fused using a feature enhancement method to strengthen structural information and suppress cross-modal feature conflicts;

[0013] S150 uses a fusion model to decode and reconstruct the fused features, generating a fused output image.

[0014] A multimodal image fusion system based on RGB images and polarization images, used to execute the multimodal image fusion method based on RGB images and polarization images, includes:

[0015] The acquisition module is used to acquire RGB images and polarization images that are synchronously collected in the same scene;

[0016] The data preprocessing module is used to perform demosaic reconstruction, polarization parameter calculation, and spatial registration on RGB and polarized images.

[0017] The multi-scale feature encoding module is used to perform multi-scale feature encoding processing on the RGB image and the polarization feature map respectively, and to obtain the corresponding first mode feature and second mode feature by extracting features at different scales;

[0018] The hierarchical fusion module is used to fuse the first modal feature and the second modal feature using differentiated fusion strategies according to the different feature levels.

[0019] The image reconstruction module is used to decode the fused features and generate a fused output image.

[0020] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal image fusion method based on RGB images and polarization images.

[0021] Compared with existing technologies, the advantages of this invention are as follows: This multimodal fusion method based on RGB and polarized images fully considers the complementary characteristics of polarization physical information and natural image information in its structural design. Through a layered fusion strategy, shallow texture and deep semantics work synergistically during the fusion process, maintaining clear structural contours and natural color reproduction in complex environments such as weak texture, strong reflection, and low contrast. On the one hand, this invention avoids the problem of insufficient structural expression in single-modal algorithms; on the other hand, it enhances the physical meaning expression of optical information through polarization consistency constraints, giving the fusion results higher reliability and engineering application value. Furthermore, the method of this invention has advantages such as reasonable model structure, controllable computational complexity, and strong versatility, and can meet the deployment requirements of visual perception systems in various scenarios. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the acquisition and preprocessing results of RGB color images and micro-polarized images in this invention;

[0023] Figure 2 This is a schematic diagram of the network structure of the fusion model used in the multimodal image fusion method of the present invention;

[0024] Figure 3 This is a schematic diagram of the shallow feature enhancement strategy in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the high-level feature residual enhancement fusion strategy in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram illustrating the fusion effect of RGB and polarization images in an embodiment of the present invention. Detailed Implementation

[0027] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0028] This invention provides a multimodal image fusion method based on RGB images and polarization images, comprising the following steps:

[0029] S110, acquire RGB images and polarization images synchronously acquired in the same scene, wherein the polarization image is acquired by a polarization imaging device containing multiple filtering units with different polarization directions;

[0030] The input data includes RGB color images acquired synchronously in the same scene and polarized images output by a micro-polarization camera. The micro-polarization camera adopts a focal plane segmentation structure and periodically arranges four-directional micro-polarization filter units at 0°, 45°, 90° and 135° on the surface of a single sensor to obtain an original mosaic format image with polarization direction sampling information.

[0031] S120, the polarization image is subjected to demosaic reconstruction processing to restore the complete image corresponding to each polarization direction, and polarization parameters are calculated based on the reconstructed image corresponding to each polarization direction to generate a polarization feature map containing at least one of polarization intensity feature, polarization degree feature and polarization angle feature; at the same time, the RGB image and the polarization feature map are spatially registered to achieve pixel-level or sub-pixel-level alignment.

[0032] Since polarization information is spatially staggered, the original output is difficult to directly use as network input. Therefore, this invention first performs demosaic reconstruction on the polarization image, and then recovers the four-directional polarization intensity image by using an improved residual iterative interpolation algorithm while maintaining the original edge gradient structure.

[0033] Polarization parameters are calculated based on the reconstructed images corresponding to each polarization direction, including: calculating Stokes parameters based on the images corresponding to different polarization directions, and obtaining at least one of polarization intensity information, degree of polarization information, and polarization angle information based on the Stokes parameters. Specifically, this process includes: calculating Stokes parameters based on polarization imaging theory: total light intensity... 0° minus 90° linear polarization component 45° minus 135° linear polarization component This yields the linear polarization degree DoLP and the polarization angle AoLP, thus forming a physically meaningful three-channel polarization characteristic vector. Subsequently, this invention employs a feature point matching-driven image registration method to achieve complete pixel-level alignment between the RGB image and the polarization feature map. Lightweight filtering is then applied to DoLP and AoLP to suppress noise, resulting in spatially consistent and signal-to-noise ratio-stable two-modal input data. The DoLP and AoLP effects are as follows: Figure 1 As shown.

[0034] S130, perform multi-scale feature encoding processing on the RGB image and the polarization feature map respectively, and obtain corresponding first modal features and second modal features by extracting features at different scales, wherein the first modal features and the second modal features each include at least low-level texture features and high-level structure features;

[0035] Multi-scale feature encoding involves using feature extraction operators with different receptive field sizes to extract features from images at multiple levels, so as to simultaneously obtain local texture features, edge features, and global structural features of the image.

[0036] S140, according to the different feature levels, the first modal features and the second modal features are fused in layers, wherein: the lower-level features are fused using a cross-modal attention interaction method to establish the correlation between RGB features and polarization features; the higher-level features are fused using a feature enhancement method to strengthen structural information and suppress cross-modal feature conflicts;

[0037] Cross-modal attention interaction methods include: calculating the correlation between RGB features and polarization features based on attention weights to establish long-range dependencies between the two modal features, thereby enhancing the shallow feature consistency between color information and polarization texture information.

[0038] Feature enhancement methods include: jointly modeling RGB features and polarization features in the channel dimension and / or spatial dimension, and maintaining the stability of deep features and structural continuity during the fusion process by introducing residual connections or feature overlay methods.

[0039] S150 uses a fusion model to decode and reconstruct the fused features, generating a fused output image.

[0040] During the training of the fusion model, a polarization consistency constraint based on polarization salient regions is introduced, wherein the polarization salient regions are determined by gradient changes or saliency information in the polarization feature map, in order to enhance the structural continuity and directional consistency of the fused output image in the polarization edge region.

[0041] Regarding the network structure of the fusion model, this invention proposes, as follows: Figure 2The diagram illustrates a dual-branch coding framework. This network structure comprises an RGB coding branch and a polarization coding branch, two independent and structurally symmetrical sub-networks that receive RGB image I_RGB and polarization feature image P as input, respectively. Both branches employ multi-scale convolutional structures to extract texture and semantic features from shallow to deep layers. The convolutional kernel sizes are 3×3, 5×5, 7×7, and 11×11, respectively. ReLU and batch normalization (BN) operations are introduced after convolution to progressively build a feature pyramid representation from local texture to overall structure. The RGB branch focuses more on extracting color, brightness, and detailed texture information, while the polarization branch focuses on encoding reflection structure, orientation consistency, and material property differences.

[0042] The fusion model also includes a dual fusion module. To achieve efficient fusion of cross-modal information, this invention designs a two-level fusion strategy, including a shallow Transformer fusion module (DF-T) and a deep residual fusion module (DF-R). The shallow fusion module adopts a lightweight Transformer structure and establishes a global dependency mapping relationship between the low-level texture features of the RGB image and the low-level texture features of the polarization feature map through a multi-head self-attention mechanism, so that the polarization texture is more naturally embedded in the color structure, thereby improving the recognizability of weak texture regions. In the deep semantic feature fusion stage, an improved residual dense convolution structure is adopted to concatenate the high-level structural features of the first and second modal features in the channel dimension, and enhance the consistency of semantic structure through depthwise separable convolution. At the same time, residual feedback is introduced to suppress gradient decay, thereby ensuring the stability and robustness of the fusion process. The fusion features obtained after the above dual-branch encoding process are fed into the hierarchical decoder (hierarchical decoding module) one by one. The resolution is restored to the original image through step-by-step upsampling and deconvolution operations, and finally a three-channel fusion image is output through 1×1 convolution, so that the fusion result has both natural color expression and clear structural outline.

[0043] In the training phase of the fusion model, to improve the overall quality of the fused image in terms of color fidelity, texture clarity, and polarization physical consistency, this invention constructs a joint loss function. This function incorporates pixel-level brightness reconstruction, gradient structure preservation, deep perception consistency, and significant polarization edge constraints into a unified optimization objective, obtaining the optimal parameter model through end-to-end training. Ultimately, the method of this invention can significantly improve the visual structure representation ability and observation stability under complex imaging conditions such as low illumination, strong reflection, and weak texture. The fused image exhibits good natural perception effects and engineering applicability, as shown in the following figures. Figure 5 As shown.

[0044] The method of the present invention is described below with reference to specific examples:

[0045] (1) Acquisition and preprocessing of RGB color images and micro-polarized images

[0046] In the specific implementation of this invention, an imaging system that simultaneously acquires RGB color images and polarization information is used, wherein the RGB color camera is used to acquire three-channel color images of real scenes. It provides complete spectral intensity and color distribution; while the micro-polarization camera is based on the division-of-focal-plane (DoFP) structure, embedding periodically arranged polarization filter units of 0°, 45°, 90° and 135° on the surface of a single image sensor, so that each physical pixel only measures the polarization intensity value in that direction, thereby forming an original polarization image with a polarization sampling mosaic structure.

[0047] Due to the non-uniformity of spatial sampling in the DoFP polarization array, direct input into the network would cause severe geometric distortion. Therefore, this invention first performs interpolation reconstruction and polarization physical quantity calculation on the DoFP data. Let the four-directional polarization sampling intensities be:

[0048] ,

[0049] The Stokes vector components are then calculated based on polarization theory:

[0050] ,

[0051] Furthermore, the degree of linear polarization and the polarization angle are calculated:

[0052] ,

[0053] in, To prevent tiny positive numbers whose denominators approach zero.

[0054] The Stokes component, along with the degree of polarization and the polarization angle, constitutes a physically meaningful polarization characteristic vector.

[0055] ,

[0056] The vector is organized into a three-channel image, serving as the input to the polarization branch of the subsequent deep network. To ensure spatial consistency between the RGB data and polarization data, this invention employs a rigid + non-rigid hybrid registration algorithm based on keypoint matching and geometric transformation estimation to register the two modal images. Based on this, [further details are needed]. and Perform guided filtering or bilateral filtering to reduce the amplification effect of polarization noise while preventing the edge structure from being overly smoothed.

[0057] After this processing, spatially consistent and noise-controlled input pairs are obtained, as shown in the following figure. Figure 1 As shown:

[0058] .

[0059] (2) Design of dual-branch coding framework

[0060] like Figure 2 The multimodal image fusion network proposed in this invention adopts a dual-branch parallel encoding and hierarchical fusion overall structural framework. The RGB image serves as the first input modality, and the polarization feature map serves as the second input modality, both input to two structurally symmetrical but parameter-independent feature encoding branches. Both branches employ a top-down multi-scale pyramid structure, extracting modal representation information at different scales through layer-by-layer convolution and downsampling operations. This allows shallow features to focus on texture details and local brightness variations, while higher-level features focus on semantic structure and overall scene relationships. Let the RGB modal encoding features be:

[0061] ,

[0062] The polarization mode coding features are:

[0063] ,

[0064] in, This represents a network hierarchical index, with the semantic abstraction level increasing progressively from shallow to deep. This represents the total number of network layers. Both encoding branches employ a combination of convolutional layers, batch normalization layers, and nonlinear activation layers in their structural design. Simultaneously, the receptive field expansion rate is carefully controlled, allowing the network to gradually enhance its cross-regional representation capabilities while maintaining sensitivity to local details. After feature extraction, the features at the same level output by the two branches are fed one-to-one into the subsequent fusion module, achieving cross-modal layer-by-layer collaborative modeling. Unlike traditional simple concatenation or weighting methods, this invention maintains the independent evolution of modal features during the encoding stage, avoiding semantic pollution or modal interference caused by early fusion. This allows each modality to fully leverage its physical property advantages, thus providing a more stable and distinguishable feature foundation for subsequent structured fusion.

[0065] (3) Two-level fusion strategy

[0066] In the fusion framework proposed in this invention, after completing the dual-branch encoding, this invention designs a hierarchical, progressive dual fusion strategy to address the differences in semantic attributes, physical origins, and information density of features at different network levels. This strategy is used to process shallow and deep features separately, ensuring that the fusion method matches the feature representation level. For example... Figure 3 As shown, for shallow features In addition to low- and mid-level features, this invention introduces a cross-modal Transformer structure. Through a multi-head self-attention mechanism, it establishes global dependencies between two modalities, achieving dynamic alignment of pixel textures, local reflections, and material details in the cross-modal space. Specifically, this structure utilizes feature concatenation to aggregate multi-source inputs, achieving initial feature alignment through linear mapping. Subsequently, a feedforward network is used to mine higher-order complex mapping relationships between multimodalities, and summation and normalization operations are combined to stabilize the inter-layer feature distribution. The above process achieves dynamic alignment of pixel textures, local reflections, and material details in the cross-modal space, and its fusion form can be expressed as:

[0067] ,

[0068] in, This represents a lightweight Transformer fusion operator. It is a low-level texture feature of an RGB image. This is a low-level texture feature of the polarization feature map. The process focuses on enhancing the consistency between RGB color texture information and polarization structure details, thereby improving texture distortion under illumination reflection interference.

[0069] like Figure 4 As shown, for high-level features This invention employs a residual-enhanced fusion strategy, which strengthens the structural skeleton and semantic contour representation through feature superposition and nonlinear mapping. The fusion form is as follows:

[0070] ,

[0071] in, This is the residual transformation function, used to adjust the proportion of polarization feature contribution and suppress mode conflict. It is a high-level structural feature of RGB images. This involves high-level structural features of the polarization feature map, which enhances scene structural stability and edge continuity while preserving the natural look of RGB. Finally, the shallow and deep fusion results are progressively fed back into the decoding module, and a fused output image is generated through upsampling and convolutional reconstruction, achieving a unified expression of color realism, structural clarity, and polarization physical consistency. This strategy first merges multimodal deep features through channel stitching, then employs DW convolution (channel-wise convolution) and PW convolution (pointwise convolution) in synergy to extract spatial and channel information. During network propagation, dense connections are introduced to establish cross-layer feature reuse to alleviate gradient vanishing, and BN (batch normalization) and ReLU activation functions are combined to improve nonlinear fitting ability and convergence speed. These operational units complete feature overlay and nonlinear mapping, thereby strengthening the structural skeleton and semantic contour expression.

[0072] (4) Training strategy and joint loss design

[0073] In the specific implementation of this invention, in order to ensure the overall color consistency and structural fidelity of the fusion network while fully preserving the structural advantages of polarization imaging at the target contour and material boundaries, a joint loss constraint mechanism was designed, and a staged training strategy was adopted to optimize the network end-to-end. Specifically, let the fused output image be... The original RGB image is Stokes component intensity image is First, a reconstruction loss based on pixel differences is introduced to constrain the overall consistency of brightness and color, and it is defined as:

[0074] ,

[0075] Furthermore, to improve the image contour structure and local edge sharpness, this invention introduces gradient constraint loss, which calculates the fused image and... To maintain structural continuity in the edge region amidst the gradient space deviation, the following form is given:

[0076] ,

[0077] in, This represents the gradient operator of the image in the horizontal and vertical directions.

[0078] To further ensure that the fused image closely approximates the effect of real natural imaging at the visual perception level, this invention introduces perceptual consistency loss and uses a pre-trained feature extraction network. Calculate high-level semantic bias within the feature space:

[0079] ,

[0080] Building upon this, to fully leverage the advantages of polarization-saliency regions in structural representation, this invention constructs a polarization consistency constraint loss, which transforms the polarization-saliency mask generated by the DoLP gradient into a polarization-saliency mask. As a weighting factor, regions with significant polarization responses receive higher weights in structure reconstruction, resulting in:

[0081] ,

[0082] in, express Norm.

[0083] Finally, the overall joint loss function is expressed as:

[0084] ,

[0085] in, These are non-negative weighting parameters used to balance the importance of each loss term.

[0086] Regarding the training strategy, this invention employs a two-stage stepwise optimization mechanism: In the first stage, the parameters of the cross-modal fusion module are kept fixed, and only the dual-branch encoder and decoder structures are trained. This stage uses reconstruction loss and perceptual consistency loss to construct an initial joint loss for network supervision, enabling the network to first possess stable pixel-level reconstruction capabilities and maintain fidelity in high-level semantic features, thereby forming a reasonable basic feature distribution. In the second stage, all parameters are unfrozen, and polarization consistency and gradient constraint losses are introduced based on the original optimization objective to construct a global joint loss including reconstruction, perception, polarization, and gradient terms. This global joint loss guides parameter updates under end-to-end conditions, enhancing the mapping relationship of the deep learning cross-modal structure. During training, the AdamW optimization algorithm is used for gradient backpropagation based on the joint loss of each stage, with an initial learning rate set to... Furthermore, a learning rate decay strategy is combined to improve convergence stability, while data augmentation methods such as random pruning, flipping, and illumination perturbation are used to improve the model's generalization ability in real-world environments.

[0087] Through the above-mentioned joint loss constraint and training strategy, the present invention can enhance the contribution of polarization information while maintaining the natural image appearance, so that the fusion result can show better stability and reliability in weak texture target recognition, strong reflection suppression and structural representation.

[0088] Table 1 Quantitative Analysis

[0089]

[0090] As shown in Table 1, compared with the current mainstream weak feature object detection networks, the algorithm of this invention exhibits superior performance and has the following advantages:

[0091] (1) This invention utilizes an RGB camera and a polarization camera to simultaneously acquire images of the same scene. The polarization camera integrates polarization filter arrays with different directions to form polarization intensity maps in directions such as 0°, 45°, 90°, and 135°. First, the mosaic image output by the polarization array is interpolated and reconstructed to obtain a complete pixel-level polarization direction image, and the Stokes parameters are calculated accordingly. The polarization degree DoLP and polarization angle AoLP are then obtained to construct a polarization feature map. Subsequently, the polarization feature map is spatially registered with the RGB image to achieve pixel-level or sub-pixel-level alignment.

[0092] (2) In this invention, the registered RGB image and polarization feature map are respectively input into a dual-branch feature encoding network, and a cross-level feature pyramid is constructed using a multi-scale feature extraction structure. The shallow features mainly extract texture, brightness and local detail information, while the deep features mainly extract semantic structure and global scene information, thereby forming a multi-scale modal feature expression that gradually becomes more abstract with each level.

[0093] (3) This invention adopts a hierarchical fusion strategy to address the differences in semantic attributes and expressions of features at different levels. For shallow features, a cross-modal attention mechanism is introduced to establish a correlation mapping relationship between RGB color information and polarization texture and material reflection features, thereby achieving global dependency modeling between the two modalities. For high-level features, a feature enhancement fusion structure is adopted, which strengthens the structural expressive ability through channel modeling and residual superposition, suppresses cross-modal feature conflicts, and ensures the stability of high-level semantic structure expression.

[0094] (4) In this invention, shallow fusion features and deep fusion features are jointly input into the decoding and reconstruction module. Through multi-level upsampling and convolutional reconstruction structure, the original features are restored. Figure 1 The resolution is adjusted, and a fused output image is generated, achieving a unified fusion of natural colors and stable structural representation.

[0095] Compared with existing technologies, the advantages of this invention are as follows: This multimodal fusion method based on RGB and polarized images fully considers the complementary characteristics of polarization physical information and natural image information in its structural design. Through a layered fusion strategy, shallow texture and deep semantics work synergistically during the fusion process, maintaining clear structural contours and natural color reproduction even in complex environments such as weak textures, strong reflections, and low contrasts. On the one hand, this invention avoids the problem of insufficient structural expression in single-modal algorithms; on the other hand, it enhances the physical meaning expression of optical information through polarization consistency constraints, giving the fusion results higher reliability and engineering application value. Furthermore, the method of this invention has advantages such as a reasonable model structure, controllable computational complexity, and strong versatility, meeting the deployment requirements of visual perception systems in various scenarios.

[0096] The present invention also provides a multimodal image fusion system for performing the above-described image fusion method, the system comprising:

[0097] The acquisition module is used to acquire RGB images and polarization images that are synchronously collected in the same scene;

[0098] The data preprocessing module is used to perform demosaic reconstruction, polarization parameter calculation, and spatial registration on RGB and polarized images.

[0099] The multi-scale feature encoding module is used to perform multi-scale feature encoding processing on the RGB image and the polarization feature map respectively, and to obtain the corresponding first mode feature and second mode feature by extracting features at different scales;

[0100] The hierarchical fusion module is used to fuse the first modal feature and the second modal feature using differentiated fusion strategies according to the different feature levels.

[0101] The image reconstruction module is used to decode the fused features and generate a fused output image.

[0102] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described multimodal image fusion method.

[0103] It should be emphasized that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A multimodal image fusion method based on RGB images and polarization images, characterized in that, Includes the following steps: S110, acquire RGB images and polarization images synchronously acquired in the same scene, wherein the polarization image is acquired by a polarization imaging device containing multiple filtering units with different polarization directions; S120, the polarization image is subjected to demosaic reconstruction processing to restore the complete image corresponding to each polarization direction, and polarization parameters are calculated based on the reconstructed image corresponding to each polarization direction to generate a polarization feature map containing at least one of polarization intensity feature, polarization degree feature and polarization angle feature; at the same time, the RGB image and the polarization feature map are spatially registered to achieve pixel-level or sub-pixel-level alignment. S130, perform multi-scale feature encoding processing on the RGB image and the polarization feature map respectively, and obtain corresponding first modal features and second modal features by extracting features at different scales, wherein the first modal features and the second modal features each include at least low-level texture features and high-level structure features; S140, according to the different feature levels, the first modal features and the second modal features are fused in layers, wherein: the low-level texture features are fused using a cross-modal attention interaction method to establish the association between the first modal features and the second modal features; High-level structural features are fused using feature enhancement techniques to strengthen structural information and suppress cross-modal feature conflicts; S150 uses a fusion model to decode and reconstruct the fused features, generating a fused output image.

2. The method according to claim 1, characterized in that, In S120, polarization parameters are calculated based on the reconstructed images corresponding to each polarization direction, including: calculating Stokes parameters based on the images corresponding to different polarization directions, and obtaining at least one of polarization intensity information, degree of polarization information, and polarization angle information based on the Stokes parameters.

3. The method according to claim 1, characterized in that, In S130, the multi-scale feature encoding includes using feature extraction operators with different receptive field sizes to perform multi-level feature extraction on the image, so as to simultaneously obtain the local texture features, edge features and global structural features of the image.

4. The method according to claim 1, characterized in that, In S140, the cross-modal attention interaction method includes: calculating the correlation between the first modal feature and the second modal feature based on attention weights, which is used to establish a long-range dependency between the two modal features, thereby enhancing the shallow feature consistency between color information and polarization texture information.

5. The method according to claim 1, characterized in that, In S140, the feature enhancement method includes jointly modeling the first modal feature and the second modal feature in the channel dimension and / or spatial dimension, and maintaining the stability of deep features and structural continuity in the fusion process by introducing residual connections or feature superposition methods.

6. The method according to claim 1, characterized in that, During the training of the fusion model, a polarization consistency constraint based on polarization salient regions is introduced, wherein the polarization salient regions are determined by gradient changes or saliency information in the polarization feature map.

7. The method according to claim 6, characterized in that, The polarization consistency constraint design is as follows: ， in, It is a polarization-significant mask generated by the polarization degree gradient. Represents the gradient operators of an image in the horizontal and vertical directions. To fuse the output images, For Stokes component intensity images, express Norm.

8. The method according to claim 1, characterized in that, The network structure of the fusion model includes a dual-branch coding framework: an RGB coding branch and a polarization coding branch, which receive RGB images and polarization feature images as inputs, respectively; both branches adopt multi-scale convolutional structures to extract texture and semantic features from shallow to deep, with convolutional kernel sizes of 3×3, 5×5, 7×7 and 11×11, and ReLU and batch normalization operations are introduced after convolution; The fusion model also includes a dual fusion module, consisting of a shallow Transformer fusion module and a deep residual fusion module. The shallow Transformer fusion module uses a lightweight Transformer structure and establishes a global dependency mapping relationship between the low-level texture features of the RGB image and the low-level texture features of the polarization feature map through a multi-head self-attention mechanism. The deep residual fusion module uses an improved residual dense convolution structure to concatenate the high-level structural features of the first and second modal features in the channel dimension, and enhances the semantic structure consistency through depthwise separable convolution. At the same time, residual feedback is introduced to suppress gradient decay, thereby ensuring the stability and robustness of the fusion process. The fused features obtained after the above dual-branch encoding process are fed into the hierarchical decoder one by one. The original image resolution is restored through step-by-step upsampling and deconvolution operations, and finally a three-channel fused image is output after 1×1 convolution.

9. A multimodal image fusion system based on RGB images and polarization images, used to execute the multimodal image fusion method based on RGB images and polarization images according to any one of claims 1-8, characterized in that, include: The acquisition module is used to acquire RGB images and polarization images that are synchronously collected in the same scene; The data preprocessing module is used to perform demosaic reconstruction, polarization parameter calculation, and spatial registration on RGB and polarized images. The multi-scale feature encoding module is used to perform multi-scale feature encoding processing on the RGB image and the polarization feature map respectively, and to obtain the corresponding first mode feature and second mode feature by extracting features at different scales; The hierarchical fusion module is used to fuse the first modal feature and the second modal feature using differentiated fusion strategies according to the different feature levels. The image reconstruction module is used to decode the fused features and generate a fused output image.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multimodal image fusion method based on RGB images and polarization images as described in any one of claims 1 to 8.

Images