Image fusion method and device based on constraint alignment, equipment and storage medium

Abstract

The application discloses a constraint alignment-based image fusion method and device, equipment and a storage medium. The method comprises the following steps: performing perceptual correlation feature extraction on a target visible light image and a target near-infrared light image based on an SKNet to obtain a visible light feature map and a near-infrared light feature map; mapping the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on a modal difference constraint through an automatic encoder to obtain a visible light modal map and a near-infrared light modal map; aligning the visible light modal map and the near-infrared light modal map according to a structural similarity loss function in the low-dimensional latent feature space to obtain a visible light alignment map and a near-infrared light alignment map; and fusing the visible light alignment map and the near-infrared light alignment map to obtain a target fusion feature map. The application can solve the technical problem that the existing image fusion method is too simple, the difference between different modal images is easily neglected, image information may be lost, and the image fusion effect is poor.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, apparatus, device and storage medium based on constraint alignment. Background Art

[0002] Image feature fusion can generate more accurate, comprehensive, and high-quality image information. Image fusion technology involves integrating image information from multiple sources using specific algorithms to generate a new image. This technology can overcome the limitations of image information obtained by a single sensor, providing richer details and higher image quality.

[0003] Existing binocular image feature fusion methods primarily rely on direct channel concat or direct pixel addition. Both of these feature fusion methods ignore the differences in feature spatial distribution between images of different modalities. This can lead to loss of feature information from one modality during feature interaction, or cause the fused features to favor a single modality, resulting in poor feature fusion performance. Summary of the Invention

[0004] The present application provides an image fusion method, apparatus, device and storage medium based on constraint alignment, which is used to solve the technical problems that existing image fusion methods are too simple, easily ignore the differences between images of different modalities, may cause loss of image information, and have poor image fusion effects.

[0005] In view of this, the first aspect of the present application provides an image fusion method based on constrained alignment, comprising:

[0006] Based on SKNet, perceptual correlation features of the target visible light image and the target near-infrared light image are extracted to obtain visible light feature maps and near-infrared light feature maps;

[0007] Mapping the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on modality difference constraints through an autoencoder to obtain a visible light modal map and a near-infrared light modal map;

[0008] In the low-dimensional latent feature space, the visible light modality map and the near-infrared light modality map are aligned and fused according to a structural similarity loss function to obtain a target fused feature map.

[0009] Preferably, the perceptual correlation feature extraction of the target visible light image and the target near-infrared light image based on SKNet to obtain the visible light feature map and the near-infrared light feature map includes:

[0010] Extracting a visible light feature map from a target visible light image, and using SKNet to extract channel-aware attention of the target visible light image;

[0011] Feature extraction is performed on the target near-infrared light image based on the channel perception attention to obtain a near-infrared light feature map.

[0012] Preferably, the mapping of the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on modality difference constraints by an autoencoder to obtain a visible light modal map and a near-infrared light modal map comprises:

[0013] Mapping the visible light feature map and the near-infrared feature map to a low-dimensional latent feature space using an autoencoder to generate a visible light mapping map and a near-infrared mapping map;

[0014] Modal constraint processing is performed on the visible light mapping image and the near-infrared mapping image through modal difference constraint to obtain a visible light modal image and a near-infrared light modal image.

[0015] Preferably, in the low-dimensional latent feature space, the visible light modality map and the near-infrared light modality map are aligned and fused according to a structural similarity loss function to obtain a target fused feature map, and then the method further includes:

[0016] The target fusion feature map is input into a preset living body detector for category analysis to obtain a living body category detection result.

[0017] A second aspect of the present application provides an image fusion device based on constraint alignment, comprising:

[0018] A feature extraction unit is used to extract perceptual correlation features from the target visible light image and the target near-infrared light image based on SKNet to obtain a visible light feature map and a near-infrared light feature map;

[0019] a constraint mapping unit, configured to map the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on a modality difference constraint through an autoencoder to obtain a visible light modal map and a near-infrared light modal map;

[0020] The alignment and fusion unit is used to align and fuse the visible light modal image and the near-infrared light modal image in the low-dimensional latent feature space according to a structural similarity loss function to obtain a target fused feature image.

[0021] Preferably, the feature extraction unit is specifically used to:

[0022] Extracting a visible light feature map from a target visible light image, and using SKNet to extract channel-aware attention of the target visible light image;

[0023] Feature extraction is performed on the target near-infrared light image based on the channel perception attention to obtain a near-infrared light feature map.

[0024] Preferably, the constraint mapping unit is specifically configured to:

[0025] mapping the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space by using an auto-encoder to generate a visible light mapping map and a near-infrared mapping map;

[0026] performing modal constraint processing on the visible light mapping map and the near-infrared mapping map by using a modal difference constraint to obtain a visible light modal map and a near-infrared light modal map.

[0027] Preferably, the method further comprises:

[0028] a classification detection unit configured to input the target fusion feature map into a preset living body detector to perform category analysis and obtain a living body category detection result.

[0029] The third aspect of the present application provides an image fusion device based on constraint alignment, the device comprising a processor and a memory;

[0030] The memory is configured to store program code and transmit the program code to the processor;

[0031] The processor is configured to execute the image fusion method based on constraint alignment according to the instructions in the program code.

[0032] The fourth aspect of the present application provides a computer readable storage medium for storing program code, the program code being used to execute the image fusion method based on constraint alignment.

[0033] As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

[0034] In the present application, an image fusion method based on constraint alignment is provided, comprising: performing perceptual correlation feature extraction on a target visible light image and a target near-infrared light image based on an SKNet to obtain a visible light feature map and a near-infrared light feature map; mapping the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on a modal difference constraint by using an auto-encoder to obtain a visible light modal map and a near-infrared light modal map; and performing alignment fusion on the visible light modal map and the near-infrared light modal map according to a structural similarity loss function in the low-dimensional latent feature space to obtain a target fusion feature map.

[0035] The image fusion method based on constraint alignment provided by the present application extracts features from images of different modalities, maps them to a low-dimensional latent feature space through an autonomous encoder according to the constraints based on modality differences, and then performs constraint-based alignment processing, so that images of different modalities can have similar feature space distributions in the same space. Taking into account the feature differences of images of different modalities, it can also avoid information loss to a certain extent, so as to ensure the effect of image fusion. Therefore, the present application can solve the technical problems that the existing image fusion method is too simple, easily ignores the differences between images of different modalities, may cause image information loss, and have poor image fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of an image fusion method based on constrained alignment provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the structure of an image fusion device based on constraint alignment provided in an embodiment of the present application;

[0038] Figure 3 Schematic diagram of the living body classification detection process based on constrained alignment image fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0040] For easier understanding, see Figure 1 , an embodiment of the image fusion method based on constrained alignment provided by the present application includes:

[0041] Step 101: Extract perceptual correlation features of the target visible light image and the target near-infrared light image based on SKNet to obtain a visible light feature map and a near-infrared light feature map.

[0042] Furthermore, step 101 includes:

[0043] Extract the visible light feature map from the target visible light image, and use SKNet to extract the channel-aware attention of the target visible light image;

[0044] The target near-infrared light image is feature extracted based on channel-aware attention to obtain a near-infrared light feature map.

[0045] The dual-mode images selected in this embodiment are visible light images and near-infrared light images , which are expressed as in the feature extraction stage, two branches are processed; Specifically, the Backbone network structure is used as the feature extraction module; Corresponding feature maps are extracted from the visible light image and the near-infrared light image. However, in order to improve the feature distribution similarity of the dual-mode image, the channel attention correlation is performed in the feature extraction process.

[0046] Specifically, the features of the target visible light image are first extracted to obtain the visible light feature map ; and in this process, the channel perception attention of the target visible light image is extracted through SKNet , and the channel perception attention is used to guide and supervise the extraction of the near-infrared light feature map , so that the two feature maps have similar feature attention. The guiding and supervising method is to multiply the channel perception attention when the near-infrared light feature extraction output is output, so as to generate the near-infrared light feature map ; In this way, it can be ensured that the extraction process of the near-infrared light feature map is guided and supervised by the visible light feature extraction.

[0047] SKNet designs a building block named Selective Kernel (SK) unit, in which multiple branches with different kernel sizes are fused using softmax attention guided by information from these branches. Different attention of these branches results in different sizes of effective receptive fields of neurons in the fusion layer, allowing each neuron to adaptively adjust its receptive field size according to multiple scales of input information.

[0048] Step 102, map the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on a modal difference constraint to obtain a visible light modal image and a near-infrared light modal image.

[0049] Further, step 102 comprises:

[0050] The visible light feature map and the near-infrared light feature map are mapped to a low-dimensional latent feature space by using an autoencoder to generate a visible light mapping image and a near-infrared mapping image;

[0051] The visible light mapping image and the near-infrared mapping image are processed by modal constraint processing through a modal difference constraint to obtain a visible light modal image and a near-infrared light modal image.

[0052] It should be noted that the autoencoder is an unsupervised learning neural network model that aims to reconstruct the input data by learning the latent feature representation of the input data. It mainly consists of two parts: an encoder and a decoder. The encoder is responsible for compressing the input data into a low-dimensional latent space, also known as encoding, while the decoder attempts to reconstruct the original input based on this low-dimensional representation. By training the autoencoder to minimize the difference between the input data and the reconstructed data, it can learn the intrinsic structure and latent features of the data. The core idea of ​​the autoencoder is to encode the input data into a low-dimensional representation through an encoder, and then decode it back to the original data space through a decoder. Because the autoencoder can learn the nonlinear latent structure of the input data, it can capture richer information in the data.

[0053] Although the visible light feature map is supervised by channel-aware attention guidance and near-infrared light characteristics The two modal images have similar feature attention, but there are channel and style feature differences between the features of different modal images. To reduce the differences between cross-modal image features, this embodiment uses an autoencoder to map the feature map from a high-dimensional space to a low-dimensional latent feature space. In order to make the feature maps of the two modalities have similar feature space distributions after the mapping operation, this embodiment needs to constrain them through modal difference constraints to obtain visible light modal maps and near-infrared light modal maps. The function expression of the modal difference constraint is:

[0054]

[0055] Where N represents the number of feature map pairs corresponding to visible light and near-infrared light, i represents the image pair index, 、 They represent the mapped visible light modal image and near-infrared light modal image respectively. This constraint can bring the feature distributions of the two modal images closer in the current feature space.

[0056] Step 103: In the low-dimensional latent feature space, the visible light modal image and the near-infrared light modal image are aligned and fused according to the structural similarity loss function to obtain a target fused feature image.

[0057] If the visible light modal graph with similar feature space distribution is and near-infrared modal images Alignment fusion can be expressed as ; In this process, cross-modal supervision is also required. Specifically, it is to extract the second half of the fusion feature Original visible light feature map Perform supervised training; similarly, extract the first half of the fusion feature Original near-infrared light feature map Perform supervised training. Based on this supervised training principle, the structural similarity loss function can be obtained:

[0058]

[0059] in, Represents the back propagation process of the current feature. It can be seen that the structural similarity loss function combines the two inverse modal supervision training losses. By minimizing this loss function, cross-modal feature interaction alignment can be further achieved, ensuring the accuracy and reliability of feature map fusion.

[0060] If the above modal difference constraint function is combined with the structural similarity loss function here, the optimization loss function of the entire system can be obtained, which can be expressed as:

[0061]

[0062] in, 、 These are all loss hyperparameters that can be adjusted during training. The aligned fusion feature map That is the target fusion feature map.

[0063] Furthermore, step 103 further includes:

[0064] The target fusion feature map is input into the preset living body detector for category analysis to obtain the living body category detection result.

[0065] The image fusion method based on constraint alignment provided in this embodiment can be applied to different target detection or recognition tasks, and can extract feature information of different image modalities to the greatest extent, helping to improve detection accuracy and reliability. Taking the task of liveness classification detection as an example, the target fusion feature map obtained can be input into the subsequent preset liveness detector for classification detection and obtain the liveness category detection result; the preset liveness detector can be designed and constructed according to actual conditions and is not limited here. For the overall feature fusion detection process, please refer to Figure 3 , the detector can be a fully connected layer, and the activation output class prediction probability.

[0066] The image fusion method based on constraint alignment provided in the embodiment of the present application extracts features from images of different modalities, maps them to a low-dimensional latent feature space through an autonomous encoder according to the constraints based on modality differences, and then performs constraint-based alignment processing, so that images of different modalities can have similar feature space distributions in the same space. Taking into account the feature differences of images of different modalities, it can also avoid information loss to a certain extent, so as to ensure the effect of image fusion. Therefore, the embodiment of the present application can solve the technical problems that the existing image fusion method is too simple, easily ignores the differences between images of different modalities, may cause image information loss, and have poor image fusion effect.

[0067] For easier understanding, see Figure 2 , the present application provides an embodiment of an image fusion device based on constraint alignment, comprising:

[0068] A feature extraction unit 201 is configured to extract perceptual correlation features from a target visible light image and a target near-infrared light image based on SKNet to obtain a visible light feature map and a near-infrared light feature map;

[0069] A constraint mapping unit 202 is configured to map the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on modality difference constraints using an autoencoder to obtain a visible light modal map and a near-infrared light modal map;

[0070] The alignment and fusion unit 203 is used to align and fuse the visible light modal image and the near-infrared light modal image in the low-dimensional latent feature space according to the structural similarity loss function to obtain a target fused feature image.

[0071] Furthermore, the feature extraction unit 201 is specifically configured to:

[0072] Extract the visible light feature map from the target visible light image, and use SKNet to extract the channel-aware attention of the target visible light image;

[0073] The target near-infrared light image is feature extracted based on channel-aware attention to obtain a near-infrared light feature map.

[0074] Furthermore, the constraint mapping unit 202 is specifically configured to:

[0075] An automatic encoder is used to map the visible light feature map and the near-infrared feature map into a low-dimensional latent feature space to generate a visible light map and a near-infrared map;

[0076] Modal constraint processing is performed on the visible light mapping image and the near-infrared mapping image through modal difference constraint to obtain the visible light modal image and the near-infrared light modal image.

[0077] Furthermore, it also includes:

[0078] The classification detection unit 204 is used to input the target fusion feature map into a preset living body detector for category analysis to obtain a living body category detection result.

[0079] The present application also provides an image fusion device based on constraint alignment, the device including a processor and a memory;

[0080] The memory is used to store program codes and transmit the program codes to the processor;

[0081] The processor is configured to execute the image fusion method based on constraint alignment in the above method embodiment according to instructions in the program code.

[0082] The present application also provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the image fusion method based on constraint alignment in the above method embodiment.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0084] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0085] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0086] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, includes a plurality of instructions for executing all or part of the steps of the method described in various embodiments of the present application by a computer device (which can be a personal computer, a server, or a network device, etc.). The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0087] The above-described and the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. Image fusion method based on constraint alignment, characterized by: include: Based on SKNet, the perceptual correlation features of the target visible light image and the target near-infrared light image are extracted to obtain the visible light feature map and the near-infrared light feature map. The specific process is as follows: Extracting a visible light feature map from a target visible light image, and using SKNet to extract channel-aware attention of the target visible light image; Extracting features of the target near-infrared light image based on the channel perception attention to obtain a near-infrared light feature map; Mapping the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on a modal difference constraint using an autoencoder to obtain a visible light modal map and a near-infrared light modal map, wherein the modal difference constraint is expressed by a modal difference loss function; In the low-dimensional latent feature space, the visible light modal image and the near-infrared light modal image are aligned and fused according to a structural similarity loss function to obtain a target fused feature image. The specific process is as follows: If the visible light modal graph with similar feature space distribution is and near-infrared modal images Perform alignment fusion, expressed as ,In the alignment and fusion process, it also includes: Extract the second half of the fusion feature Original visible light feature map Perform supervised training; similarly, extract the first half of the fusion feature Original near-infrared light feature map Conduct supervised training.

2. The image fusion method based on constraint alignment according to claim 1, characterized in that: The method maps the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on modality difference constraints by an autoencoder to obtain a visible light modal map and a near-infrared light modal map, including: Mapping the visible light feature map and the near-infrared feature map to a low-dimensional latent feature space using an autoencoder to generate a visible light mapping map and a near-infrared mapping map; Modal constraint processing is performed on the visible light mapping image and the near-infrared mapping image through modal difference constraint to obtain a visible light modal image and a near-infrared light modal image.

3. The image fusion method based on constraint alignment according to claim 1, characterized in that: In the low-dimensional latent feature space, the visible light modal image and the near-infrared light modal image are aligned and fused according to a structural similarity loss function to obtain a target fused feature image, and then the method further includes: The target fusion feature map is input into a preset living body detector for category analysis to obtain a living body category detection result.

4. An image fusion device based on constraint alignment, characterized in that: include: A feature extraction unit is used to extract perceptual correlation features from the target visible light image and the target near-infrared light image based on SKNet to obtain a visible light feature map and a near-infrared light feature map. The feature extraction unit is specifically used to: Extracting a visible light feature map from a target visible light image, and using SKNet to extract channel-aware attention of the target visible light image; Extracting features of the target near-infrared light image based on the channel perception attention to obtain a near-infrared light feature map; a constraint mapping unit, configured to map the visible light feature map and the near-infrared light feature map to a low-dimensional latent feature space based on a modal difference constraint using an autoencoder to obtain a visible light modal map and a near-infrared light modal map, wherein the modal difference constraint is expressed by a modal difference loss function; The alignment and fusion unit is used to align and fuse the visible light modal image and the near-infrared light modal image in the low-dimensional latent feature space according to the structural similarity loss function to obtain a target fused feature image. The specific process is as follows: If the visible light modal graph with similar feature space distribution is and near-infrared modal images Perform alignment fusion, expressed as ,In the alignment and fusion process, it also includes: Extract the second half of the fusion feature Original visible light feature map Perform supervised training; similarly, extract the first half of the fusion feature Original near-infrared light feature map Conduct supervised training.

5. The image fusion device based on constraint alignment according to claim 4, characterized in that: The constraint mapping unit is specifically used to: Mapping the visible light feature map and the near-infrared feature map to a low-dimensional latent feature space using an autoencoder to generate a visible light mapping map and a near-infrared mapping map; Modal constraint processing is performed on the visible light mapping image and the near-infrared mapping image through modal difference constraint to obtain a visible light modal image and a near-infrared light modal image.

6. The image fusion device based on constraint alignment according to claim 4, characterized in that: Also includes: The classification detection unit is used to input the target fusion feature map into a preset living body detector for category analysis to obtain a living body category detection result.

7. An image fusion device based on constraint alignment, characterized in that: The device includes a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the image fusion method based on constraint alignment according to any one of claims 1 to 3 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the image fusion method based on constraint alignment according to any one of claims 1 to 3.

Images