Infrared-Visible Feature Separation and Enhancement Method Based on Laplace Consensus Matrix

Abstract

This invention discloses an infrared-visible light feature separation and enhancement method based on a Laplacian consensus matrix to address the problem of weakened discriminative features after fusion due to the weak coupling of semantic information of different modal features under modality failure conditions. The method includes: First, feature extraction, normalization, concatenation along the channel dimension, dimensionality reduction fusion, and graph Laplacian embedding are performed on visible light and infrared images to obtain a consensus Laplacian matrix representing the activated discriminative features common to both modalities. Second, based on the consensus Laplacian matrix, modality-specific attention weight matrices are extracted from the visible light and infrared images, and discriminative feature enhancement is performed to obtain enhanced visible light and infrared image features. Finally, the enhanced visible light and infrared image features are concatenated along the channel dimension, and depthwise separable convolution is used to extract and fuse global information from the concatenated image features to obtain fused image features, which are then input into the decoder to obtain the final prediction result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image information fusion technology based on deep learning, and particularly to an infrared-visible light feature separation and enhancement method based on the Laplacian consensus matrix. Background Technology

[0002] Multimodal cognitive computing theories and methods have played a significant role in overcoming the research challenges of multi-sensor image information fusion technology. Because visible light images are limited by their spectral range, they cannot guarantee the robustness of detection systems in practical environments such as darkness, rain, and fog. Infrared sensors, on the other hand, capture the thermal radiation of objects to form images, unaffected by ambient lighting conditions. Therefore, researchers have begun to investigate how to jointly utilize the complementary information between visible light and infrared images to enhance the environmental perception capabilities of detection systems under harsh lighting conditions.

[0003] Currently, progressive fusion is the main method for fusing image features from different modalities. It uses complex deep learning network structures to progressively fuse image features from different modalities. However, the complex multimodal feature fusion modules in progressive fusion lack interpretability. On the one hand, it does not consider the weak coupling of semantic information between different modalities in the case of modality failure, which may weaken modality-specific discriminative features during feature fusion. On the other hand, while fusing complementary information between modalities, it introduces too much redundant noise information, reducing the robustness of the perception system.

[0004] Therefore, how to achieve complementary information fusion between different modal features under modal failure conditions, and provide more stable and reliable discriminative feature information for subsequent downstream detection tasks, is an urgent problem to be solved in the field of image information fusion. Summary of the Invention

[0005] To overcome the above-mentioned technical defects, this invention provides an infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix to solve the problem that the discriminative features after fusion are weakened due to the non-strong coupling of semantic information of different modal features under modal failure conditions.

[0006] In a first aspect, embodiments of the present invention provide an infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix, specifically including the following steps:

[0007] S4. Perform graph Laplacian embedding on the fused spatial dimension weight matrix to obtain the consensus Laplacian matrix of the visible light image and the infrared image. It represents the common, activated discriminative features of the two modes. The fused spatial dimension weight matrix is ​​obtained by feature extraction, global pooling operation, concatenation along the channel dimension and dimensionality reduction fusion of the visible light matrix and the infrared image matrix.

[0008] S5. Based on the consensus Laplacian matrix, the spatial dimension weight matrices extracted from the visible light image and the infrared image are multiplied element-wise, and then normalized to obtain the attention matrices of visible light and infrared features compared with homogeneous features, namely the visible light attention matrix and the infrared attention matrix.

[0009] S6. Invert the visible light attention matrix and the infrared attention matrix to obtain the mode-specific attention weight matrices in the visible light image and the infrared image, namely the visible light attention weight matrix and the infrared attention matrix.

[0010] S7. Discriminative feature enhancement is performed on visible light image features and infrared image features using attention matrix and attention weight matrix to obtain enhanced visible light image features and infrared image features;

[0011] S8. The enhanced visible light image features and infrared image features are concatenated along the channel dimension, and depthwise separable convolution is used to extract and fuse global information of the concatenated image features to obtain the fused image features.

[0012] Optionally, in one possible implementation of the first aspect,

[0013] Prior to S4, the specific steps for obtaining the fused spatial dimension weight matrix included:

[0014] S1. Input the visible light image and the infrared image into the feature extractor respectively, and extract the features of the visible light image and the infrared image;

[0015] S2. Extract the discriminant features for each mode from the visible light image features and infrared image features through global pooling operation, and normalize the discriminant features of each mode as the mode-specific spatial dimension weight matrix.

[0016] S3. First, the spatial dimension weight matrices extracted from the visible light image and the infrared image are concatenated along the channel dimension. Then, the convolution operation is used to reduce the dimension of the concatenated weight matrix and fuse it to obtain the fused spatial dimension weight matrix.

[0017] Optionally, in one possible implementation of the first aspect, the specific steps of the above-described graph Laplacian embedding include:

[0018] First, the fused spatial dimension weight matrix V is calculated using the K-nearest neighbor algorithm. fusion First, calculate the adjacency matrix A; second, calculate the spatial dimension weight matrix V. fusion degree matrix Where DA is the spatial dimension weight matrix V. fusionThe consensus Laplace matrix is ​​calculated as follows:

[0019] M cons =D -1 / 2 (DA)D -1 / 2 ,in,

[0020] Optionally, in one possible implementation of the first aspect, the calculation method for the above-mentioned discriminative feature enhancement is as follows:

[0021]

[0022] Among them, t rgb With t th These are the attention enhancement factors for the visible light mode and the infrared mode, respectively. Their initial values ​​are both set to 1 and are continuously updated during backpropagation of the deep learning network. Multiply by the channel dimension.

[0023] Optionally, in one possible implementation of the first aspect, the specific calculation method for the extraction and fusion of the aforementioned global information is as follows:

[0024]

[0025] Among them, f ReLU (·) is the ReLU activation function, f dw (·) is a depthwise separable convolution operation, and cat(·) is a vector concatenation operation along the channel dimension.

[0026] Secondly, another aspect of the present invention provides a computer device, comprising:

[0027] At least one processor; and

[0028] A memory that is communicatively connected to the at least one processor;

[0029] Wherein: the memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0030] Thirdly, another aspect of the present invention provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the method described above.

[0031] Fourthly, another aspect of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the method described above.

[0032] The above-mentioned technical solution adopted in the embodiments of the present invention can achieve the following technical effects: by constructing a consensus Laplacian matrix of multimodal image features, such as visible light mode and infrared mode, homogeneous redundant information and heterogeneous complementary information in different modes can be separated, thereby achieving effective separation of foreground target information and background noise information, and improving the accuracy and reliability of the perception system in non-ideal environments in real-world scenarios. Attached Figure Description

[0033] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0034] Figure 1 This is a flowchart illustrating the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix in an embodiment of this application.

[0035] Figure 2 This is a schematic diagram comparing the feature intensity visualization heatmaps of the method in this application embodiment with other methods;

[0036] Figure 3 This is a schematic diagram comparing the qualitative experiments of the method in this application embodiment with other methods under different environments;

[0037] Figure 4 This is a schematic diagram of the hardware structure of a computer device in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0039] It should be noted that the descriptions involving "first," "second," etc., in the embodiments of the present invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0040] In the description of this invention, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are performed, but are only used to facilitate the description of this invention and to distinguish each step, and therefore should not be construed as a limitation of this invention.

[0041] To facilitate understanding of the technical solutions in this invention, some technical terms used in this field will be briefly introduced below:

[0042] Modality: Modality refers to different types of information input or output forms in a system, device, or sensing process. The modalities applicable to this solution are not limited to image modalities such as visible light and infrared, but also include video, sound, and text modalities.

[0043] Modal failure: Modal failure refers to a problem in the transmission or processing of information in one or more modes within a system, device, or sensing process, resulting in the system's inability to correctly understand or respond. The system needs to have mechanisms to cope with such failures in order to continue operating normally.

[0044] Discriminative features: Discriminative features refer to the features in a perception system that can distinguish different categories of data to the greatest extent.

[0045] Foreground object information: In image or video processing tasks, foreground object information refers to the main objects or regions of interest. These objects are typically distinguishable from the background and contain the key information required for the task.

[0046] Background redundancy or noise information: Background redundancy or noise information refers to unnecessary or irrelevant parts of an image, video, or signal processing system. This information often interferes with the analysis or decision-making of the perception system.

[0047] Homogeneous redundancy across different modalities: Homogeneous redundancy across different modalities refers to the fact that different modalities in a multimodal perception system express the same or similar information, which reduces the perception system's ability to extract key information.

[0048] Heterogeneous complementary information across different modalities: Heterogeneous complementary information across different modalities refers to the fact that information from different modalities can complement each other, providing unique discriminative information in different dimensions. This helps the model to consider more information when facing complex decisions, thereby making a more informed choice.

[0049] ResNet algorithm: The ResNet algorithm refers to a deep learning neural network architecture that is often used for feature extraction and embedding of data such as images and text.

[0050] Example 1

[0051] Next, the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix of the present invention will be described in detail with reference to the accompanying drawings and specific embodiments, as follows:

[0052] like Figure 1 As shown, the specific steps of the method described in this invention include:

[0053] S1. Input the visible light image and the infrared image into the feature extractor respectively, and extract the features of the visible light image and the infrared image.

[0054] It should be noted that the technical solution in this application is applicable not only to image modalities such as visible light and infrared, but also to feature separation and enhancement of modalities such as video, audio, and text. This application only uses visible light images and infrared images as examples for illustration.

[0055] Specifically, the feature extractor is constructed using a backbone network based on the ResNet algorithm. The aforementioned visible light image features are denoted as x. RGB Infrared image features are denoted as x T Therefore, the above-mentioned inputting infrared and visible light images into the feature extractor can be specifically represented as follows:

[0056] x rgb =Resnet(X rgb ), and, x th =Resnet(X th );

[0057] Among them, X rgb and X th These represent visible light and infrared images, respectively. The visible light and infrared images have the same resolution, for example, both are 480*640. The feature embedding dimensions of the visible light and infrared image features are also the same, for example, x... RGB With x th The feature embedding dimension is set to 256.

[0058] S2. Extract the discriminant features for each mode from the visible light image features and infrared image features through global pooling operation, and normalize the discriminant features of each mode as the mode-specific spatial dimension weight matrix.

[0059] Specifically, in the visible light image, the feature is denoted as x. RGB The mode-specific spatial dimension weight matrix obtained by performing globalization and normalization operations on visible light image features in S2 is denoted as V. rgb Similarly, in visible light image features, denoted as x th The mode-specific spatial dimension weight matrix obtained by performing globalization and normalization operations on visible light image features in S2 is denoted as V.th .

[0060] Furthermore, V rgb For: V rgb =σ(f MLP (f GMP (x rgb )));V th For: V th =σ(f MLP (f GMP (x th ))), where X rgb and X th Let these represent the visible light image and the infrared image, respectively, where σ(·) is the sigmoid activation function, and f MLP (·) represents a multi-level linear mapping, f GMP (·) is the global max pooling function used to perform global pooling operations.

[0061] S3. First, the spatial dimension weight matrices extracted from the visible light image and the infrared image are concatenated along the channel dimension. Then, the convolution operation is used to reduce the dimension of the concatenated weight matrix and fuse it to obtain the fused spatial dimension weight matrix.

[0062] Specifically, the channel dimension mentioned above refers to the number of eigenvectors in the spatial dimension matrix; the fused spatial dimension weight matrix is ​​denoted as V. fusion Specifically: V fusion =Conv 1×1 (cat(V rgb V th ));

[0063] Where cat(·) is the matrix concatenation operation along the channel dimension, Conv 1×1 (·) represents a convolutional dimensionality reduction operation with a kernel size of 1. V fusion The dimensions of the weight matrix are the same as the feature embedding dimensions of the visible light image features, for example, 256.

[0064] Furthermore, matrix concatenation along the channel dimension can preserve all feature information of the weight matrix; matrix concatenation along the channel dimension can compress the number of channel dimensions and retain key feature information, reducing computational complexity.

[0065] Matrix concatenation along the channel dimension can preserve all feature information of the weight matrix; matrix concatenation along the channel dimension can compress the number of channel dimensions and retain key feature information, reducing computational complexity.

[0066] S4. Perform graph Laplacian embedding on the fused spatial dimension weight matrix to obtain the consensus Laplacian matrix of the visible light image and the infrared image, which represents the common, activated discriminative features of the two modes.

[0067] Specifically, the aforementioned consensus Laplace matrix is ​​denoted as M. cons Graph Laplacian embedding refers to capturing the local geometric structure of data based on the Laplacian operator in graph theory, thereby preserving the similarity relationships between data points in a low-dimensional space.

[0068] Optionally, in some implementations, the specific steps of the above-described graph Laplacian embedding include:

[0069] First, the fused spatial dimension weight matrix V is calculated using the K-nearest neighbor algorithm. fusion First, calculate the adjacency matrix A; second, calculate the spatial dimension weight matrix V. fusion degree matrix Where DA is the spatial dimension weight matrix V. fusion The consensus Laplace matrix is ​​calculated as follows:

[0070] M cons =D -1 / 2 (DA)D -1 / 2 ,in,

[0071] S5. Using the consensus Laplacian matrix as a benchmark, the spatial dimension weight matrices extracted from the visible light image and the infrared image are multiplied element-wise, and then normalized to obtain the attention matrices of visible light and infrared features compared to homogeneous features, namely the visible light attention matrix and the infrared attention matrix.

[0072] Specifically, the attention matrices for infrared and visible light features compared to homogeneous features (i.e., the visible light attention matrix and the infrared attention matrix) are denoted as M, respectively. rgb and M th Specifically, they are represented as: M rgb =Softmax(M cons *V rgb ) and M th =Softmax(M cons *V th );

[0073] Where Softmax(·) is the Softmax normalization function, and the visible light attention matrix M rgb With infrared attention matrix M th The parts with the smallest weights represent unique non-consistent information in the features of visible light images and infrared images, respectively.

[0074] S6. Invert the visible light attention matrix and the infrared attention matrix to obtain the mode-specific attention weight matrices in the visible light image and the infrared image, namely the visible light attention weight matrix and the infrared attention matrix.

[0075] Specifically, the visible light attention weight matrix and the infrared attention matrix are denoted as follows: and Specifically, they are: and

[0076] S7. Discriminative feature enhancement is performed on visible light image features and infrared image features using attention matrix and attention weight matrix to obtain enhanced visible light image features and infrared image features.

[0077] Specifically, the enhanced visible light image features are denoted as... The enhanced infrared image features are denoted as Optionally, in some possible implementations, the calculation method for the above-mentioned discriminative feature enhancement is as follows:

[0078]

[0079] Among them, t rgb With t th These are the attention enhancement factors for the visible light mode and the infrared mode, respectively. Their initial values ​​are both set to 1 and are continuously updated during backpropagation of the deep learning network. Multiply by the channel dimension.

[0080] S8. The enhanced visible light image features and infrared image features are concatenated along the channel dimension, and depthwise separable convolution is used to extract and fuse global information of the concatenated image features to obtain the fused image features.

[0081] Specifically, the fused image features are denoted as Optionally, in some implementations, the specific calculation method for the extraction and fusion of the above-mentioned global information is as follows:

[0082]

[0083] Among them, f ReLU (·) is the ReLU activation function, f dw (·) is a depthwise separable convolution operation, and cat(·) is a vector concatenation operation along the channel dimension.

[0084] S9. Input the fused image features into the decoder to obtain the final prediction result.

[0085] In this embodiment of the invention, a consensus Laplacian matrix of multimodal image features is constructed, such as visible light mode and infrared mode, thereby separating homogeneous redundant information in different modes, M. rgb With M th and heterogeneous complementary information, and This achieves effective separation of foreground target information and background noise information, improving the accuracy and reliability of the perception system in non-ideal environments in real-world scenarios.

[0086] Based on the above technical solution, the following experiments were conducted. The dataset used in the experiments was the open-source dataset MFNet. The MFNet dataset contains 1569 pairs of visible light and infrared image pairs (820 pairs were collected during the day and 749 pairs were collected at night), and is labeled with 9 categories including pedestrians and vehicles. The algorithm network of this invention is implemented based on the PyTorch framework, using AdamW as the model training optimizer. The initial learning rate is set to 0.0001, and the learning rate is reduced by 10 every 3 rounds. -1 The training duration was 12 rounds. Considering the small number of training samples in the visible light and infrared image datasets, this invention employs random cropping and random horizontal flipping as data augmentation strategies during the training process. Figure 2 and Figure 3 The above experiments are compared with other methods in terms of the performance of fusing feature matrix heatmaps and semantic segmentation tasks.

[0087] Figure 2 The presentation demonstrates a visual comparison of the fusion feature matrix heatmaps of the present invention with those of other methods. (a) is a visible light image, (b) is an infrared image, (c) is a visible light feature matrix heatmap, (d) is an infrared feature matrix heatmap, (e) is a FEANet fusion feature matrix heatmap, (f) is a GMNet fusion feature matrix heatmap, and (g) is the fusion feature matrix heatmap of the present invention. The highlighted areas in the heatmaps represent the areas focused by the sensing system. Under conditions such as daytime, glare, and low light, compared to other methods, the fusion feature matrix heatmap of the present invention focuses more intently on the foreground target, exhibiting reliable and accurate sensing capabilities.

[0088] Figure 3The paper presents a qualitative experimental comparison of the method of this invention with other methods on semantic segmentation tasks. (a) is a visible light image, (b) is an infrared image, (c) is the semantic segmentation result of CMNeXt, (d) is the semantic segmentation result of CMX, (e) is the semantic segmentation result of EAEFNet, (f) is the semantic segmentation result of SpiderMesh, (g) is the semantic segmentation result of this invention, and (h) is the ground truth semantic segmentation value. Based on the ground truth semantic segmentation value in (h), the completeness and accuracy of this invention for different segmented targets in different scenarios are higher than other comparative methods. For example, bicycles under glare interference and triangular cones in low-light environments.

[0089] Example 2

[0090] Figure 4 This illustration schematically depicts the hardware architecture of a computer device 10000, suitable for implementing an infrared-visible light feature separation and enhancement method based on a Laplace consensus matrix, according to Embodiment 3 of the present invention. In some embodiments, the computer device 10000 may be a terminal device such as a smartphone, wearable device, tablet computer, personal computer, in-vehicle terminal, game console, virtual machine, workbench, digital assistant, set-top box, or robot. In other embodiments, the computer device 10000 may be a rack server, blade server, tower server, or cabinet server (including independent servers or server clusters composed of multiple servers), etc. Figure 4 As shown, the computer device 10000 includes, but is not limited to, a memory 10010, a processor 10020, and a network interface 10030 that can communicate and be linked with each other via a system bus.

[0091] in:

[0092] The memory 10010 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of a computer device 10000, such as the hard disk or memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 10000. Of course, the memory 10010 may also include both the internal storage module and the external storage device of the computer device 10000. In this embodiment, the memory 10010 is typically used to store the operating system and various application software installed on the computer device 10000, such as the program code for an infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix. Furthermore, the memory 10010 can also be used to temporarily store various types of data that have been output or will be output.

[0093] In some embodiments, processor 10020 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other chip. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data interaction or communication with computer device 10000. In this embodiment, processor 10020 is used to run program code stored in memory 10010 or process data.

[0094] Network interface 10030 may include a wireless network interface or a wired network interface, which is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and communication link between computer device 10000 and the external terminal. The network may be an intranet, the Internet, Global System for Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, or other wireless or wired networks.

[0095] It should be pointed out that, Figure 4 Only computer devices with components 10010-10030 are shown; however, it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0096] In this embodiment, the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix stored in the memory 10010 can be further divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present invention.

[0097] Example 3

[0098] This invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix in the embodiments.

[0099] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix in this embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0100] Example 4

[0101] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above embodiments.

[0102] Obviously, those skilled in the art should understand that the modules or steps of the above-described embodiments of the present invention can be implemented using general-purpose computer devices. They can be centralized on a single computer device or distributed across a network of multiple computer devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computer device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular combination of hardware and software.

[0103] It should be noted that the above are merely preferred embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for infrared-visible feature separation and enhancement based on the Laplacian consensus matrix, characterized in that, Specifically, the following steps are included: S4. Perform graph Laplacian embedding on the fused spatial dimension weight matrix to obtain the consensus Laplacian matrix of the visible light image and the infrared image. It represents the common, activated discriminative features of the two modes. The fused spatial dimension weight matrix is ​​obtained by feature extraction, global pooling operation, concatenation along the channel dimension and dimensionality reduction fusion of the visible light matrix and the infrared image matrix. S5. Based on the consensus Laplacian matrix, the spatial dimension weight matrices extracted from the visible light image and the infrared image are multiplied element-wise, and then normalized to obtain the attention matrices of visible light and infrared features compared with homogeneous features, namely the visible light attention matrix and the infrared attention matrix. S6. Invert the visible light attention matrix and the infrared attention matrix to obtain the mode-specific attention weight matrices in the visible light image and the infrared image, namely the visible light attention weight matrix and the infrared attention matrix. S7. Discriminative feature enhancement is performed on visible light image features and infrared image features using attention matrix and attention weight matrix to obtain enhanced visible light image features and infrared image features; S8. The enhanced visible light image features and infrared image features are concatenated along the channel dimension, and depthwise separable convolution is used to extract and fuse global information of the concatenated image features to obtain the fused image features.

2. The method according to claim 1, characterized in that, Prior to S4, the specific steps for obtaining the fused spatial dimension weight matrix included: S1. Input the visible light image and the infrared image into the feature extractor respectively, and extract the features of the visible light image and the infrared image; S2. Extract the discriminant features for each mode from the visible light image features and infrared image features through global pooling operation, and normalize the discriminant features of each mode as the mode-specific spatial dimension weight matrix. S3. First, the spatial dimension weight matrices extracted from the visible light image and the infrared image are concatenated along the channel dimension. Then, the convolution operation is used to reduce the dimension of the concatenated weight matrix and fuse it to obtain the fused spatial dimension weight matrix.

3. The method according to claim 1 or 2, characterized in that... ， The method according to claim 1, characterized in that, In S4, the specific steps of the Graph Laplace embedding include: First, the fused spatial dimension weight matrix V is calculated using the K-nearest neighbor algorithm. fusion First, calculate the adjacency matrix A; second, calculate the spatial dimension weight matrix V. fusion degree matrix Where DA is the spatial dimension weight matrix V fusion The consensus Laplace matrix is ​​calculated as follows: M cons =D -1 / 2 (DA)D -1 / 2 ,in, 4. The method according to claim 1 or 2, characterized in that, In S7, the specific calculation method for discriminative feature enhancement is as follows: Among them, t rgb With t th These are the attention enhancement factors for the visible light mode and the infrared mode, respectively. Their initial values ​​are both set to 1 and are continuously updated during backpropagation of the deep learning network. Multiply by the channel dimension.

5. The method according to claim 2, characterized in that, In S8, the specific calculation method for extracting and fusing global information is as follows: Among them, f ReLU (·) is the ReLU activation function, f dw (·) is a depthwise separable convolution operation, and cat(·) is a vector concatenation operation along the channel dimension.

6. A computer device, characterized in that, include: At least one processor; and A memory communicatively connected to the at least one processor; wherein: The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the infrared-visible feature separation and enhancement method based on the Laplace consensus matrix as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix as described in any one of claims 1 to 5.

8. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the infrared-visible feature separation and enhancement method based on the Laplace consensus matrix as described in any one of claims 1 to 5.

Images