Infrared-visible light feature separation and enhancement method based on Laplacian consensus matrix

Abstract

The embodiment of the invention discloses an infrared-visible light feature separation and enhancement method based on a Laplacian consensus matrix to solve the problem that fused discriminative features are weakened due to the fact that semantic information of different modal features is not strongly coupled under the condition of modal failure. The method comprises the following steps: firstly, performing feature extraction, normalization operation, splicing along channel dimensions, dimension reduction fusion and graph Laplacian embedding on a visible light image and an infrared image to obtain a consensus Laplacian matrix corresponding to an activated discriminative feature which represents that two modals are common; secondly, by taking the consensus Laplacian matrix as a reference, extracting attention weight matrixes with specific modalities in the visible light image and the infrared image, and performing discriminative feature enhancement to obtain enhanced visible light image features and infrared image features; and finally, splicing the enhanced visible light image features and the infrared image features along channel dimensions, performing global information extraction and fusion on the spliced image features by using depth separable convolution to obtain fused image features, and inputting the fused image features into a decoder to obtain a final prediction result.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of image information fusion based on deep learning, and in particular to an infrared-visible light feature separation and enhancement method based on a Laplace consensus matrix. Background Art

[0002] The theory and methods of multimodal cognitive computing have an important impact on overcoming the scientific research difficulties of multi-sensor image information fusion technology. Since visible light images are limited by the spectral application range, they cannot guarantee the robustness of the detection system in practical application environments such as night, rain and fog. Infrared sensors capture the thermal radiation of objects for imaging and are not affected by the surrounding lighting conditions. Therefore, researchers began to study how to jointly utilize the complementary information between visible light images and infrared images to enhance the environmental perception ability of the detection system under harsh lighting conditions.

[0003] At present, progressive fusion is the main method for fusing image features of different modalities. It performs progressive fusion operations on image features of different modalities by designing complex deep learning network structures. However, the complex multimodal feature fusion module in progressive fusion lacks interpretability. On the one hand, it does not take into account the non-strong coupling of semantic information of different modal features in the case of modal failure. Feature fusion instead weakens the modality-specific discriminative features. On the other hand, while fusing complementary information between modalities, it introduces too much redundant noise information, reducing the robustness of the perception system.

[0004] Therefore, how to achieve complementary information fusion between different modal features in the case of modal failure and provide more stable and reliable discriminative feature information for subsequent downstream detection tasks is an urgent problem to be solved in the field of image information fusion. Summary of the invention

[0005] In order to overcome the above technical defects, an embodiment of the present invention provides an infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix to solve the problem that under modal failure conditions, the semantic information of different modal features is not strongly coupled, resulting in the weakening of the discriminative features after fusion.

[0006] In a first aspect, an embodiment of the present invention provides an infrared-visible light feature separation and enhancement method based on a Laplace consensus matrix, which specifically includes the following steps:

[0007] S4. Perform graph Laplacian embedding on the fused spatial dimension weight matrix to obtain the consensus Laplacian matrix of the visible light image and the infrared image, which represents the common and activated discriminative features of the two modalities. The fused spatial dimension weight matrix is ​​obtained by performing feature extraction, global pooling operation, splicing along the channel dimension and dimensionality reduction fusion on the visible light matrix and the infrared image matrix;

[0008] S5. Based on the consensus Laplace matrix, the spatial dimension weight matrices extracted from the visible light image and the infrared image are respectively multiplied element by element, and then the attention matrices of the visible light and infrared features compared with the homogeneous features are obtained through normalization operation, namely, the visible light attention matrix and the infrared attention matrix;

[0009] S6, performing an inversion operation on the visible light attention matrix and the infrared attention matrix to obtain modality-specific attention weight matrices in the visible light image and the infrared image, namely, the visible light attention weight matrix and the infrared attention matrix;

[0010] S7, using the attention matrix and the attention weight matrix to perform discriminative feature enhancement on the visible light image features and the infrared image features to obtain enhanced visible light image features and infrared image features;

[0011] S8. Splice the enhanced visible light image features and infrared image features along the channel dimension, and use depthwise separable convolution to extract and fuse the global information of the spliced ​​image features to obtain the fused image features.

[0012] Optionally, in a possible implementation of the first aspect,

[0013] Before S4, the specific steps for obtaining the fused spatial dimension weight matrix include:

[0014] S1, inputting the visible light image and the infrared image into the feature extractor respectively, extracting the visible light image features and the infrared image features;

[0015] S2, extracting the discriminant features of each modality in the visible light image features and the infrared image features through a global pooling operation, and normalizing the discriminant features of each modality as a modality-specific spatial dimension weight matrix;

[0016] S3. First, the spatial dimension weight matrices extracted from the visible light image and the infrared image are spliced ​​along the channel dimension, and then the convolution operation is used to reduce the dimension of the spliced ​​weight matrices to obtain the fused spatial dimension weight matrix.

[0017] Optionally, in a possible implementation manner of the first aspect, the specific steps of the graph Laplacian embedding include:

[0018] First, the fused spatial dimension weight matrix V is calculated according to the K nearest neighbor algorithm fusion The adjacency matrix A; secondly, calculate the spatial dimension weight matrix V fusion The degree matrix Where DA is the spatial dimension weight matrix V fusionThe specific calculation method of the consensus Laplace matrix is:

[0019] M cons =D -1 / 2 (DA)D -1 / 2 ,in,

[0020] Optionally, in a possible implementation of the first aspect, the calculation method of the above-mentioned discriminative feature enhancement is specifically as follows:

[0021]

[0022] Among them, t rgb With t th are the attention enhancement factors for the visible light modality and the infrared modality, respectively. Their initial values ​​are both set to 1 and are continuously updated with the back propagation of the deep learning network. Multiply by the channel dimension.

[0023] Optionally, in a possible implementation manner of the first aspect, a specific calculation method for extracting and fusing the global information is as follows:

[0024]

[0025] Among them, f ReLU (·) is the ReLU activation function, f dw (·) is a depth-wise separable convolution operation, and cat(·) is a vector concatenation operation along the channel dimension.

[0026] In a second aspect, another aspect of an embodiment of the present invention provides a computer device, including:

[0027] at least one processor; and

[0028] a memory communicatively coupled to the at least one processor;

[0029] Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0030] In the third aspect, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0031] In a fourth aspect, another aspect of an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0032] The above-mentioned technical scheme is adopted in the embodiment of the present invention to achieve the following technical effects: by constructing a consensus Laplace matrix of multimodal image features, such as visible light modality and infrared modality, and then separating homogeneous redundant information and heterogeneous complementary information in different modalities, the foreground target information and the background noise information are effectively separated, thereby improving the accuracy and reliability of the perception system in non-ideal environments in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0034] Figure 1 A schematic diagram of a flow chart of an infrared-visible light feature separation and enhancement method based on a Laplace consensus matrix in an embodiment of the present application;

[0035] Figure 2 A schematic diagram showing a comparison of the characteristic intensity visualization heat map of the method in the embodiment of the present application with other methods;

[0036] Figure 3 This is a schematic diagram of qualitative experimental comparison between the method in the embodiment of the present application and other methods in different environments;

[0037] Figure 4 A schematic diagram of the hardware structure of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0039] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present invention are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0040] In the description of the present invention, it should be understood that the numerical labels before the steps do not identify the order of executing the steps, but are only used to facilitate the description of the present invention and distinguish each step, and therefore cannot be understood as a limitation of the present invention.

[0041] In order to facilitate understanding of the technical solutions in the present invention, some technical terms involved in the field are briefly introduced below:

[0042] Modality: Modality refers to different types of information input or output forms in a system, device, or perception process. The modalities applicable to this solution are not limited to image modalities such as visible light and infrared, but also include video, sound, text, and other modalities.

[0043] Modal failure: Modal failure refers to a problem in the transmission or processing of information in one or more modes in a system, device, or perception process, which causes the system to be unable to understand or respond correctly. The system needs to have a mechanism to deal with this failure in order to continue to work normally.

[0044] Discriminative features: Discriminative features refer to the features that can distinguish different categories of data to the greatest extent in the perception system.

[0045] Foreground target information: Foreground target information refers to the main object or area of ​​interest in image or video processing tasks. These objects are usually distinguished from the background and contain key information required for the task.

[0046] Background redundancy or noise information: Background redundancy or noise information refers to unnecessary parts that are irrelevant to the target in image, video or signal processing. This information often interferes with the analysis or decision-making of the perception system.

[0047] Homogeneous redundant information under different modalities: Homogeneous redundant information under different modalities means that in a multimodal perception system, different modalities express the same or similar information, which reduces the perception system's ability to extract key information.

[0048] Heterogeneous complementary information under different modalities: Heterogeneous complementary information under different modalities means that information from different modalities can complement each other and provide unique discriminative information in different dimensions, which helps the model to comprehensively consider more information when facing complex decisions and make more informed choices.

[0049] Resnet algorithm: Resnet algorithm refers to a deep learning neural network architecture, which is often used for feature information extraction and embedding of data such as pictures and texts.

[0050] Embodiment 1

[0051] Next, the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix in the present invention is described in detail in combination with the accompanying drawings and specific embodiments, as follows:

[0052] like Figure 1 As shown, the specific steps of the method described in the present invention include:

[0053] S1. Input the visible light image and the infrared image into the feature extractor respectively to extract the visible light image features and the infrared image features.

[0054] It should be noted that the technical solution in this application is not only applicable to image modalities such as visible light and infrared, but also to feature separation and enhancement of modalities such as video, sound, and text. This application only uses visible light images and infrared images as examples for illustration.

[0055] Specifically, the feature extractor is constructed using the backbone network of the Resnet algorithm. The above visible light image feature is denoted as x RGB , the infrared image feature is recorded as x T , and then the above-mentioned input of the infrared image and the visible light image into the feature extractor can be specifically expressed as:

[0056] x rgb =Resnet(X rgb ), and x th =Resnet(X th );

[0057] Among them, X rgb and X th They represent visible light images and infrared images respectively. The image resolutions of visible light images and infrared images are the same, for example, both are 480*640. The feature embedding dimensions of visible light image features and infrared image features are the same, for example, x RGB With x th The feature embedding dimensions of the models are all set to 256.

[0058] S2. The discriminant features of each modality in the visible light image features and infrared image features are extracted through global pooling operations, and the discriminant features of each modality are normalized and used as the modality-specific spatial dimension weight matrix.

[0059] Specifically, the visible light image feature is recorded as x RGB The modality-specific spatial dimension weight matrix obtained by globalizing and normalizing the visible light image features in S2 is denoted as V rgb ; Similarly, the visible light image feature is recorded as x th The modality-specific spatial dimension weight matrix obtained by globalizing and normalizing the visible light image features in S2 is denoted as Vth .

[0060] Further, V rgb =V rgb =σ(f MLP (f GMP (x rgb )));V th =V th =σ(f MLP (f GMP (x th ))), where X rgb and X th represent visible light image and infrared image respectively, σ(·) is the sigmoid activation function, and f MLP (·) is a multi-layer linear mapping, f GMP (·) is the global maximum pooling function used to perform the global pooling operation.

[0061] S3. First, the spatial dimension weight matrices extracted from the visible light image and the infrared image are spliced ​​along the channel dimension, and then the convolution operation is used to reduce the dimension of the spliced ​​weight matrices to obtain the fused spatial dimension weight matrix.

[0062] Specifically, the channel dimension mentioned above refers to the number of eigenvectors in the spatial dimension matrix; the fused spatial dimension weight matrix is ​​denoted as V fusion , specifically: V fusion =Conv 1×1 (cat(V rgb ,V th ));

[0063] Among them, cat(·) is the matrix concatenation operation along the channel dimension, Conv 1×1 (·) is the convolution dimension reduction operation with a convolution kernel size of 1. V fusion The dimension of the weight matrix is ​​the same as the feature embedding dimension of the visible light image feature, for example, 256.

[0064] Furthermore, the matrix concatenation operation along the channel dimension can retain all feature information of the weight matrix; the matrix concatenation operation along the channel dimension can compress the number of channel dimensions, retain key feature information, and reduce computational complexity;

[0065] The matrix concatenation operation along the channel dimension can retain all the feature information of the weight matrix; the matrix concatenation operation along the channel dimension can compress the number of channel dimensions, retain key feature information, and reduce computational complexity.

[0066] S4. Perform graph Laplacian embedding on the fused spatial dimension weight matrix to obtain the consensus Laplacian matrix of the visible light image and the infrared image, which represents the common and activated discriminative features of the two modalities.

[0067] Specifically, the consensus Laplace matrix is ​​denoted as M cons Graph Laplace embedding refers to capturing the local geometric structure of data based on the Laplace operator in graph theory, thereby preserving the similarity relationship between data points in a low-dimensional space.

[0068] Optionally, in some implementations, the specific steps of the graph Laplace embedding include:

[0069] First, the fused spatial dimension weight matrix V is calculated according to the K nearest neighbor algorithm fusion The adjacency matrix A; secondly, calculate the spatial dimension weight matrix V fusion The degree matrix Where DA is the spatial dimension weight matrix V fusion The specific calculation method of the consensus Laplace matrix is:

[0070] M cons =D -1 / 2 (DA)D -1 / 2 ,in,

[0071] S5. Based on the consensus Laplace matrix, the spatial dimension weight matrices extracted from the visible light image and the infrared image are respectively multiplied element-by-element, and then the attention matrices of the visible light and infrared features compared with the homogeneous features are obtained through normalization operation, namely, the visible light attention matrix and the infrared attention matrix.

[0072] Specifically, the attention matrices of infrared and visible light features compared to homogeneous features (i.e., visible light attention matrix and infrared attention matrix) are denoted as M rgb and M th , which are specifically expressed as: M rgb =Softmax(M cons *V rgb ) and M th =Softmax(M cons *V th );

[0073] Among them, Softmax(·) is the Softmax normalization function, and the visible light attention matrix M rgb With the infrared attention matrix M th The parts with the smallest weights represent the unique inconsistency information in the visible light image and infrared image features.

[0074] S6. Perform an inversion operation on the visible light attention matrix and the infrared attention matrix to obtain the modality-specific attention weight matrices in the visible light image and the infrared image, namely, the visible light attention weight matrix and the infrared attention matrix.

[0075] Specifically, the visible light attention weight matrix and the infrared attention matrix are recorded as and Specifically, they are: and

[0076] S7. Use the attention matrix and the attention weight matrix to perform discriminative feature enhancement on the visible light image features and the infrared image features to obtain enhanced visible light image features and infrared image features.

[0077] Specifically, the enhanced visible light image features are recorded as The enhanced infrared image features are recorded as Optionally, in some possible implementations, the calculation method of the above discriminative feature enhancement is specifically as follows:

[0078]

[0079] Among them, t rgb With t th are the attention enhancement factors for the visible light modality and the infrared modality, respectively. Their initial values ​​are both set to 1 and are continuously updated with the back propagation of the deep learning network. Multiply by the channel dimension.

[0080] S8. Splice the enhanced visible light image features and infrared image features along the channel dimension, and use depthwise separable convolution to extract and fuse the global information of the spliced ​​image features to obtain the fused image features.

[0081] Specifically, the fused image features are recorded as Optionally, in some implementations, the specific calculation method for extracting and fusing the above global information is as follows:

[0082]

[0083] Among them, f ReLU (·) is the ReLU activation function, f dw (·) is a depth-wise separable convolution operation, and cat(·) is a vector concatenation operation along the channel dimension.

[0084] S9. Input the fused image features into the decoder to obtain the final prediction result.

[0085] In the embodiment of the present invention, by constructing a consensus Laplacian matrix of multimodal image features, such as visible light modality and infrared modality, the homogeneous redundant information in different modalities is separated. rgb With M th , and heterogeneous complementary information, and It achieves effective separation of foreground target information and background noise information, improving the accuracy and reliability of the perception system in non-ideal environments in real scenarios.

[0086] Based on the above technical solution, the following experiment was conducted. The dataset used in the experiment was the open source dataset MFNet. The MFNet dataset contains 1569 pairs of visible light and infrared images (820 pairs collected during the day and 749 pairs collected at night), and 9 categories such as pedestrians and vehicles are annotated. The example algorithm network of the present invention is implemented based on the Pytorch framework, and AdamW is used as the model training optimizer. The initial learning rate is set to 0.0001, and the learning rate is reduced by 10 every 3 rounds. -1 The training time is 12 rounds. Considering the small number of training samples in the visible light and infrared image datasets, during the training process, the present invention example adopts random cropping and random horizontal flipping as data enhancement strategies in the implementation process. Figure 2 and Figure 3 The above experiments are compared with other methods in terms of the effects of fusion feature matrix heat map and semantic segmentation tasks.

[0087] Figure 2 The visualization comparison of the fusion feature matrix heat map of the technical solution of the present invention and other methods is shown, (a) is a visible light image, (b) is an infrared image, (c) is a visible light feature matrix heat map, (d) is an infrared feature matrix heat map, (e) is a FEANet fusion feature matrix heat map, (f) is a GMNet fusion feature matrix heat map, and (g) is a fusion feature matrix heat map of the present invention. The highlighted areas in the heat map represent where the perception system focuses. Under environmental conditions such as daytime, glare, and low light, compared with other methods, the fusion feature matrix heat map of the present invention focuses more on the foreground target, showing reliable and accurate perception capabilities.

[0088] Figure 3The qualitative experimental comparison of the method of the present invention and other methods on the semantic segmentation task is shown, (a) is a visible light image, (b) is an infrared image, (c) is the semantic segmentation result of CMNeXt, (d) is the semantic segmentation result of CMX, (e) is the semantic segmentation result of EAEFNet, (f) is the semantic segmentation result of SpiderMesh, (g) is the semantic segmentation result of the present invention, and (h) is the true value of semantic segmentation. Based on the true value of semantic segmentation (h), the completeness and accuracy of the present invention for different segmentation targets in different scenarios are higher than those of other comparison methods. For example, a bicycle disturbed by glare, a triangular cone in a low-light environment, etc.

[0089] Embodiment 2

[0090] Figure 4 The hardware architecture diagram of a computer device 10000 suitable for implementing the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix according to the third embodiment of the present invention is schematically shown. In some embodiments, the computer device 10000 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 10000 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 4 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 which can be interconnected and communicated via a system bus.

[0091] in:

[0092] The memory 10010 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10010 may be an internal storage module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 may also be an external storage device of the computer device 10000, such as a plug-in hard disk equipped on the computer device 10000, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10010 may also include both the internal storage module of the computer device 10000 and its external storage device. In this embodiment, the memory 10010 is generally used to store the operating system and various application software installed in the computer device 10000, such as the program code of the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix, etc. In addition, the memory 10010 can also be used to temporarily store various data that have been output or are to be output.

[0093] In some embodiments, the processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication with the computer device 10000. In this embodiment, the processor 10020 is used to run the program code stored in the memory 10010 or process data.

[0094] The network interface 10030 may include a wireless network interface or a wired network interface, and the network interface 10030 is generally used to establish a communication link between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0095] It should be pointed out that Figure 4 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0096] In this embodiment, the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present invention.

[0097] Embodiment 3

[0098] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix in the embodiment are implemented.

[0099] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the computer-readable storage medium can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store the operating system and various application software installed on the computer device, such as the program code of the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix in the embodiment. In addition, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or are to be output.

[0100] Embodiment 4

[0101] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0102] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, and optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0103] It should be noted that the above are only preferred embodiments of the present invention, and the patent protection scope of the present invention is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for separating and enhancing infrared-visible light features based on Laplace consensus matrix, characterized in that: The specific steps include: S4. Perform graph Laplacian embedding on the fused spatial dimension weight matrix to obtain the consensus Laplacian matrix of the visible light image and the infrared image, which represents the common and activated discriminative features of the two modalities. The fused spatial dimension weight matrix is ​​obtained by performing feature extraction, global pooling operation, splicing along the channel dimension and dimensionality reduction fusion on the visible light matrix and the infrared image matrix; S5. Based on the consensus Laplace matrix, the spatial dimension weight matrices extracted from the visible light image and the infrared image are respectively multiplied element by element, and then the attention matrices of the visible light and infrared features compared with the homogeneous features are obtained through normalization operation, namely, the visible light attention matrix and the infrared attention matrix; S6, performing an inversion operation on the visible light attention matrix and the infrared attention matrix to obtain modality-specific attention weight matrices in the visible light image and the infrared image, namely, the visible light attention weight matrix and the infrared attention matrix; S7, using the attention matrix and the attention weight matrix to perform discriminative feature enhancement on the visible light image features and the infrared image features to obtain enhanced visible light image features and infrared image features; S8. Splice the enhanced visible light image features and infrared image features along the channel dimension, and use depthwise separable convolution to extract and fuse the global information of the spliced ​​image features to obtain the fused image features.

2. The method according to claim 1, characterized in that: Before S4, the specific steps for obtaining the fused spatial dimension weight matrix include: S1, inputting the visible light image and the infrared image into the feature extractor respectively, extracting the visible light image features and the infrared image features; S2, extracting the discriminant features of each modality in the visible light image features and the infrared image features through a global pooling operation, and normalizing the discriminant features of each modality as a modality-specific spatial dimension weight matrix; S3. First, the spatial dimension weight matrices extracted from the visible light image and the infrared image are spliced ​​along the channel dimension, and then the convolution operation is used to reduce the dimension of the spliced ​​weight matrices to obtain the fused spatial dimension weight matrix.

3. The method according to claim 1 or 2, characterized in that ， The method according to claim 1, characterized in that In S4, the specific steps of graph Laplacian embedding include: First, the fused spatial dimension weight matrix V is calculated according to the K nearest neighbor algorithm fusion The adjacency matrix A; secondly, calculate the spatial dimension weight matrix V fusion The degree matrix Where DA is the spatial dimension weight matrix V fusion The specific calculation method of the consensus Laplace matrix is: M cons =D -1 / 2 (DA)D -1 / 2 ,in, 4. The method according to claim 1 or 2, characterized in that: In S7, the calculation method of discriminative feature enhancement is as follows: Among them, t rgb With t th are the attention enhancement factors for the visible light modality and the infrared modality, respectively. Their initial values ​​are both set to 1 and are continuously updated with the back propagation of the deep learning network. Multiply by the channel dimension.

5. The method according to claim 2, characterized in that: In S8, the specific calculation method for extracting and fusing global information is as follows: Among them, f ReLU (·) is the ReLU activation function, f dw (·) is a depth-wise separable convolution operation, and cat(·) is a vector concatenation operation along the channel dimension.

6. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by the processor, the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix as described in any one of claims 1 to 5 is implemented.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the infrared-visible light feature separation and enhancement method based on the Laplace consensus matrix described in any one of claims 1 to 5 is implemented.

Images