Infrared and visible image fusion method based on Q-Bench large model assisted perception

Abstract

The present invention relates to an infrared-visible image fusion method for Q‑Bench large model assisted perception, and belongs to the field of image fusion. The present invention comprises: inputting an infrared image and a visible light image into respective feature encoders, obtaining infrared image features and visible light image features, and then splicing and fusing them to obtain fusion features, and reconstructing a fused image through the fusion features; inputting the fused image into the Q‑Bench large model to obtain feedback text, a reconciliation prompt generation module obtains a reconciliation prompt through feedback text features and fusion features, inputting the reconciliation prompt and the fusion features into a mapping reconciliation module together, correcting the fusion features, and reconstructing a fused image with excellent visual effects using the corrected post-fusion features. The present invention utilizes the powerful image analysis capability of the large model to evaluate the quality of the fused image and provide feedback on the fusion features, thereby obtaining a fused image with higher quality and improving the fusion performance.

Description

Technical Field

[0001] The invention relates to an infrared visible image fusion method for Q-Bench large model assisted perception, and belongs to the technical field of image fusion. Background Art

[0002] Visible light sensors can clearly image objects visible to the human eye, and their imaging results have the advantages of high spatial resolution, rich texture details, and better conformity to human visual perception. However, this sensor cannot clearly image objects in bad weather scenes such as fog and haze. Infrared sensors measure the temperature difference between the target and the background for imaging, which can effectively avoid the impact of bad conditions on target imaging, but still cannot clearly image texture detail information at a relatively constant temperature. In order to integrate the advantages of these two types of imaging sensors, infrared and visible light image fusion technology came into being. This technology can integrate the complementary information extracted by the two types of sensors into an image with clear salient targets, backgrounds, and texture details, achieving a more comprehensive description of the scene. At present, large models have been initially applied in various fields due to their powerful reasoning and analysis capabilities. The existing infrared and visible light image fusion methods do not have a more objective solution for the visual quality evaluation of images, which limits the further improvement of the quality of fused images. Therefore, in response to the above problems, an infrared and visible image fusion method assisted by Q-Bench large model perception is proposed. Summary of the invention

[0003] In order to address the shortcomings of the existing methods, the present invention provides an infrared-visible image fusion method assisted by Q-Bench large model perception. The method utilizes the large model's powerful understanding and reasoning capabilities of the image content, uses the text of the large model's feedback on image quality as a prompt, participates in the reconciliation process of fusion features, and specifically improves the quality of the fused image, obtaining a fusion result with superior visual effects.

[0004] The technical solution of the present invention is: a Q-Bench large model assisted perception infrared visible image fusion method, the method comprising:

[0005] Step 1: Obtain a training data set for infrared image and visible light image fusion and perform preprocessing;

[0006] Step 2: Input the infrared image and the visible light image into their respective feature encoders to obtain infrared image features and visible light image features;

[0007] Step 3: Splice and fuse the infrared image features and the visible light image features to obtain fused features, and reconstruct the fused image through the fused features;

[0008] Step 4: Input the fused image into the Q-Bench large model to obtain feedback text. The reconciliation prompt generation module obtains the reconciliation prompt through the feedback text features and the fusion features. The reconciliation prompt and the fusion features are input into the mapping reconciliation module together. The fusion features are corrected and the corrected post-fusion features are used to reconstruct a fused image with excellent visual effects.

[0009] Furthermore, in step 1, each infrared-visible light image pair in the training data set consists of an infrared image and a visible light image, and the size of the image of each modality is 256×256; the method for preprocessing the training data set includes: random flipping, random rotation, and normalizing the image.

[0010] Furthermore, the step 2 comprises:

[0011] The visible light image Input visible light image feature encoder In the infrared image Input infrared image feature encoder In the example, the visible light image features are obtained respectively. and infrared image features ; The feature extraction process of visible light image and infrared image is expressed as:

[0012] ;

[0013] Each visible light image feature encoder and infrared image feature encoder is composed of three 3×3 convolutional layers and feature extraction blocks consisting of ReLU activation functions connected by dense connections, and finally composed of a Concat operation and a 1×1 convolutional layer.

[0014] Furthermore, in step 3, the infrared image feature and visible light image features Perform splicing and fusion to obtain fusion features , and reconstruct the fused image by fusing features; obtain the fused image The process is as follows:

[0015] F f = [ F ir , F vis ] I f = D f ( F f ) ;

[0016] in, It is a decoder consisting of a 3×3 convolutional layer and a Tanh activation function.

[0017] Furthermore, in step 3, texture loss is used to retain the texture details of the infrared image and the visible light image, and then used to retain the texture details of the fused image; texture loss The calculation formula is as follows:

[0018] ;

[0019] in, express norm, represents pixel-by-pixel maximum selection, , , express , , The gradient of

[0020] In terms of preserving image content information, content loss is defined to bring the fused image closer to the source image at the pixel level; content loss The calculation formula is as follows:

[0021] ; ;

[0022] in, and Infrared images and visible light images The significance matrix of and They are and The weight graph of .

[0023] Furthermore, the step 4 comprises:

[0024] Step 4.1: Fuse the image The query text is input into the pre-trained Q-Bench model to obtain feedback text about image quality; the LLaMA model is used to extract features from the feedback text to obtain feedback text features ;Features of feedback text Mapping is used to make feedback text features Map the image features into the same feature space to get text prompts , text prompt The calculation formula is as follows:

[0025] ;

[0026] in, Represents a mapping operation;

[0027] Step 4.2: Text prompt As feedback and fusion features Calculate the relationship matrix; Relationship Matrix The calculation formula is as follows:

[0028] ;

[0029] in, represents the dot product operation, and softmax() represents the softmax activation function;

[0030] Step 4.3: The calculated relationship matrix Acts on text prompts , get tips for adjusting image quality:

[0031] ;

[0032] in, Represents a transpose operation;

[0033] Step 4.4: Combine the reconciliation hint H with the fusion feature Input into the mapping and reconciliation module together to fusion feature Correction is performed in order to improve the visual effect of the fused image and reconstruct a fused image with excellent visual effect. .

[0034] Furthermore, in step 4.4, the fused image with excellent visual effect is reconstructed The process includes:

[0035] (1) Use 1×1 convolution to swap and hint H and fusion features To map:

[0036] ;

[0037] in, , , Represents 3 different 1×1 convolutions; Indicates that The fusion features after mapping, Indicates that Blending tips after mapping, Indicates that Reconciliation tips after mapping;

[0038] (2) Parameter modulation is performed on the above mapped features to obtain the corrected fusion features :

[0039] ;

[0040] in, is an element-wise multiplication operation, It is an element-by-element addition operation;

[0041] (3) The corrected fusion features are obtained Input Decoder In the process of reconstructing a fusion image with excellent visual effects .

[0042] Furthermore, in step 4, a joint gradient loss is used to preserve the texture details of the infrared image and the visible light image, thereby preserving a fused image with excellent visual effects. Texture details; Joint gradient loss The calculation formula is as follows:

[0043] ;

[0044] in, express The gradient of

[0045] In terms of image content information preservation, defining pixel intensity loss will produce visually excellent fused images Closer to the pixel intensity of the source image, pixel intensity loss is defined as follows:

[0046] ;

[0047] ;

[0048] in, and They are and The significance matrix of .

[0049] The present invention provides an infrared-visible image fusion system for Q-Bench large model assisted perception, and the system comprises: a module for executing the infrared-visible image fusion method for Q-Bench large model assisted perception.

[0050] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the infrared-visible image fusion method for assisting perception with a Q-Bench large model is implemented.

[0051] The beneficial effects of the present invention are:

[0052] 1. The present invention introduces the Q-Bench large model to participate in the evaluation of the visualization quality of fused images, thus filling the gap in the objective evaluation of the visualization quality of fused images in the field of infrared and visible light image fusion;

[0053] 2. The present invention obtains text feedback by inputting the fusion result into the Q-Bench large language model, and then uses the text feedback to modulate the fusion result, thereby realizing the organic combination of text and image;

[0054] 3. The present invention uses the Q-Bench large model, the reconciliation prompt generation module, and the mapping reconciliation module, and utilizes the large model's powerful analysis and reasoning capabilities for image content. The text content fed back by the large model is used as a prompt to participate in the reconciliation of fusion features, thereby obtaining a higher quality fusion image and improving the fusion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0056] Figure 2 It is a structural schematic diagram of a feature encoder of the present invention;

[0057] Figure 3 A schematic diagram of a reconciliation prompt generation process of the present invention;

[0058] Figure 4 Schematic diagram of the mapping and reconciliation module of the present invention. DETAILED DESCRIPTION

[0059] Example 1: Figure 1-Figure 4 As shown, the infrared visible image fusion method for Q-Bench large model assisted perception includes:

[0060] Step 1: Obtain a training data set for infrared image and visible light image fusion and perform preprocessing; each infrared-visible light image pair in the training data set consists of an infrared image and a visible light image, and the size of each modality image is 256×256; the method of preprocessing the training data set includes: random flipping, random rotation, and normalizing the image.

[0061] Step 2: Input the infrared image and the visible light image into their respective feature encoders to obtain infrared image features and visible light image features;

[0062] Furthermore, the step 2 comprises:

[0063] The visible light image Input visible light image feature encoder In the infrared image Input infrared image feature encoder In the example, the visible light image features are obtained respectively. and infrared image features ; The feature extraction process of visible light image and infrared image is expressed as:

[0064] ;

[0065] Each visible light image feature encoder and infrared image feature encoder is composed of three 3×3 convolutional layers and feature extraction blocks consisting of ReLU activation functions connected by dense connections, and finally composed of a Concat operation and a 1×1 convolutional layer.

[0066] Step 3: Splice and fuse the infrared image features and the visible light image features to obtain fused features, and reconstruct the fused image through the fused features;

[0067] Furthermore, in step 3, the infrared image feature and visible light image features Perform splicing and fusion to obtain fusion features , and reconstruct the fused image by fusing features; obtain the fused image The process is as follows:

[0068] F f = [ F ir , F vis ] I f = D f ( F f ) ;

[0069] in, It is a decoder consisting of a 3×3 convolutional layer and a Tanh activation function.

[0070] Furthermore, in step 3, the fused image should have clear texture detail information and significant target information to achieve an effective and comprehensive representation of the scene. In order to retain as many texture details in the infrared image and the visible light image as possible, a texture loss is defined. The texture loss is used to retain the texture details of the infrared image and the visible light image, and then used to retain the texture details of the fused image. Texture loss The calculation formula is as follows:

[0071] ;

[0072] in, express norm, represents pixel-by-pixel maximum selection, , , express , , The gradient of

[0073] In terms of preserving image content information, content loss is defined to bring the fused image closer to the source image at the pixel level; content loss The calculation formula is as follows:

[0074] ;

[0075] ;

[0076] in, and Infrared images and visible light images The significance matrix of and They are and The weight graph of .

[0077] Step 4: Input the fused image into the Q-Bench large model to obtain feedback text. The reconciliation prompt generation module obtains the reconciliation prompt through the feedback text features and the fusion features. The reconciliation prompt and the fusion features are input into the mapping reconciliation module together. The fusion features are corrected and the corrected post-fusion features are used to reconstruct a fused image with excellent visual effects.

[0078] Furthermore, the step 4 comprises:

[0079] Step 4.1: Fuse the image The query text “How can we improve the quality of the image?” is input into the pre-trained Q-Bench model to obtain feedback text about the image quality (for example, “Make the pedestrian more salient, Enhance the contrast of the image”). In order to obtain robust text features, the LLaMA model is used to extract features from the feedback text to obtain feedback text features. ; Due to the large differences in the fundamental properties of text features and image features, the feedback text features Mapping is used to make feedback text features Map the image features into the same feature space to get text prompts , text prompt The calculation formula is as follows:

[0080] ;

[0081] in, Represents a mapping operation;

[0082] Step 4.2: Text prompt As feedback and fusion features Calculate the relationship matrix; Relationship Matrix The calculation formula is as follows:

[0083] ;

[0084] in, represents the dot product operation, and softmax() represents the softmax activation function;

[0085] Step 4.3: The calculated relationship matrix Acts on text prompts , get tips for adjusting image quality:

[0086] ;

[0087] in, Represents a transpose operation;

[0088] Step 4.4: Combine the reconciliation hint H with the fusion feature Input into the mapping and reconciliation module together to fusion feature Correction is performed in order to improve the visual effect of the fused image and reconstruct a fused image with excellent visual effect. .

[0089] Furthermore, in step 4.4, the fused image with excellent visual effect is reconstructed The process includes:

[0090] (1) Use 1×1 convolution to swap and hint H and fusion features To map:

[0091] ;

[0092] in, , , Represents 3 different 1×1 convolutions; Indicates that The fusion features after mapping, Indicates that Blending tips after mapping, Indicates that Reconciliation tips after mapping;

[0093] (2) Parameter modulation is performed on the above mapped features to obtain the corrected fusion features :

[0094] ;

[0095] in, is an element-wise multiplication operation, It is an element-by-element addition operation;

[0096] (3) The corrected fusion features are obtained Input Decoder In the process of reconstructing a fusion image with excellent visual effects .

[0097] Furthermore, in step 4, in order to ensure the quality of the corrected fused image and retain as many texture details in the infrared image and the visible light image as possible, the following joint gradient loss is defined, which is used to retain the texture details of the infrared image and the visible light image, and then to retain the fused image with excellent visual effect. Texture details; Joint gradient loss The calculation formula is as follows:

[0098] ;

[0099] in, express The gradient of

[0100] In terms of image content information preservation, defining pixel intensity loss will produce visually excellent fused images Closer to the pixel intensity of the source image, pixel intensity loss is defined as follows:

[0101] ;

[0102] ;

[0103] in, and They are and The significance matrix of .

[0104] The present invention provides an infrared-visible image fusion system for Q-Bench large model assisted perception, and the system comprises: a module for executing the infrared-visible image fusion method for Q-Bench large model assisted perception.

[0105] The present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the infrared-visible image fusion method for assisting perception with a Q-Bench large model is implemented.

[0106] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. The infrared and visible image fusion method assisted by Q-Bench large model perception is characterized by: The method comprises: Step 1: Obtain a training data set for infrared image and visible light image fusion and perform preprocessing; Step 2: Input the infrared image and the visible light image into their respective feature encoders to obtain infrared image features and visible light image features; Step 3: Splice and fuse the infrared image features and the visible light image features to obtain fused features, and reconstruct the fused image through the fused features; Step 4: Input the fused image into the Q-Bench large model to obtain feedback text. The reconciliation prompt generation module obtains the reconciliation prompt through the feedback text features and the fusion features. The reconciliation prompt and the fusion features are input into the mapping reconciliation module together. The fusion features are corrected and the corrected post-fusion features are used to reconstruct a fused image with excellent visual effects. The step 4 comprises: Step 4.1: Fuse the image The query text is input into the pre-trained Q-Bench model to obtain feedback text about image quality; the LLaMA model is used to extract features from the feedback text to obtain feedback text features ;Features of feedback text Mapping is used to make feedback text features Map the image features into the same feature space to get text prompts , text prompt The calculation formula is as follows: ； in, Represents a mapping operation; Step 4.2: Text prompt As feedback and fusion features Calculate the relationship matrix; Relationship Matrix The calculation formula is as follows: ； in, represents the dot product operation, and softmax() represents the softmax activation function; Step 4.3: The calculated relationship matrix Acts on text prompts , get tips for adjusting image quality: ； in, Represents a transpose operation; Step 4.4: Combine the reconciliation hint H with the fusion feature Input into the mapping and reconciliation module together to fusion feature Correction is performed in order to improve the visual effect of the fused image and reconstruct a fused image with excellent visual effect. .

2. The infrared-visible image fusion method for Q-Bench large model assisted perception according to claim 1 is characterized by: In step 1, each infrared-visible light image pair in the training data set consists of an infrared image and a visible light image, and the size of the image of each modality is 256×256; Methods for preprocessing the training data set include: random flipping, random rotation, and normalizing the image.

3. The infrared-visible image fusion method for Q-Bench large model assisted perception according to claim 1 is characterized by: The step 2 comprises: The visible light image Input visible light image feature encoder In the infrared image Input infrared image feature encoder In the example, the visible light image features are obtained respectively. and infrared image features ; The feature extraction process of visible light image and infrared image is expressed as: ； Each visible light image feature encoder and infrared image feature encoder is composed of three 3×3 convolutional layers and feature extraction blocks consisting of ReLU activation functions connected by dense connections, and finally composed of a Concat operation and a 1×1 convolutional layer.

4. The infrared-visible image fusion method for Q-Bench large model assisted perception according to claim 1 is characterized by: In step 3, the infrared image features and visible light image features Perform splicing and fusion to obtain fusion features , and reconstruct the fused image by fusing features; obtain the fused image The process is as follows: ； in, It is a decoder consisting of a 3×3 convolutional layer and a Tanh activation function.

5. The infrared-visible image fusion method for Q-Bench large model assisted perception according to claim 1 is characterized by: In step 3, texture loss is used to retain the texture details of the infrared image and the visible light image, and then used to retain the texture details of the fused image; texture loss The calculation formula is as follows: ； in, express norm, represents pixel-by-pixel maximum selection, , , express , , The gradient of In terms of preserving image content information, content loss is defined to bring the fused image closer to the source image at the pixel level; content loss The calculation formula is as follows: ； ； in, and Infrared images and visible light images The significance matrix of and They are and The weight graph of .

6. The infrared-visible image fusion method for Q-Bench large model assisted perception according to claim 1 is characterized by: In step 4.4, the fused image with excellent visual effect is reconstructed The process includes: (1) Use 1×1 convolution to swap and hint H and fusion features To map: ； in, , , Represents 3 different 1×1 convolutions; Indicates that The fusion features after mapping, Indicates that Blending tips after mapping, Indicates that Reconciliation tips after mapping; (2) Parameter modulation is performed on the above mapped features to obtain the corrected fusion features : ； in, is an element-wise multiplication operation, It is an element-by-element addition operation; (3) The corrected fusion features are obtained Input Decoder In the process of reconstructing a fusion image with excellent visual effects .

7. The infrared-visible image fusion method for Q-Bench large model assisted perception according to claim 1 is characterized by: In step 4, a joint gradient loss is used to preserve the texture details of the infrared image and the visible light image, thereby preserving a fused image with excellent visual effects. Texture details; Joint gradient loss The calculation formula is as follows: ； in, express The gradient of In terms of image content information preservation, defining pixel intensity loss will produce a fused image with excellent visual effect Closer to the pixel intensity of the source image, pixel intensity loss is defined as follows: ； ； in, and They are and The significance matrix of .

8. Q-Bench large model assisted perception infrared visible image fusion system, characterized by: The system includes: a module for executing the infrared-visible image fusion method for Q-Bench large model assisted perception as described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the infrared-visible image fusion method for Q-Bench large model assisted perception as described in any one of claims 1 to 7 is implemented.

Images