Visible light infrared image fusion method and device for image enhancement of region of interest

Abstract

The present invention discloses a visible light infrared image fusion method and device for enhancing the image of the region of interest, the method comprising: inputting the image group to be fused into a trained saliency detection network to obtain the image of the region of interest of the image group; inputting the image group to be fused and the image of the region of interest of the image group into the trained global twin mask autoencoder and the trained twin mask autoencoder of the region of interest respectively to obtain the fused global image and the fused image of the region of interest; the global and region of interest twin mask autoencoders are trained according to the text features of the sample image group and the standard fused image corresponding to the sample image group, and the text features are used to describe the key feature information and target information in the image; splicing the fused region of interest image and the fused global image to obtain the target fused image. The method provided by the present invention can improve the attention of the fusion process to the region of interest, thereby enhancing the fusion effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a visible light infrared image fusion method and device for enhancing an image of a region of interest. Background Art

[0002] Infrared and visible light fusion is an important technical direction in the field of computer vision and plays a vital role in image processing. This technology provides an effective solution for complex and changing monitoring and analysis tasks by cleverly integrating the thermal radiation sensitivity of infrared images and the rich appearance information of visible light images.

[0003] However, current fusion strategies generally fuse the entire image, lacking attention to the image of the region of interest in the image, which often leads to poor fusion effect of the image of the region of interest or omission of the region. At the same time, full-image fusion may introduce a large amount of background noise and irrelevant information of images in non-regions of interest, and may also cover up or interfere with important features of the image in the region of interest, thereby reducing the quality and information value of the fused image, affecting subsequent downstream task analysis, and becoming a bottleneck restricting the overall performance improvement of the system. Summary of the invention

[0004] The embodiments of the present invention provide a visible light infrared image fusion method and device for enhancing the image of the region of interest, which can solve the problem that the current image fusion solution that fuses the entire image as a whole does not highlight the region of interest.

[0005] In a first aspect, an embodiment of the present invention provides a visible light infrared image fusion method for region of interest image enhancement, the method comprising:

[0006] Inputting the image group to be fused into the trained saliency detection network to obtain the region of interest image of the image group, wherein the image group includes at least one pair of infrared image and visible light image with the same shooting content but different expression forms, and the region of interest image of the image group has a certain probability of including key information;

[0007] Input the image group to be fused and the region of interest image of the image group into the trained global twin mask autoencoder and the trained region of interest twin mask autoencoder respectively to obtain a fused global image and a fused region of interest image;

[0008] Among them, the global twin mask autoencoder and the region of interest twin mask autoencoder are trained based on the text features of the sample image group and the standard fusion image corresponding to the sample image group. The text features are used to describe the key feature information and target information in the image.

[0009] The target fused image is obtained by splicing the fused region of interest image and the fused global image.

[0010] In a second aspect, an embodiment of the present invention provides a visible light infrared image fusion device for enhancing an image of a region of interest, the device comprising:

[0011] A saliency detection module, which includes a saliency detection network. The saliency detection module is used to input the image group to be fused into the trained saliency detection network to obtain an image of the region of interest of the image group, wherein the image group includes a pair of infrared images and visible light images with the same shooting content but different expressions, and the image of the region of interest of the image group has a certain probability of including key information;

[0012] A feature extraction and fusion module, which includes a global twin mask autoencoder and a region of interest twin mask autoencoder. The feature extraction and fusion module is used to input the image group to be fused and the region of interest image of the image group into the trained global twin mask autoencoder and the trained region of interest twin mask autoencoder, respectively, to obtain a fused global image and a fused region of interest image;

[0013] Among them, the global twin mask autoencoder and the region of interest twin mask autoencoder are trained based on the text features of the sample image group and the standard fusion image corresponding to the sample image group. The text features are used to describe the key feature information and target information in the image.

[0014] The stitching and reconstruction module is used to stitch the fused region of interest image and the fused global image to obtain the target fused image.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: according to the method provided by the present invention, an image of a region of interest is detected through a saliency detection network, and the global and region of interest of the two images are fused respectively, and then the fused global image and the fused region of interest image are fused to obtain a target fused image, which can focus on the target and key features during the fusion process, reduce the influence of background noise and irrelevant details in non-interest regions on the fusion process, and enhance the fusion accuracy and effect of the target fused image; training the twin mask autoencoder through text feature can enhance the model's feature extraction capability for the region of interest, and further enhance the fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a visible light infrared image fusion method for enhancing an image of a region of interest provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of a scene for generating a target fused image provided by an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of the structure of a twin masked autoencoder provided in an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of the structure of a saliency detection network provided by an embodiment of the present invention;

[0020] Figure 5 A flowchart of a training method for a saliency network and a twin masked autoencoder provided in an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of a scenario in which a language guidance model is used to extract text features according to an embodiment of the present invention;

[0022] Figure 7 A schematic structural diagram of a visible light infrared image fusion device for region of interest image enhancement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0024] The visible light infrared image fusion method for region of interest image enhancement provided by the embodiment of the present invention can be applied to electronic devices such as mobile terminals, personal notebook computers, supercomputers, etc. The embodiment of the present invention does not impose any limitation on the specific type of electronic devices.

[0025] Figure 1 The flowchart shown is a method for implementing a visible light infrared image fusion method for enhancing an image of a region of interest provided by an embodiment of the present invention. As an example but not a limitation, the method 100 may include steps S101-S103, and each step is described below.

[0026] S101, inputting the image group to be fused into a trained saliency detection network to obtain an image of a region of interest of the image group.

[0027] Exemplarily, the image group to be fused may include at least one pair of infrared images and visible light images with the same shooting content but different presentation forms. For example, the image group to be fused may be images obtained by shooting the same scene at the same location simultaneously with an infrared camera and a visible light camera.

[0028] Exemplarily, the region of interest image has a certain probability of including key information and the probability of including key information is relatively high. For example, if the user wants to extract information related to people in the image in the subsequent image processing process, the region of interest image can be an image of the area where features related to people appear in the original image. The region of interest images of the image group include the region of interest images of the infrared image and the region of interest images of the visible light image.

[0029] For example, see Figure 2 , the region of interest image of the image group to be fused can be extracted through the saliency detection network.

[0030] S102, input the image group to be fused and the region of interest image of the image group into the trained global twin mask autoencoder and the trained region of interest twin mask autoencoder respectively to obtain a fused global image and a fused region of interest image.

[0031] Exemplarily, both the global twin mask autoencoder and the region of interest twin mask autoencoder can be trained based on the text features of the sample image group and the standard fusion image corresponding to the sample image group. The text features of the image are used to describe the key feature information and target information in the image.

[0032] Since text is naturally structured, easy to process and analyze by computers, and can carry a large amount of abstract information and specific details, the text-based modal fusion model can better understand the context and objectives of the task. Therefore, the twin masked autoencoder trained with text features can pay more attention to the features of the region of interest in the image.

[0033] In some embodiments, see Figure 2 , the twin masked autoencoder can extract the features of the image and fuse images from different sources to obtain a fused image.

[0034] Specifically, the global twin mask autoencoder can extract the features of the infrared image and the visible light image in the image group to be fused respectively, and then fuse the infrared image and the visible light image based on the extracted features to obtain the fused global image. The region of interest twin mask autoencoder can extract the features of the region of interest image of the infrared image and the region of interest image of the visible light image respectively, and then fuse the region of interest image of the infrared image and the region of interest image of the visible light image based on the extracted features to obtain the fused region of interest image.

[0035] Optionally, since the region of interest image usually includes irregular contours, before the region of interest image of the image group is input into the region of interest twin mask autoencoder, the irregular contours in the region of interest image of the image group can be padded into the smallest rectangle including the region of interest image, and then the padded rectangle is scaled to half the size of the image in the image group. For example, the size of the image input to the global twin mask autoencoder is 224×224, and the size of the image input to the region of interest twin mask autoencoder is 112×112.

[0036] Since the global twin masked autoencoder and the region of interest twin masked autoencoder have the same structure and the same operating principle, they only process different objects and have different output results. Specifically, the global twin masked autoencoder inputs the image group to be fused and outputs the fused global image, and the region of interest twin masked autoencoder inputs the region of interest image of the image group and outputs the fused region of interest image. Therefore, in the following introduction to the twin masked autoencoder, the global twin masked autoencoder and the region of interest twin masked autoencoder are not specifically distinguished, and the inputs of the two twin masked autoencoders are collectively referred to as the input image group, and the outputs are collectively referred to as the fused image.

[0037] In one possible implementation, see Figure 3 , the twin masked self-encoder 300 may include an asymmetric mask module 310, a first encoder 320, a second encoder 330, a cross self-attention module 340 and a decoder 350.

[0038] Specifically, the twin masked self-encoder 300 can be used to: randomly sample the infrared image and the visible light image in the input image group to obtain infrared image samples and visible light image samples. The infrared image samples and the visible light image samples are input to the asymmetric mask module 310, and the asymmetric mask module 310 performs asymmetric mask processing on the infrared image samples and the visible light image samples based on different mask ratios to obtain infrared image masks and visible light image masks. The infrared image mask and the visible light image mask are respectively input to the first encoder 320 and the second encoder 330 to extract the features of the infrared image and the visible light image to obtain the output result of the first encoder and the output result of the second encoder. The output result of the first encoder and the output result of the second encoder are input to the cross self-attention module 340 for two groups of cross self-attention operations to fuse the features of the visible light image and the infrared image to obtain the first group of fusion results and the second group of fusion results. The first group of fusion results and the second group of fusion results are input to the decoder 350 to obtain the output fusion image.

[0039] Exemplarily, since the infrared image is a single-channel image and the visible light image is a three-channel image, the infrared image can be copied into three channels and then input into the twin mask self-encoder 300.

[0040] In one example, if the input image group is a video of L frames, the twin masked autoencoder 300 can randomly sample 2 frames of the input image group to obtain a pair of infrared images and visible light images with the same shooting content. The distance between the sampled infrared image and the visible light image can be a random value selected from a preset inter-frame range. Then, the two frames of images are each converted into non-overlapping N×N block sequences, and each block sequence is positionally embedded and an image modality tag is attached to each block sequence to obtain infrared image sampling and visible light image sampling. In this process, there is no need to embed the image in time sequence position.

[0041] Exemplarily, the image modality mark is used to identify whether this frame of image is a visible light image or an infrared image.

[0042] For example, the image obtained by sampling the first frame can be an infrared image or a visible light image, and the second frame can be the opposite. If the image obtained by sampling the first frame is fixed as an infrared image or a visible light image, the first encoder and the second encoder will have insufficient feature extraction capabilities for the other modality. Therefore, not fixing the input positions of the infrared image and the visible light image can improve the feature extraction capabilities of the encoder.

[0043] In one example, asymmetric masking refers to using different mask ratios. , , calculate the infrared image mask and the visible light image mask.

[0044] In one example, the structures of the first encoder 320 and the second encoder 330 may be the same as the encoder structure of the Transformer network.

[0045] Exemplarily, since the input positions of the infrared image and the visible light image are not fixed, the output result of the first encoder may be an infrared image feature or a visible light image feature, while the output result of the second encoder is the opposite.

[0046] In one example, the twin masked autoencoder 300 can use the output of the first encoder 320 as a query vector, and the output of the second encoder 330 as a key vector and a value vector to perform a first set of cross self-attention operations to obtain a first set of fusion results. Then, the output of the second encoder 330 can be used as a query vector, and the output of the first encoder 320 can be used as a key vector and a value vector to perform a second set of cross self-attention operations to obtain a second set of fusion results.

[0047] By having the first encoder and the second encoder process different types of data and the cross-modal attention mechanism respectively, it is possible to promote delicate cross-modal fusion and make full use of the multimodal joint learning capability of the twin masked autoencoder to learn the respective advantages of the two modal images, thereby achieving better image fusion effects and providing a more solid foundation for subsequent visual tasks.

[0048] In one example, the decoder 350 may preliminarily reconstruct the first set of fusion results and the second set of fusion results, and further process and fuse the two sets of fusion results to ensure the integrity and consistency of image information, and obtain an output fused image.

[0049] Exemplarily, the structure of the decoder 350 may be the same as the decoder structure of the Transformer network.

[0050] S103, stitching the fused region of interest image and the fused global image to obtain a target fused image.

[0051] In one example, see Figure 2 , the fused region of interest image and the fused global image can be spliced ​​to obtain the target fused image.

[0052] For example, the fused region of interest image and the fused global image can be spliced ​​first, and then sent to a 1×1 convolution layer for final reconstruction. The 1×1 convolution layer can integrate the spliced ​​features and generate the final target fused image. Through two sets of cross self-attention operations, the complementary information of the infrared image and the visible light image can be fully utilized to generate a target fused image with higher quality and richer details.

[0053] According to the method provided by the present invention, an image of a region of interest is detected through a saliency detection network, and the global and region of interest of the two images are fused respectively, and then the fused global image and the fused region of interest image are fused to obtain a target fused image. During the fusion process, the target and key features can be focused on, the influence of background noise and irrelevant details in non-interested regions on the fusion process can be reduced, and the fusion accuracy and effect of the target fused image can be enhanced; through text feature training and twin mask autoencoders, the model's feature extraction capability for the region of interest can be enhanced, and the fusion effect can be further enhanced.

[0054] Figure 4 FIG. 1 is a schematic diagram of the structure of a saliency detection network provided by an embodiment of the present invention.

[0055] In one possible implementation, see Figure 4The saliency detection network 400 may include a cascaded first saliency encoder 411, a second saliency encoder 412, a third saliency encoder 413, and a fourth saliency encoder 414, a first saliency decoder 421, a second saliency decoder 422, a third saliency decoder 423, a fourth saliency decoder 424, a global context feature extraction module 430, and a context refinement module 440.

[0056] Specifically, see Figure 4 , the images in the image group to be fused can be sequentially input into the saliency detection network 400, and the first saliency encoder 411 encodes them to extract the saliency features of the images, and then the output of the previous encoder is used as the input of the next saliency encoder, and the saliency features therein are sequentially extracted to obtain feature maps of different sizes. Each level of saliency encoder inputs the extracted feature maps into the global context feature extraction module 430 and its corresponding saliency decoder. The global context feature extraction module 430 can perform saliency prediction on the image group to be fused based on the feature map to extract the global context features in the image, and output the global context features to the context refinement module 440. The saliency decoder decodes the input features to obtain decoder features and then inputs the decoder features into the context refinement module 440. The context refinement module 440 extracts the region of interest image of the image group to be fused based on the decoder features and the global context features.

[0057] Exemplarily, the feature map output by each level of encoder has a different size, and the output sizes of the first saliency encoder 411, the second saliency encoder 412, the third saliency encoder 413 and the third saliency encoder 414 are 56×56, 28×28, 14×14 and 7×7 respectively.

[0058] In one example, a Transformer network can be used as the backbone of the saliency encoder. Furthermore, in order to achieve more lightweight calculations, a pyramid vision Transformer-V2 model can be used as the basic architecture of the saliency encoder.

[0059] In one example, the global context feature extraction module 430 may perform saliency prediction on each image block into which each image in the image group to be fused is divided, and obtain the probability that each image block includes a key feature or a target. , and at the same time obtain the global context features of each image during the saliency prediction process.

[0060] Exemplary,significance prediction process It can be expressed as:

[0061] ;

[0062] in, It is obtained by fusing the feature maps output by multiple saliency encoders. represents a tensor operation, Indicates that the global context feature extraction module has a feature extractor with a Transformer network structure. is a multi-layer perceptron, Represents the activation function.

[0063] Specifically, the feature map output by the mth saliency encoder and the m-1th level fusion result are concatenated, and then fused through a convolution operation to obtain the mth level fusion result. m=2,3,4. The first level fusion result is the feature map output by the first saliency encoder, and the fourth level fusion result is .

[0064] Exemplarily, the feature extractor inside the global context feature extraction module outputs the global context feature.

[0065] In one example, in order to match the dimensions of the global context features and the feature maps output by each saliency encoder, the context refinement module 440 may use different scaling factors to reorder the pixels of the global context features. The reordered global context features and the decoder features are fused through the first feature fusion layer Perform the first stage of prediction to obtain a preliminary area of ​​interest , such as complex patterns, boundaries or small objects. Based on this, the second feature fusion layer Continue to extract the features as the refined attention map , to guide the focus during the second prediction Then based on the third feature fusion layer and a Transformer network Perform feature mapping on the refined attention map to make a second prediction on the region of interest to obtain the region of interest image of the image group to be fused .

[0066] Exemplary decoder features I It is also a feature map. , C is the total number of feature channels.

[0067] For example, pixel rearrangement is to transform an image from Rearrange to a multi-channel low-resolution image ,in, , are the height and width of the decoder feature, and r is the scaling factor. Compared with the traditional downsampling method, pixel rearrangement does not lose any pixels, so that the integrity of the image structure is preserved.

[0068] Specifically, the pixel rearrangement process can be expressed as:

[0069] ;

[0070] in, is the decoder feature to be rearranged, , , are the pixel coordinates and channel index of the feature in the high-resolution space, respectively. represents the modulo operation, Represents a pixel rearrangement operation.

[0071] Exemplarily, the refined attention map can satisfy the following formula:

[0072] ;

[0073] in, is the encoder feature, is a multiplication operation, Represents element-wise multiplication.

[0074] Exemplarily, the region of interest of the image group to be fused may satisfy: .

[0075] Figure 5 The flowchart of the implementation of a method for training a saliency network and a twin masked autoencoder provided by an embodiment of the present invention is shown. Method 500 is used to train the above-mentioned saliency network, global twin masked autoencoder and region of interest twin masked autoencoder. Method 500 may include steps S501-S508, and each step is described below.

[0076] S501, inputting the sample image group into the trained saliency detection network to obtain the region of interest image of the sample image group.

[0077] Exemplarily, the saliency detection network is the saliency detection network 400 described above.

[0078] Similarly, the sample image group may include infrared sample images and visible light sample images with the same shooting content but different expression forms.

[0079] S502, input the sample image group and the region of interest image of the sample image group into the global twin mask autoencoder and the region of interest twin mask autoencoder respectively, to obtain the global image features of the sample image group, the fused sample global image and the region of interest image features of the sample image group, and the fused sample region of interest image.

[0080] Exemplarily, the outputs of the first encoder and the second encoder of the global twin mask autoencoder are global image features of the sample image group, which may include global features of visible light sample images and global features of infrared sample images. The decoder in the global twin mask autoencoder can output a fused sample global image.

[0081] Exemplarily, the outputs of the first encoder and the second encoder of the region of interest twin mask autoencoder are the region of interest image features of the sample image group, which may include the region of interest features of the visible light sample image and the region of interest features of the infrared sample image. The decoder in the region of interest twin mask autoencoder can output the fused sample region of interest image.

[0082] S503, splicing the fused sample global image and the fused sample region of interest image to obtain a target fused sample image.

[0083] Exemplarily, the principles of steps S501-S503 in method 500 are similar to those of method S101-S103, except that the images input to each network model are different. The input in method 100 is a group of images to be fused, while the input in method 500 is a group of sample images. For details, please refer to the relevant description in method 100, which will not be repeated here.

[0084] S504, inputting the sample image group, the region of interest image of the sample image group and the standard fused image corresponding to the sample image group into the language guidance model to obtain infrared image text features, visible light image text features, region of interest text features and fused image text features.

[0085] Exemplarily, the standard fused image may be an image that meets the fusion quality requirement after a pair of visible light sample images and infrared sample images in the sample image group are fused.

[0086] In one example, see Figure 6 , the language-guided model can include an image-text extraction module, such as a ChatGPT language model; a text encoder, such as a Transformer network.

[0087] Specifically, the image text extraction module can generate detailed text descriptions based on the sample image group consisting of infrared sample images and visible light sample images, the image of the region of interest of the sample image group, and the standard fusion image to obtain infrared image text description, visible light image text description, region of interest text description, and fusion image text description. These text descriptions are then input into the text encoder, which can obtain feature representations rich in semantic information from these text descriptions, namely, infrared image text features. , visible light image text features , Interesting text features , Fusion of image and text features .

[0088] Example, text description of region of interest Can accurately describe the appearance characteristics and location information of the core elements in the image; infrared image text description It can deeply analyze the temperature distribution and high temperature area under the infrared perspective; text description of visible light image It can explain the structure and details of the scene captured under conventional vision; it can also integrate image and text description The overall picture of the image combining infrared information and visible light information can be comprehensively summarized.

[0089] S505, comparing the global image features and infrared image text features and visible light image text features of the sample image group, comparing the fused sample global image, the fused sample region of interest image and the fused image text features, comparing the region of interest image features and the region of interest text features, and obtaining the comparative learning loss.

[0090] In an example, the global features of the infrared sample image and the text features of the infrared image can be used as a set of contrast features, the global features of the visible light sample image and the text features of the visible light image can be used as a set of contrast features, the fused sample global image and the text features of the fused image can be used as a set of contrast features, the fused sample region of interest image and the text features of the fused image can be used as a set of contrast features, the region of interest features of the visible light sample image and the text features of the region of interest can be used as a set of contrast features, and the region of interest features of the infrared sample image and the text features of the region of interest can be used as a set of contrast features; and the contrast learning loss is determined based on these contrast features.

[0091] Exemplarily, contrastive learning loss can satisfy:

[0092] ;

[0093] in, For contrastive learning loss, For the A sample of queries, For the The query sample corresponds to the Positive samples, For the The query sample corresponds to the negative samples, is the temperature parameter, represents the similarity measure between two feature vectors, K is the total number of negative samples.

[0094] Exemplarily, each feature can have multiple ones, and one of the features can be selected from a group of comparison features as a query sample, and positive samples related to it can be queried from another feature, and the remaining ones in the other feature are negative samples corresponding to the query sample.

[0095] For example, a region of interest feature of an infrared sample image including a target person can be selected as a query sample, and the region of interest text features used to describe the target person can be queried from the region of interest text features as positive samples, and the remaining region of interest text features describing the surrounding scene features are negative samples.

[0096] S506, determining the twin encoder loss according to the visible light image text features, the infrared image text features, the fused image text features, and the sample image group.

[0097] In one possible implementation, the twin encoder loss can be composed of structural similarity loss, texture loss and intensity loss.

[0098] Exemplarily, the twin encoder loss may satisfy the following formula:

[0099] ;

[0100] in, is the twin encoder loss, , , They are structural similarity loss weight, texture loss weight and intensity loss weight, respectively. , , They are structural similarity loss, texture loss, and intensity loss respectively.

[0101] In one example, the intensity loss can be used to guide the Siamese mask autoencoder to capture moderate intensity information when fusion.

[0102] Exemplarily, the strength loss may satisfy:

[0103] ;

[0104] in, is the strength loss, is the length of the image in the sample image group, is the width of the image in the sample image group, is the Sobel gradient operator, To fuse image and text features, is the visible light image text feature, is the infrared image text feature, represents the element-wise mean aggregation operation, represents the L1 norm.

[0105] In one example, the structural similarity loss can reflect the distortion of the image from three aspects: illumination, contrast, and structure. Training the twin mask autoencoder with the structural similarity loss can constrain the structural similarity between the target fusion image and the sample image group.

[0106] Exemplarily, the structural similarity loss can satisfy:

[0107] ;

[0108] in, is the structural similarity loss, represents the structural similarity operation, , are two weight values, is the infrared sample image in the sample image group, is the visible light sample image in the sample image group, To fuse image and text features.

[0109] In one example, training a Siamese mask autoencoder with a texture loss can guide it to preserve as much texture details of an image as possible.

[0110] Exemplarily, texture loss may satisfy:

[0111] ;

[0112] in, is the texture loss, is the length of the image in the sample image group, W is the width of the image in the sample image group, is the Sobel gradient operator, To fuse image and text features, Indicates the absolute value operation. represents the L1 norm.

[0113] S507, train the global twin mask autoencoder and the region-of-interest twin mask autoencoder based on the contrastive learning loss and the twin encoder loss to obtain a trained global twin mask autoencoder and a trained region-of-interest twin mask autoencoder.

[0114] Exemplarily, the language guidance model can also be trained by contrastive learning loss.

[0115] Since the contrastive learning loss and the twin encoder loss in the present invention are both determined based on the text features of the sample image group extracted by the language guidance module, training the twin mask autoencoder based on these two losses can supplement the key elements or details in the image with richer semantic information, and enhance the model's expressiveness and depth of understanding of the region of interest; thereby enhancing the model's ability to extract features of the image in the region of interest, achieving more accurate image fusion, and improving the performance and accuracy of the model in target detection, scene parsing and other downstream tasks.

[0116] Figure 7 The structure diagram of a visible light infrared image fusion device for enhancing an image of a region of interest provided by an embodiment of the present invention is shown. As an example but not a limitation, the device 700 may include:

[0117] A saliency detection module 710, the saliency detection module includes a saliency detection network, and the saliency detection module is used to input the image group to be fused into the trained saliency detection network to obtain an image of a region of interest of the image group, wherein the image group includes a pair of infrared images and visible light images with the same shooting content but different expressions, and the image of the region of interest of the image group has a certain probability of including the shooting target;

[0118] The feature extraction and fusion module 720 includes a global twin mask autoencoder and a region of interest twin mask autoencoder. The feature extraction and fusion module 720 is used to input the image group to be fused and the region of interest image of the image group into the trained global twin mask autoencoder and the trained region of interest twin mask autoencoder, respectively, to obtain a fused global image and a fused region of interest image;

[0119] The stitching and reconstruction module 730 is used to stitch the fused region of interest image and the fused global image to obtain a target fused image.

[0120] Exemplarily, the global twin mask autoencoder and the region of interest twin mask autoencoder are trained based on the text features of the sample image group and the standard fusion image corresponding to the sample image group, and the text features are used to describe the key feature information and target information in the image.

[0121] According to the device provided by the present invention, an image of a region of interest is detected through a saliency detection network, and the global and region of interest of the two images are fused respectively, and then the fused global image and the fused region of interest image are fused to obtain a target fused image. During the fusion process, the target and key features can be focused on, the influence of background noise and irrelevant details in non-interested regions on the fusion process can be reduced, and the fusion accuracy and effect of the target fused image can be enhanced; training the twin mask autoencoder through text features can enhance the model's feature extraction capability for the region of interest, and further enhance the fusion effect.

[0122] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0123] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

Claims

1. A visible light infrared image fusion method for region of interest image enhancement, characterized in that: include: Inputting the image group to be fused into the trained saliency detection network to obtain the region of interest image of the image group, wherein the image group includes at least one pair of infrared image and visible light image with the same shooting content but different expression forms, and the region of interest image of the image group has a certain probability of including key information; Inputting the image group to be fused and the region of interest image of the image group into the trained global twin mask autoencoder and the trained region of interest twin mask autoencoder respectively, to obtain a fused global image and a fused region of interest image; The global twin mask autoencoder and the region of interest twin mask autoencoder are trained based on the text features of the sample image group and the standard fusion image corresponding to the sample image group, and the text features are used to describe the key feature information and target information in the image; splicing the fused region of interest image and the fused global image to obtain a target fused image; Among them, the training method of the global twin mask autoencoder and the region of interest twin mask autoencoder includes: Inputting the sample image group into the trained saliency detection network to obtain the region of interest image of the sample image group; The sample image group and the region of interest image of the sample image group are respectively input into the global twin mask autoencoder and the region of interest twin mask autoencoder to obtain the global image features of the sample image group, the fused sample global image, the region of interest image features of the sample image group, and the fused sample region of interest image; wherein the global image features of the sample image group include global features of visible light sample images and global features of infrared sample images, and the region of interest image features of the sample image group include region of interest features of visible light sample images and region of interest features of infrared sample images; Splicing the fused sample global image and the fused sample region of interest image to obtain a target fused sample image; The sample image group, the region of interest image of the sample image group and the standard fused image corresponding to the sample image group are input into the language guidance model to obtain the infrared image text features, visible light image text features, region of interest text features and fused image text features of the sample image group; wherein the standard fused image is an image that meets the fusion quality requirements after the visible light sample image and the infrared sample image in the sample image group are fused; Compare the global features of the infrared sample image and the text features of the infrared image, compare the global features of the visible light sample image and the text features of the visible light image, compare the fused sample global image and the text features of the fused image, compare the fused sample region of interest image and the text features of the fused image, compare the region of interest features of the visible light sample image and the text features of the region of interest, and compare the region of interest features of the infrared sample image and the text features of the region of interest to obtain a comparative learning loss; Determine a twin encoder loss according to the visible light image text feature, the infrared image text feature, the fused image text feature, and the sample image group; The global twin mask autoencoder and the region-of-interest twin mask autoencoder are trained based on the contrastive learning loss and the twin encoder loss to obtain the trained global twin mask autoencoder and the trained region-of-interest twin mask autoencoder.

2. The method according to claim 1, characterized in that The global twin mask autoencoder and the region of interest twin mask autoencoder with the same structure both include: an asymmetric mask module, a first encoder, a second encoder, a cross attention module, and a decoder; The global twin mask autoencoder / the region of interest twin mask autoencoder is used for: Randomly sampling the image group to be fused / the image of the region of interest of the image group to obtain infrared image sampling and visible light image sampling; Inputting the infrared image samples and the visible light image samples into the asymmetric mask module, and allowing the asymmetric mask module to perform asymmetric mask processing on the infrared image samples and the visible light image samples based on different mask ratios to obtain an infrared image mask and a visible light image mask; Inputting the infrared image mask and the visible light image mask into the first encoder and the second encoder respectively to extract features of the infrared image and the visible light image, and obtaining an output result of the first encoder and an output result of the second encoder; Inputting the output result of the first encoder and the output result of the second encoder into the cross attention module to perform two groups of cross self-attention operations on the output result of the first encoder and the output result of the second encoder, and fusing the features of the infrared image and the features of the visible light image to obtain a first group of fusion results and a second group of fusion results; The first group of fusion results and the second group of fusion results are input into the decoder for reconstruction to obtain the fused global image / the fused region of interest image.

3. The method according to claim 2, characterized in that The performing two groups of cross self-attention operations on the output results of the first encoder and the output results of the second encoder to fuse the features of the infrared image and the features of the visible light image to obtain a first group of fusion results and a second group of fusion results, including: Using the output result of the first encoder as the query vector, and using the output result of the second encoder as the key vector and the value vector to perform a first set of cross self-attention operations to obtain the first set of fusion results; The output result of the second encoder is used as the query vector, and the output result of the first encoder is used as the key vector and the value vector to perform a second set of cross self-attention operations to obtain the second set of fusion results.

4. The method according to claim 1, characterized in that: The contrastive learning loss satisfies the following formula: Among them, L InfoNCE is the contrastive learning loss, z i is the i-th query sample, is the jth positive sample corresponding to the i-th query sample, z k is the kth negative sample corresponding to the ith query sample, τ is the temperature parameter, sim(·) represents the similarity measure between two feature vectors, and K is the total number of negative samples.

5. The method according to claim 1, characterized in that The twin encoder loss consists of structural similarity loss, texture loss and intensity loss.

6. The method according to claim 5, characterized in that The strength loss satisfies the following formula: Among them, L intensity is the intensity loss, H2 is the length of the image in the sample image group, W2 is the width of the image in the sample image group, is the Sobel gradient operator, I F is the fused image text feature, I RGB is the visible light image text feature, I IR is the infrared image text feature, mean(·) represents the element-by-element mean aggregation operation, and ||·||1 represents the L1 norm.

7. The method according to claim 5, characterized in that The structural similarity loss satisfies the following formula: L ssim =w1·(1-ssim(I F ,I1))+w2·(1-ssim(I F ,I2)); Among them, L ssim is the structural similarity loss, ssim(·) represents the structural similarity operation, w1 and w2 are two weight values, I1 is the infrared sample image in the sample image group, I2 is the visible light sample image in the sample image group, and I F is the fused image text feature.

8. The method according to claim 5, characterized in that The texture loss function is expressed as: Among them, L texture is the texture loss function, H2 is the length of the image in the sample image group, W2 is the width of the image in the sample image group, is the Sobel gradient operator, I F is the fused image text feature, I1 is the infrared sample image in the sample image group, I2 is the visible light sample image in the sample image group, |·| represents the absolute value operation, and ||·||1 represents the L1 norm.

9. A visible light infrared image fusion device for enhancing the image of the region of interest, characterized in that: include: A saliency detection module, the saliency detection module comprising a saliency detection network, the saliency detection module being used to input the image group to be fused into the trained saliency detection network to obtain an image of a region of interest of the image group, wherein the image group comprises a pair of infrared images and visible light images having the same shooting content but different expression forms, and the image of the region of interest of the image group has a certain probability of including key information; A feature extraction and fusion module, wherein the feature extraction and fusion module comprises a global twin mask autoencoder and a region of interest twin mask autoencoder, and the feature extraction and fusion module is used to input the image group to be fused and the region of interest image of the image group into the trained global twin mask autoencoder and the trained region of interest twin mask autoencoder, respectively, to obtain a fused global image and a fused region of interest image; The global twin mask autoencoder and the region of interest twin mask autoencoder are trained based on the text features of the sample image group and the standard fusion image corresponding to the sample image group, and the text features are used to describe the key feature information and target information in the image; A splicing and reconstruction module, wherein the splicing and reconstruction module is used to splice the fused region of interest image and the fused global image to obtain a target fused image; Among them, the training method of the global twin mask autoencoder and the region of interest twin mask autoencoder includes: Inputting the sample image group into the trained saliency detection network to obtain the region of interest image of the sample image group; The sample image group and the region of interest image of the sample image group are respectively input into the global twin mask autoencoder and the region of interest twin mask autoencoder to obtain the global image features of the sample image group, the fused sample global image, the region of interest image features of the sample image group, and the fused sample region of interest image; wherein the global image features of the sample image group include global features of visible light sample images and global features of infrared sample images, and the region of interest image features of the sample image group include region of interest features of visible light sample images and region of interest features of infrared sample images; Splicing the fused sample global image and the fused sample region of interest image to obtain a target fused sample image; The sample image group, the region of interest image of the sample image group and the standard fused image corresponding to the sample image group are input into the language guidance model to obtain the infrared image text features, visible light image text features, region of interest text features and fused image text features of the sample image group; wherein the standard fused image is an image that meets the fusion quality requirements after the visible light sample image and the infrared sample image in the sample image group are fused; Compare the global features of the infrared sample image and the text features of the infrared image, compare the global features of the visible light sample image and the text features of the visible light image, compare the fused sample global image and the text features of the fused image, compare the fused sample region of interest image and the text features of the fused image, compare the region of interest features of the visible light sample image and the text features of the region of interest, and compare the region of interest features of the infrared sample image and the text features of the region of interest to obtain a comparative learning loss; Determine a twin encoder loss according to the visible light image text feature, the infrared image text feature, the fused image text feature, and the sample image group; The global twin mask autoencoder and the region-of-interest twin mask autoencoder are trained based on the contrastive learning loss and the twin encoder loss to obtain the trained global twin mask autoencoder and the trained region-of-interest twin mask autoencoder.

Images