Infrared image segmentation method and device based on visible light image enhancement

Abstract

The present invention provides an infrared image segmentation method and device based on visible light image enhancement, the method comprising: inputting the infrared image to be segmented into an infrared image encoder in an infrared image segmentation model, obtaining the features of the infrared image to be segmented output by the infrared image encoder; determining a similarity graph between the features of the infrared image to be segmented and the features of a reference visible light image of a preset target category, the features of the reference visible light image being obtained in advance based on the visible light image encoder in the infrared image segmentation model; inputting the features of the infrared image to be segmented and the similarity graph into an infrared image decoder in the infrared image segmentation model, obtaining the predicted segmentation result of the infrared image to be segmented output by the infrared image decoder. The present invention only requires a certain amount of labeled infrared images and a small amount of visible light images of preset target categories to train the model, and only needs to provide infrared images as input in the test phase, thereby reducing the test complexity and improving the test efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an infrared image segmentation method and device based on visible light image enhancement. Background Art

[0002] Compared with visible light imaging, infrared imaging technology has natural advantages in adverse weather conditions (rain, snow, fog, haze, etc.) and adverse imaging conditions (reflection, dark light, etc.). However, due to the low resolution, weak texture information and lack of color information of infrared images, infrared image segmentation is extremely challenging.

[0003] In order to further improve the accuracy of infrared image segmentation, two new research ideas have emerged in the existing technology. One method is to use visible light images as an additional modality and fuse the features of infrared and visible light modalities at different layers of the neural network. This type of method requires simultaneous input of infrared and visible light images during the testing phase, and requires strict registration between them. During the testing phase, visible light and infrared images need to be provided at the same time, and cannot be used in situations where only infrared images are available.

[0004] Another method only uses information from visible light images during training and transfers knowledge from the visible light domain to the infrared domain. This type of method does not require visible light images during the test phase, but requires a large amount of unlabeled infrared data. However, this requirement is difficult to meet for some application scenarios (such as military), and requires a certain amount of accurately registered infrared and visible light data, which is also difficult to meet in most cases. Summary of the invention

[0005] The present invention provides an infrared image segmentation method and device based on visible light image enhancement, which are used to solve the defects in the prior art that visible light and infrared images need to be provided simultaneously in the test phase, and a large amount of unlabeled infrared data or a certain amount of accurately aligned infrared and visible light data is needed in the training phase. The method and device can realize accurate segmentation of infrared images with only a certain amount of labeled infrared images and a small amount of reference visible light images of preset target categories.

[0006] The present invention provides an infrared image segmentation method based on visible light image enhancement, comprising:

[0007] Inputting the infrared image to be segmented into an infrared image encoder in the infrared image segmentation model, and obtaining features of the infrared image to be segmented output by the infrared image encoder;

[0008] Determine a similarity graph between features of the infrared image to be segmented and features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are pre-obtained based on a visible light image encoder in the infrared image segmentation model;

[0009] Inputting the features of the infrared image to be segmented and the similarity map into an infrared image decoder in the infrared image segmentation model, and obtaining a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder;

[0010] The infrared image segmentation model is obtained by pre-training the infrared image encoder and the visible light image encoder using the first training sample and then training them using the second training sample;

[0011] The first training sample includes a first infrared image sample and a visible light image sample paired with the first infrared image sample;

[0012] The second training sample includes a second infrared image sample, a segmentation result annotation of the second infrared image sample, and the reference visible light image.

[0013] According to an infrared image segmentation method based on visible light image enhancement provided by the present invention, before inputting the infrared image to be segmented into an infrared image encoder in an infrared image segmentation model and obtaining the features of the infrared image to be segmented output by the infrared image encoder, the method further includes:

[0014] Inputting the first infrared image sample into the infrared image encoder to obtain features of the first infrared image sample output by the infrared image encoder;

[0015] Inputting the visible light image sample into the visible light image encoder to obtain features of the visible light image sample output by the visible light image encoder;

[0016] determining a loss between a feature of the first infrared image sample and a feature of the visible light image sample;

[0017] Parameters of the infrared image encoder and the visible light image encoder are adjusted according to the loss.

[0018] According to an infrared image segmentation method based on visible light image enhancement provided by the present invention, the determining the loss between the feature of the first infrared image sample and the feature of the visible light image sample includes:

[0019] Determine a dot product between a feature of a current first infrared image sample and a feature of a visible light image sample corresponding to the current first infrared image sample;

[0020] Determine a first sum of dot products between the feature of the current first infrared image sample and the features of the visible light image samples corresponding to each first infrared image sample;

[0021] determining a first loss between a feature of the first infrared image sample and a feature of the visible light image sample based on the dot product and the first sum;

[0022] Determine a second sum of dot products between the feature of the visible light image sample corresponding to the current first infrared image sample and the features of each of the first infrared image samples;

[0023] determining a second loss between a feature of the first infrared image sample and a feature of the visible light image sample based on the dot product and the second sum;

[0024] The loss between the feature of the current first infrared image sample and the feature of the visible light image sample corresponding to the current first infrared image sample is determined according to the first loss and / or the second loss.

[0025] According to an infrared image segmentation method based on visible light image enhancement provided by the present invention, the first infrared image sample is input into the infrared image encoder to obtain the features of the first infrared image sample output by the infrared image encoder; the visible light image sample is input into the visible light image encoder to obtain the features of the visible light image sample output by the visible light image encoder, including:

[0026] Fusing the first infrared image sample and the visible light image sample to obtain a fused image;

[0027] Detecting a region of interest from the fused image and obtaining a target suggestion box of the region of interest;

[0028] Cutting out corresponding image regions from the first infrared image sample and the visible light image sample according to each target suggestion frame;

[0029] Inputting the image area corresponding to the first infrared image sample into the infrared image encoder to obtain the features of the first infrared image sample output by the infrared image encoder;

[0030] The image area corresponding to the visible light image sample is input into the visible light image encoder to obtain the feature of the visible light image sample output by the visible light image encoder.

[0031] According to an infrared image segmentation method based on visible light image enhancement provided by the present invention, the first infrared image sample and the visible light image sample are fused to obtain a fused image, including:

[0032] When the brightness of the visible light image sample is less than a preset threshold, performing contrast enhancement on the visible light image sample;

[0033] The first infrared image sample and the contrast-enhanced visible light image sample are fused to obtain the fused image.

[0034] According to an infrared image segmentation method based on visible light image enhancement provided by the present invention, before inputting the infrared image to be segmented into an infrared image encoder in an infrared image segmentation model and obtaining the features of the infrared image to be segmented output by the infrared image encoder, the method further includes:

[0035] Inputting the second infrared image sample into the infrared image encoder to obtain features of the second infrared image sample output by the infrared image encoder;

[0036] Inputting the reference visible light images of each preset target category into the visible light image encoder, and obtaining the features of the reference visible light images of each preset target category output by the visible light image encoder;

[0037] Determining a similarity graph between features of the second infrared image sample and features of the reference visible light images of each preset target category;

[0038] Inputting the feature of the second infrared image sample and the similarity map into the infrared image decoder to obtain a predicted segmentation result of the second infrared image sample output by the infrared image decoder;

[0039] determining a segmentation loss of the second infrared image sample according to a probability that the predicted segmentation result of each pixel in the second infrared image sample is labeled as the segmentation result of each pixel;

[0040] determining, according to a probability that the similarity graph corresponding to each pixel in the second infrared image sample is labeled as the segmentation result, an alignment loss between features of the second infrared image sample and features of the reference visible light images of each preset target category;

[0041] According to the segmentation loss and / or alignment loss, the parameters of the infrared image segmentation model are adjusted.

[0042] According to an infrared image segmentation method based on visible light image enhancement provided by the present invention, the determining of a similarity graph between features of the second infrared image sample and features of reference visible light images of each preset target category includes:

[0043] generating each vector of the second infrared image sample according to the pixel value at the same position in the feature of the second infrared image sample;

[0044] Obtaining a representation vector of each preset target category according to the features of the reference visible light image of each preset target category;

[0045] Determining the similarity between each vector of the second infrared image sample and the representative vector of each preset target category;

[0046] According to the similarity between each vector of the second infrared image sample and the representation vector of each preset target category, a similarity map between the feature of the second infrared image sample and the feature of the reference visible light image of each preset target category is determined.

[0047] The present invention also provides an infrared image segmentation device based on visible light image enhancement, comprising:

[0048] An encoding module, used for inputting the infrared image to be segmented into an infrared image encoder in the infrared image segmentation model, and obtaining the features of the infrared image to be segmented output by the infrared image encoder;

[0049] an enhancement module, configured to determine a similarity graph between features of the infrared image to be segmented and features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are obtained in advance based on a visible light image encoder in the infrared image segmentation model;

[0050] A segmentation module, used for inputting the features of the infrared image to be segmented and the similarity map into the infrared image decoder in the infrared image segmentation model, and obtaining a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder;

[0051] The infrared image segmentation model is obtained by pre-training the infrared image encoder and the visible light image encoder using the first training sample and then training them using the second training sample;

[0052] The first training sample includes a first infrared image sample and a visible light image sample paired with the first infrared image sample;

[0053] The second training sample includes a second infrared image sample, a segmentation result annotation of the second infrared image sample, and the reference visible light image.

[0054] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the infrared image segmentation method based on visible light image enhancement as described above is implemented.

[0055] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the infrared image segmentation method based on visible light image enhancement as described in any one of the above is implemented.

[0056] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the infrared image segmentation method based on visible light image enhancement as described above is implemented.

[0057] The infrared image segmentation method and device based on visible light image enhancement provided by the present invention, by using paired first infrared image samples and visible light image samples to pre-train the infrared image encoder and the visible light image encoder in the infrared image segmentation model before training the infrared image segmentation model, for specific infrared image applications, only a certain amount of labeled infrared images and a small amount of visible light images of preset target categories are required for model training to achieve accurate segmentation of infrared images, and the scope of application is wider; in the test stage, only the infrared image to be segmented needs to be provided as input, and the category representation of the infrared image to be segmented needs to be calculated in advance, so as to reduce the test complexity and improve the test efficiency while ensuring the accurate segmentation of the infrared image. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 This is one of the flow charts of the infrared image segmentation method based on visible light image enhancement provided by the present invention;

[0060] Figure 2 It is a structural schematic diagram of an infrared image segmentation model in the infrared image segmentation method based on visible light image enhancement provided by the present invention;

[0061] Figure 3 This is the second flow chart of the infrared image segmentation method based on visible light image enhancement provided by the present invention;

[0062] Figure 4 It is a schematic diagram of the structure of a pre-trained model in the infrared image segmentation method based on visible light image enhancement provided by the present invention;

[0063] Figure 5 It is a schematic diagram of the process of constructing infrared and visible light paired data sets in the infrared image segmentation method based on visible light image enhancement provided by the present invention;

[0064] Figure 6 It is a structural schematic diagram of an infrared image segmentation device based on visible light image enhancement provided by the present invention;

[0065] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Combine the following Figure 1 The present invention describes an infrared image segmentation method based on visible light image enhancement, comprising:

[0068] Step 101, inputting the infrared image to be segmented into an infrared image encoder in an infrared image segmentation model, and obtaining features of the infrared image to be segmented output by the infrared image encoder;

[0069] like Figure 2 As shown, the infrared image segmentation model includes an infrared image encoder, a visible light image encoder and an infrared image decoder.

[0070] The infrared image encoder and the visible light image encoder may be a visual transformer network ViT-B / 16 with 12 layers, and the output image block size is 16×16.

[0071] In the pre-training stage of infrared image segmentation model training, the parameters of infrared image encoder and visible light image encoder can be initialized using ViT (Vision Transformer) pre-trained on ImageNet. Adam optimizer is used for optimization, with an initial learning rate of 1e-5 and 100 iterations.

[0072] When training the infrared image segmentation model, the visible light image encoder and the infrared image encoder are respectively initialized using the corresponding model parameters obtained in the pre-training stage.

[0073] The infrared image to be segmented is an infrared image that needs to be segmented. In the testing phase of the infrared image segmentation model, the infrared image to be segmented is input into the infrared image encoder for feature extraction.

[0074] Step 102, determining a similarity graph between features of the infrared image to be segmented and features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are obtained in advance based on a visible light image encoder in the infrared image segmentation model;

[0075] The preset target category includes the category of the target in the infrared image to be segmented. The reference visible light image of the preset target category is input into the infrared image segmentation model for feature extraction to obtain the features of the reference visible light image.

[0076] In fact, the same category representation is used for infrared images of a specific domain, that is, the features of the reference visible light image of the preset target category. By pre-computing the category representation, the visible light image can be omitted during the test phase. In this way, the inference complexity at test time is almost not increased.

[0077] The similarity between the pixels at the same position in the features of the infrared image to be segmented and the features of the reference visible light image is determined to obtain a similarity map.

[0078] The cross attention between the features of the infrared image to be segmented and the features of the reference visible light image may also be calculated, and the features after the cross attention may be used as the input of the infrared image decoder.

[0079] Step 103, inputting the features of the infrared image to be segmented and the similarity map into an infrared image decoder in the infrared image segmentation model, and obtaining a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder;

[0080] The similarity map is input into the infrared image decoder as additional information. The similarity map can reflect the semantic information of the target in the infrared image to be segmented and provide additional clues of the target location and category.

[0081] In order to utilize the useful information in the similarity graph, the similarity graph and the features of the infrared image to be segmented can be simultaneously input into the infrared image decoder. Specifically, the similarity graph and the features of the infrared image to be segmented can be spliced ​​in the channel dimension and then input into the infrared image decoder, so that the infrared image decoder outputs a more accurate predicted segmentation result of the infrared image to be segmented.

[0082] The infrared image segmentation model is obtained by pre-training the infrared image encoder and the visible light image encoder using the first training sample and then training them using the second training sample;

[0083] The first training sample includes a first infrared image sample and a visible light image sample paired with the first infrared image sample;

[0084] The second training sample includes a second infrared image sample, a segmentation result annotation of the second infrared image sample, and the reference visible light image.

[0085] Before using the infrared image segmentation model to segment the infrared image to be segmented, the infrared image segmentation model is trained in two stages. In the first training stage, the infrared image encoder and the visible light image encoder are pre-trained. The goal of the pre-training is, on the one hand, to learn the infrared image encoder, which will serve as the backbone network of the infrared image segmentation model; on the other hand, to learn the visible light image encoder and align the infrared features with the visible light features. In this aligned feature space, the infrared features and the visible light features can directly perform similarity calculations to obtain a similarity graph. The aligned infrared image encoder and visible light image encoder will encode the pairing knowledge of infrared and visible light. This pairing knowledge will be used in the image segmentation model to enhance the segmentation performance of infrared images.

[0086] In the second training stage, the pre-trained infrared and visible light pairing knowledge is injected into the infrared image segmentation model to improve the infrared feature extraction capability and thus improve the accuracy of infrared image segmentation.

[0087] The idea of ​​the infrared image segmentation method based on visible light image enhancement proposed in this embodiment is to calculate the similarity between the features of the infrared image to be segmented and the features of the reference visible light image at the same pixel position in the space where the infrared and visible light are aligned, thereby obtaining a similarity map, which is input into the infrared image decoder as additional information. Since the alignment of the visible light and infrared features is performed before, the similarity map can reflect the semantic information of the target and provide additional clues to the target position and category.

[0088] The complete flow chart of the infrared image segmentation method based on visible light image enhancement is as follows: Figure 3 As shown, the following steps are included:

[0089] Step 301, collecting publicly available paired first infrared image samples and visible light image samples from the Internet for comparison pre-training between infrared and visible light;

[0090] Step 302, using the large-scale paired first infrared image samples and visible light image samples to train a two-stream deep neural network, i.e., an infrared image encoder and a visible light image encoder, learn an infrared feature extraction model and a visible light feature extraction model, and align the features output by the last layer;

[0091] Step 303: for a specific application, a certain number of second infrared image samples are collected and annotated at the pixel level. At the same time, for each target category to be segmented, several reference visible light images are collected;

[0092] Step 304: construct an infrared image segmentation model based on the pre-trained model of the comparison between infrared and visible light. Fine-tune the model based on the data in step 303;

[0093] Step 305, the infrared image segmentation model of step 304 is used to segment the infrared image to be segmented. At this time, only the infrared image to be segmented needs to be input.

[0094] This embodiment uses paired first infrared image samples and visible light image samples to pre-train the infrared image encoder and the visible light image encoder in the infrared image segmentation model before training the infrared image segmentation model. For specific infrared image applications, only a certain amount of labeled infrared images and a small amount of visible light images of preset target categories are required for model training to achieve accurate segmentation of infrared images, which has a wider range of applications. In the testing phase, only the infrared image to be segmented needs to be provided as input, and the category representation of the infrared image to be segmented is pre-calculated. While ensuring the accuracy of infrared image segmentation, the test complexity is reduced and the test efficiency is improved.

[0095] On the basis of the above embodiment, in the infrared image encoder inputting the infrared image to be segmented into the infrared image segmentation model, before obtaining the features of the infrared image to be segmented output by the infrared image encoder, the present embodiment further includes:

[0096] Inputting the first infrared image sample into the infrared image encoder to obtain features of the first infrared image sample output by the infrared image encoder;

[0097] Inputting the visible light image sample into the visible light image encoder to obtain features of the visible light image sample output by the visible light image encoder;

[0098] determining a loss between a feature of the first infrared image sample and a feature of the visible light image sample;

[0099] Parameters of the infrared image encoder and the visible light image encoder are adjusted according to the loss.

[0100] In the first training stage, the infrared image encoder and the visible light image encoder are pre-trained. The features of the first infrared image sample output by the infrared image encoder are compared with the features of the visible light image sample output by the visible light image encoder to determine the loss between the two.

[0101] For example, the similarity between the feature of the first infrared image sample and the feature of the paired visible light image sample may be determined, and the loss between the two may be determined based on the similarity, where the higher the similarity, the smaller the loss. This embodiment does not limit the calculation method of the loss between the two features.

[0102] In this embodiment, before training the infrared image segmentation model, the infrared image encoder and the visible light image encoder in the infrared image segmentation model are pre-trained using the paired first infrared image sample and the visible light image sample, so that the features output by the two encoders are aligned. The two aligned encoders will encode the pairing knowledge of infrared and visible light, and the pairing knowledge will be used in the image segmentation model to enhance the segmentation performance of the infrared image.

[0103] Based on the above embodiment, the step of determining the loss between the feature of the first infrared image sample and the feature of the visible light image sample in this embodiment includes:

[0104] Determine a dot product between a feature of a current first infrared image sample and a feature of a visible light image sample corresponding to the current first infrared image sample;

[0105] Determine a first sum of dot products between the feature of the current first infrared image sample and the features of the visible light image samples corresponding to each first infrared image sample;

[0106] determining a first loss between a feature of the first infrared image sample and a feature of the visible light image sample based on the dot product and the first sum;

[0107] Determine a second sum of dot products between the feature of the visible light image sample corresponding to the current first infrared image sample and the features of each of the first infrared image samples;

[0108] determining a second loss between a feature of the first infrared image sample and a feature of the visible light image sample based on the dot product and the second sum;

[0109] The loss between the feature of the current first infrared image sample and the feature of the visible light image sample corresponding to the current first infrared image sample is determined according to the first loss and / or the second loss.

[0110] This embodiment proposes a method based on cross-modal contrastive learning to pre-train the infrared image encoder and the visible light image encoder at the same time. Figure 4 The structural diagram of the model used for pre-training is given. The model is a two-stream structure, including an infrared image encoder and a visible light image encoder, denoted as E thm and E rgb .

[0111] The model processes n first infrared image samples per batch during training And n samples of visible light images paired with it

[0112] In order to enhance the generalization effect of the model, data augmentation is performed on the first infrared image sample and the visible light image sample. The data augmentation operations are represented as A thm and A rgb ,Commonly used data augmentation methods include color transformation and cutmix.

[0113] The infrared encoder and the visible light encoder extract features from the amplified image respectively to obtain feature I i and V i :

[0114]

[0115] Among them, Norm represents the L2 normalization operation, I i and V i The dimensions are the same. For each I i , the first loss maximizes the probability of being correctly classified as the i-th class by minimizing the cross entropy loss:

[0116]

[0117] where 𝜎 is a learnable temperature parameter. Similarly, for each V i , the second loss maximizes the probability of being correctly classified as the i-th class by minimizing the cross entropy loss:

[0118]

[0119] Finally, the contrastive learning loss function for each batch of data is obtained:

[0120]

[0121] in, and Also known as cross-modal contrastive learning loss.

[0122] In this embodiment, the infrared image encoder and the visible light image encoder are trained by the loss between the features of the first infrared image sample and the features of the visible light image sample, so that Figure 4 The feature dot product between the paired first infrared image sample and the visible light image sample is as large as possible, and the feature dot product between the unpaired first infrared image sample and the visible light image sample is as small as possible, thereby achieving feature alignment of the paired first infrared image sample and the visible light image sample, facilitating subsequent similarity graph calculation, encoding the pairing knowledge of infrared and visible light, and enhancing the segmentation performance of infrared images.

[0123] On the basis of the above embodiment, in this embodiment, the step of inputting the first infrared image sample into the infrared image encoder to obtain the features of the first infrared image sample output by the infrared image encoder; and inputting the visible light image sample into the visible light image encoder to obtain the features of the visible light image sample output by the visible light image encoder include:

[0124] Fusing the first infrared image sample and the visible light image sample to obtain a fused image;

[0125] Detecting a region of interest from the fused image and obtaining a target suggestion box of the region of interest;

[0126] Cutting out corresponding image regions from the first infrared image sample and the visible light image sample according to each target suggestion frame;

[0127] Inputting the image area corresponding to the first infrared image sample into the infrared image encoder to obtain the features of the first infrared image sample output by the infrared image encoder;

[0128] The image area corresponding to the visible light image sample is input into the visible light image encoder to obtain the feature of the visible light image sample output by the visible light image encoder.

[0129] This embodiment constructs an infrared and visible light pairing dataset with the goal of learning the pairing knowledge of infrared and visible light, which will be used to enhance the infrared image segmentation model. Public infrared and visible light pairing data is collected and processed appropriately to meet actual needs. Pairing means that the visible light image and the infrared image are registered so that the same pixel corresponds to the same position in the physical world.

[0130] For video data (such as LasHeR), considering the redundant information between adjacent video frames, video frames are extracted at a certain time interval. For non-video data (such as LLVIP), all images are used.

[0131] The collected data will be processed through Figure 5The processing shown in the figure obtains the infrared and visible light paired data set. First, the first infrared image sample and the visible light image sample are fused based on the fusion algorithm of infrared image and visible light image to obtain a fused three-channel color image; then the selective search algorithm or other target candidate frame extraction method can be used to detect the region of interest in the fused image, and the selective search algorithm will return a certain number of target suggestion frames; then the target suggestion frame is used to crop the corresponding image area from the original first infrared image sample and the visible light image sample. In order to retain certain target context information, the target suggestion frame can be enlarged by a preset ratio, such as a preset ratio of 10%. The two image areas cropped according to the same target suggestion frame are saved to construct the infrared and visible light paired data set.

[0132] It is worth noting that the first infrared image sample and the entire visible light image sample can also be used for comparative learning. Considering that infrared images are mostly outdoor scene images, if the target occupies a small number of pixels, it is difficult to learn the fine-grained features of the target by directly using the uncropped original image for model training.

[0133] This embodiment fuses the first infrared image sample and the visible light image sample, accurately detects the target suggestion frame from the fused image, and crops the corresponding image area from the first infrared image sample and the visible light image sample according to the target suggestion frame, thereby enhancing the model's feature extraction capability for small targets and improving the accuracy of infrared image segmentation.

[0134] On the basis of the above embodiment, the method of fusing the first infrared image sample and the visible light image sample to obtain a fused image in this embodiment includes:

[0135] When the brightness of the visible light image sample is less than a preset threshold, performing contrast enhancement on the visible light image sample;

[0136] The first infrared image sample and the contrast-enhanced visible light image sample are fused to obtain the fused image.

[0137] like Figure 5 As shown, for each paired first infrared image sample and visible light image sample input, it is first determined whether the visible light image sample is a low-light image, that is, whether the brightness is less than a preset threshold. If the visible light image sample is a low-light image, histogram equalization can be used to enhance the contrast.

[0138] The reason for contrast enhancement of low-light visible light image samples is that some of the public paired data are constructed for nighttime recognition tasks, and there are cases where the brightness of visible light images is low. This embodiment performs contrast enhancement on low-brightness visible light image samples and then feeds them into the subsequent image fusion algorithm to improve the fusion effect.

[0139] On the basis of the above embodiments, in the present embodiment, before the infrared image encoder inputting the infrared image to be segmented into the infrared image segmentation model obtains the features of the infrared image to be segmented output by the infrared image encoder, the method further includes:

[0140] Inputting the second infrared image sample into the infrared image encoder to obtain features of the second infrared image sample output by the infrared image encoder;

[0141] Inputting the reference visible light images of each preset target category into the visible light image encoder, and obtaining the features of the reference visible light images of each preset target category output by the visible light image encoder;

[0142] Determining a similarity graph between features of the second infrared image sample and features of the reference visible light images of each preset target category;

[0143] Inputting the feature of the second infrared image sample and the similarity map into the infrared image decoder to obtain a predicted segmentation result of the second infrared image sample output by the infrared image decoder;

[0144] determining a segmentation loss of the second infrared image sample according to a probability that the predicted segmentation result of each pixel in the second infrared image sample is labeled as the segmentation result of each pixel;

[0145] determining, according to a probability that the similarity graph corresponding to each pixel in the second infrared image sample is labeled as the segmentation result, an alignment loss between features of the second infrared image sample and features of the reference visible light images of each preset target category;

[0146] According to the segmentation loss and / or alignment loss, the parameters of the infrared image segmentation model are adjusted.

[0147] In the second stage of training the infrared image segmentation model, we use Figure 2 The model structure shown. Each second infrared image sample X thm After the infrared image encoder E thm After encoding, the feature map F of the second infrared image sample is obtained thm =E thm (X thm). For the second infrared image sample with a size of 640×640, F thm The size of can be 40×40×768, where 40 is the spatial dimension and 768 is the feature dimension.

[0148] During the second stage of training, reference visible light images of preset target categories need to be provided. The preset target categories are determined according to the application field of the second infrared image samples. For military applications, reference visible light images can be obtained through reconnaissance aircraft.

[0149] Assume that there are K preset target categories, and each preset target category provides M reference visible light images. For the m∈{1,…,M}th reference visible light image of the k∈{1,…,K}th category, the symbol After being processed by the visible light image encoder, the feature map is obtained. The size of each feature map is 40×40×768.

[0150] The features of the second infrared image sample may be compared with the pixel values ​​at the same position in the features of the reference visible light images of the preset target categories, and a similarity map S may be obtained based on the comparison results.

[0151] In order to utilize the useful information in S, S and F thm At the same time, it is sent to the infrared image decoder to obtain the predicted segmentation result of the second infrared image sample.

[0152] Two loss functions are used in the second stage training process, namely the segmentation loss of the second infrared image sample, and the infrared and visible light alignment loss.

[0153] The segmentation loss of the second infrared image sample may be a cross entropy loss, which requires the prediction of the infrared image decoder to be close to the true segmentation result label. Assume that the infrared image decoder predicts that the pixel at position (i, j) in the second infrared image sample is the true segmentation result label l ij The probability of P[i,j,l ij ], the probability of the k-th category is P[i,j,k], then the segmentation loss is defined as:

[0154]

[0155] The infrared and visible light alignment loss requires that the similarity map S is close to the true segmentation result annotation. This loss can use the cross entropy loss function. Assume that the infrared image decoder predicts that the pixel at position (i, j) in the similarity map S is the true segmentation result annotation l ij The probability of S[i,j,l ij ], the probability of the k-th category is S[i,j,k], then the infrared and visible light alignment loss is defined as:

[0156]

[0157] Among them, τ is the preset temperature parameter. The loss function of the final training infrared image segmentation model can be:

[0158] L seg =loss task +β·loss align

[0159] Among them, β is a hyperparameter.

[0160] This embodiment adjusts the parameters of the infrared image segmentation model according to the segmentation loss and / or alignment loss, so that the infrared features and visible light features obtained after the infrared image segmentation model is trained are aligned, and the predicted segmentation result is close to the segmentation result annotation. After enhancing the infrared image based on the visible light image, the infrared image is accurately segmented.

[0161] On the basis of the above embodiment, the method of determining a similarity graph between the features of the second infrared image sample and the features of the reference visible light images of each preset target category in this embodiment includes:

[0162] generating each vector of the second infrared image sample according to the pixel value at the same position in the feature of the second infrared image sample;

[0163] Obtaining a representation vector of each preset target category according to the features of the reference visible light image of each preset target category;

[0164] Determining the similarity between each vector of the second infrared image sample and the representative vector of each preset target category;

[0165] According to the similarity between each vector of the second infrared image sample and the representation vector of each preset target category, a similarity map between the feature of the second infrared image sample and the feature of the reference visible light image of each preset target category is determined.

[0166] For each preset target category, the representation vector of each preset target category can be obtained in the following manner, such as a 768-dimensional vector:

[0167]

[0168] Where t is the dimension index, which ranges from 1 to 768. is the representation vector of the kth class, is the value of the tth element.

[0169] Based on F thm and The similarity graph can be calculated as follows:

[0170]

[0171] Among them, S represents the similarity graph, S[i,j,k] is the element value corresponding to the kth channel in its i-th row and j-th column, and the dot represents the calculation of the inner product of two vectors. thm [i,j,:] is F thm The eigenvector corresponding to the i-th row and j-th column, ‖‖ represents the L2 norm of the vector.

[0172] This embodiment uses a reference visible light image to extract category representation, calculates a similarity map between infrared features and category representation, and finally introduces the similarity map into the decoder as auxiliary information, thereby improving the model's perception of infrared target positions and categories.

[0173] The infrared image segmentation device based on visible light image enhancement provided by the present invention is described below. The infrared image segmentation device based on visible light image enhancement described below and the infrared image segmentation method based on visible light image enhancement described above can refer to each other.

[0174] like Figure 6 As shown, the device includes a coding module 601, an enhancement module 602 and a segmentation module 603, wherein:

[0175] The encoding module 601 is used to input the infrared image to be segmented into the infrared image encoder in the infrared image segmentation model, and obtain the features of the infrared image to be segmented output by the infrared image encoder;

[0176] The enhancement module 602 is used to determine a similarity graph between the features of the infrared image to be segmented and the features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are obtained in advance based on a visible light image encoder in the infrared image segmentation model;

[0177] The segmentation module 603 is used to input the features of the infrared image to be segmented and the similarity map into the infrared image decoder in the infrared image segmentation model, and obtain the predicted segmentation result of the infrared image to be segmented output by the infrared image decoder;

[0178] The infrared image segmentation model is obtained by pre-training the infrared image encoder and the visible light image encoder using the first training sample and then training them using the second training sample;

[0179] The first training sample includes a first infrared image sample and a visible light image sample paired with the first infrared image sample;

[0180] The second training sample includes a second infrared image sample, a segmentation result annotation of the second infrared image sample, and the reference visible light image.

[0181] This embodiment uses paired first infrared image samples and visible light image samples to pre-train the infrared image encoder and the visible light image encoder in the infrared image segmentation model before training the infrared image segmentation model. For specific infrared image applications, only a certain amount of labeled infrared images and a small amount of visible light images of preset target categories are required for model training to achieve accurate segmentation of infrared images, which has a wider range of applications. In the testing phase, only the infrared image to be segmented needs to be provided as input, and the category representation of the infrared image to be segmented is pre-calculated. While ensuring the accuracy of infrared image segmentation, the test complexity is reduced and the test efficiency is improved.

[0182] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the infrared image segmentation method based on visible light image enhancement, the method comprising: inputting the infrared image to be segmented into the infrared image encoder in the infrared image segmentation model, obtaining the features of the infrared image to be segmented output by the infrared image encoder; determining the similarity graph between the features of the infrared image to be segmented and the features of the reference visible light image of the preset target category, the features of the reference visible light image being obtained in advance based on the visible light image encoder in the infrared image segmentation model; inputting the features of the infrared image to be segmented and the similarity graph into the infrared image decoder in the infrared image segmentation model, obtaining the predicted segmentation result of the infrared image to be segmented output by the infrared image decoder.

[0183] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0184] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the infrared image segmentation method based on visible light image enhancement provided by the above-mentioned methods, and the method includes: inputting the infrared image to be segmented into an infrared image encoder in an infrared image segmentation model to obtain the features of the infrared image to be segmented output by the infrared image encoder; determining a similarity map between the features of the infrared image to be segmented and the features of a reference visible light image of a preset target category, and the features of the reference visible light image are pre-obtained based on the visible light image encoder in the infrared image segmentation model; inputting the features of the infrared image to be segmented and the similarity map into an infrared image decoder in the infrared image segmentation model to obtain a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder.

[0185] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the infrared image segmentation method based on visible light image enhancement provided by the above-mentioned methods, the method comprising: inputting the infrared image to be segmented into an infrared image encoder in an infrared image segmentation model, and obtaining features of the infrared image to be segmented output by the infrared image encoder; determining a similarity map between features of the infrared image to be segmented and features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are pre-obtained based on the visible light image encoder in the infrared image segmentation model; inputting the features of the infrared image to be segmented and the similarity map into an infrared image decoder in the infrared image segmentation model, and obtaining a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder.

[0186] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared image segmentation method based on visible light image enhancement, characterized in that: include: Inputting the infrared image to be segmented into an infrared image encoder in the infrared image segmentation model, and obtaining features of the infrared image to be segmented output by the infrared image encoder; Determine a similarity graph between features of the infrared image to be segmented and features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are pre-obtained based on a visible light image encoder in the infrared image segmentation model; Inputting the features of the infrared image to be segmented and the similarity map into an infrared image decoder in the infrared image segmentation model, and obtaining a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder; The infrared image segmentation model is obtained by pre-training the infrared image encoder and the visible light image encoder using the first training sample and then training them using the second training sample; The first training sample includes a first infrared image sample and a visible light image sample paired with the first infrared image sample; The second training sample includes a second infrared image sample, a segmentation result annotation of the second infrared image sample, and the reference visible light image.

2. The infrared image segmentation method based on visible light image enhancement according to claim 1 is characterized in that: Before the infrared image to be segmented is input into the infrared image encoder in the infrared image segmentation model and the features of the infrared image to be segmented output by the infrared image encoder are obtained, the method further includes: Inputting the first infrared image sample into the infrared image encoder to obtain features of the first infrared image sample output by the infrared image encoder; Inputting the visible light image sample into the visible light image encoder to obtain features of the visible light image sample output by the visible light image encoder; determining a loss between a feature of the first infrared image sample and a feature of the visible light image sample; Parameters of the infrared image encoder and the visible light image encoder are adjusted according to the loss.

3. The infrared image segmentation method based on visible light image enhancement according to claim 2 is characterized in that: The determining of the loss between the feature of the first infrared image sample and the feature of the visible light image sample comprises: Determine a dot product between a feature of a current first infrared image sample and a feature of a visible light image sample corresponding to the current first infrared image sample; Determine a first sum of dot products between the feature of the current first infrared image sample and the features of the visible light image samples corresponding to each first infrared image sample; determining a first loss between a feature of the first infrared image sample and a feature of the visible light image sample based on the dot product and the first sum; Determine a second sum of dot products between the feature of the visible light image sample corresponding to the current first infrared image sample and the features of each of the first infrared image samples; determining a second loss between a feature of the first infrared image sample and a feature of the visible light image sample based on the dot product and the second sum; The loss between the feature of the current first infrared image sample and the feature of the visible light image sample corresponding to the current first infrared image sample is determined according to the first loss and / or the second loss.

4. The infrared image segmentation method based on visible light image enhancement according to claim 2 is characterized in that: inputting the first infrared image sample into the infrared image encoder to obtain the features of the first infrared image sample output by the infrared image encoder; Inputting the visible light image sample into the visible light image encoder to obtain features of the visible light image sample output by the visible light image encoder includes: Fusing the first infrared image sample and the visible light image sample to obtain a fused image; Detecting a region of interest from the fused image and obtaining a target suggestion box of the region of interest; Cutting out corresponding image regions from the first infrared image sample and the visible light image sample according to each target suggestion frame; Inputting the image area corresponding to the first infrared image sample into the infrared image encoder to obtain the features of the first infrared image sample output by the infrared image encoder; The image area corresponding to the visible light image sample is input into the visible light image encoder to obtain the feature of the visible light image sample output by the visible light image encoder.

5. The infrared image segmentation method based on visible light image enhancement according to claim 4 is characterized in that: The fusing the first infrared image sample and the visible light image sample to obtain a fused image includes: When the brightness of the visible light image sample is less than a preset threshold, performing contrast enhancement on the visible light image sample; The first infrared image sample and the contrast-enhanced visible light image sample are fused to obtain the fused image.

6. The infrared image segmentation method based on visible light image enhancement according to any one of claims 1 to 5, characterized in that: Before the infrared image to be segmented is input into the infrared image encoder in the infrared image segmentation model and the features of the infrared image to be segmented output by the infrared image encoder are obtained, the method further includes: Inputting the second infrared image sample into the infrared image encoder to obtain features of the second infrared image sample output by the infrared image encoder; Inputting the reference visible light images of each preset target category into the visible light image encoder, and obtaining the features of the reference visible light images of each preset target category output by the visible light image encoder; Determining a similarity graph between features of the second infrared image sample and features of the reference visible light images of each preset target category; Inputting the feature of the second infrared image sample and the similarity map into the infrared image decoder to obtain a predicted segmentation result of the second infrared image sample output by the infrared image decoder; determining a segmentation loss of the second infrared image sample according to a probability that the predicted segmentation result of each pixel in the second infrared image sample is labeled as the segmentation result of each pixel; determining, according to a probability that the similarity graph corresponding to each pixel in the second infrared image sample is labeled as the segmentation result, an alignment loss between features of the second infrared image sample and features of the reference visible light images of each preset target category; According to the segmentation loss and / or alignment loss, the parameters of the infrared image segmentation model are adjusted.

7. The infrared image segmentation method based on visible light image enhancement according to claim 6 is characterized in that: The determining of a similarity graph between features of the second infrared image sample and features of the reference visible light images of each preset target category includes: generating each vector of the second infrared image sample according to the pixel value at the same position in the feature of the second infrared image sample; Obtaining a representation vector of each preset target category according to the features of the reference visible light image of each preset target category; Determining the similarity between each vector of the second infrared image sample and the representative vector of each preset target category; According to the similarity between each vector of the second infrared image sample and the representation vector of each preset target category, a similarity map between the feature of the second infrared image sample and the feature of the reference visible light image of each preset target category is determined.

8. An infrared image segmentation device based on visible light image enhancement, characterized in that: include: An encoding module, used for inputting the infrared image to be segmented into an infrared image encoder in the infrared image segmentation model, and obtaining the features of the infrared image to be segmented output by the infrared image encoder; an enhancement module, configured to determine a similarity graph between features of the infrared image to be segmented and features of a reference visible light image of a preset target category, wherein the features of the reference visible light image are obtained in advance based on a visible light image encoder in the infrared image segmentation model; A segmentation module, used for inputting the features of the infrared image to be segmented and the similarity map into the infrared image decoder in the infrared image segmentation model, and obtaining a predicted segmentation result of the infrared image to be segmented output by the infrared image decoder; The infrared image segmentation model is obtained by pre-training the infrared image encoder and the visible light image encoder using the first training sample and then training them using the second training sample; The first training sample includes a first infrared image sample and a visible light image sample paired with the first infrared image sample; The second training sample includes a second infrared image sample, a segmentation result annotation of the second infrared image sample, and the reference visible light image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the infrared image segmentation method based on visible light image enhancement as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the infrared image segmentation method based on visible light image enhancement as claimed in any one of claims 1 to 7 is implemented.

Images