Image translation model construction method, image translation method and device

Abstract

The present invention provides an image translation model construction method, an image translation method and a device, and the construction method includes: obtaining a training sample group; extracting image features of a visible light sample image based on a coding module in an initial model; obtaining a prompt feature, a learnable mask output feature and an infrared output feature, and splicing the prompt feature, the mask output feature and the infrared output feature to obtain a spliced ​​feature; performing mask image prediction and infrared image prediction based on the image feature and the spliced ​​feature to obtain a predicted mask image and a predicted infrared image respectively; performing parameter iteration on the initial model based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image to obtain an image translation model. The method and device provided by the present invention can realize interactive image translation and improve the accuracy of image translation.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image translation model construction method, an image translation method and a device. Background Art

[0002] Compared with the visual perception system based on visible light, the visual perception system based on infrared imaging has a natural advantage in low light, rain, fog and other bad weather conditions, so the former is a necessary supplement to the latter. The effectiveness of infrared visual perception algorithms based on deep learning is highly dependent on large-scale labeled training data, so its development is largely restricted by the problem of insufficient data. Transforming visible light images into infrared images (called pseudo-infrared images) through deep learning algorithms can make up for the problem of insufficient real data to a certain extent.

[0003] In the related art, the visible light image is usually transformed into a pseudo infrared image using an adversarial learning method. The generator obtains the pseudo infrared image based on the visible light image, and the discriminator is responsible for reducing the difference between the infrared and visible light domains.

[0004] However, current methods usually use small models, which make it difficult to effectively capture the complex nonlinear transformation between visible light and infrared images, resulting in poor image translation results. In addition, current methods are not interactive, that is, they can only process the entire image and cannot perform interactive image translation by specifying a certain area. Summary of the invention

[0005] The present invention provides an image translation model construction method, an image translation method and a device, which are used to solve the defects in the prior art of converting visible light images into pseudo infrared images, that is, the image translation effect is poor and there is no interactivity.

[0006] The present invention provides a method for constructing an image translation model, comprising:

[0007] Obtain a training sample group, where any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0008] Extracting image features of the visible light sample image based on the encoding module in the initial model;

[0009] Acquire a prompt feature, a learnable mask output feature, and an infrared output feature, and concatenate the prompt feature, the mask output feature, and the infrared output feature to obtain a concatenated feature;

[0010] Based on the image features and the splicing features, respectively, a mask image prediction and an infrared image prediction are performed to obtain a predicted mask image and a predicted infrared image;

[0011] Based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, the initial model is iterated on parameters to obtain the image translation model.

[0012] According to the image translation model construction method provided by the present invention, the mask image prediction and the infrared image prediction are performed based on the image features and the splicing features respectively to obtain the predicted mask image and the predicted infrared image respectively, including:

[0013] Inputting the image features and the splicing features into a decoding module of the initial model, obtaining output image features and output splicing features output by the decoding module, and calculating the correlation between the output image features and the output splicing features;

[0014] Performing mask image prediction based on the correlation, the output image features and a mask multi-layer perceptron to obtain the predicted mask image;

[0015] Infrared image prediction is performed based on the correlation, the output image features and an infrared multi-layer perceptron to obtain the predicted infrared image.

[0016] According to the image translation model construction method provided by the present invention, the infrared image prediction is performed based on the correlation, the output image features and the infrared multi-layer perceptron to obtain the predicted infrared image, including:

[0017] Based on the feature fusion module in the initial model, the image feature and the output image feature are fused to obtain a fused image feature;

[0018] Based on the infrared multi-layer perceptron, the correlation is transformed to obtain an infrared prediction weight;

[0019] Based on the infrared prediction weights, the fused image features are transformed to obtain the predicted infrared image.

[0020] According to the image translation model construction method provided by the present invention, the encoding module includes multiple encoding layers, the image features include image features of each level output by each encoding layer, and the feature fusion module based on the initial model fuses the image features with the output image features to obtain fused image features, including:

[0021] Based on the feature fusion module in the initial model, at least two levels of image features in the image features of each level are fused to obtain an initial fused feature;

[0022] The initial fusion feature and the output image feature are fused to obtain the fusion image feature.

[0023] According to the image translation model construction method provided by the present invention, the initial model is iterated based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, to obtain the image translation model, including:

[0024] determining an infrared prediction loss based on a difference between the infrared tag image and the predicted infrared image;

[0025] determining a mask prediction loss based on a difference between the mask label image and the predicted mask image;

[0026] Based on the infrared prediction loss and the mask prediction loss, parameter iteration is performed on the learnable infrared output feature, the infrared multi-layer perceptron and the feature fusion module to obtain the image translation model.

[0027] According to the image translation model construction method provided by the present invention, the step of determining the mask label image includes:

[0028] Inputting the visible light sample image into the SAM segmentation model, segmenting the visible light sample image to obtain a multi-value segmentation map;

[0029] Based on an intersection-over-union ratio threshold and a stability score threshold, the multi-value segmentation map is converted into a binary segmentation map, and the binary segmentation map is used as the mask label image.

[0030] The present invention also provides an image translation method, comprising:

[0031] Obtaining a visible light image to be translated;

[0032] Inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model;

[0033] The image translation model is constructed based on the image translation model construction method.

[0034] According to the image translation method provided by the present invention, the visible light image is input into the image translation model to obtain the infrared image and the mask image output by the image translation model, including:

[0035] Receive prompt information input by the user;

[0036] The prompt information and the visible light image are input into an image translation model to obtain an infrared image and a mask image corresponding to the prompt information output by the image translation model.

[0037] The present invention also provides an image translation model construction device, comprising:

[0038] A sample group acquisition unit, used to acquire a training sample group, wherein any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0039] A feature extraction unit, used for extracting image features of the visible light sample image based on the encoding module in the initial model;

[0040] A feature splicing unit, used for obtaining a prompt feature, a learnable mask output feature and an infrared output feature, and splicing the prompt feature, the mask output feature and the infrared output feature to obtain a spliced ​​feature;

[0041] An image prediction unit, used to perform mask image prediction and infrared image prediction based on the image features and the splicing features, respectively, to obtain a predicted mask image and a predicted infrared image, respectively;

[0042] A parameter iteration unit is used to perform parameter iteration on the initial model based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, so as to obtain the image translation model.

[0043] The present invention also provides an image translation device, comprising:

[0044] An image acquisition unit, used for acquiring a visible light image to be translated;

[0045] An image translation unit, used for inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model;

[0046] The image translation model is constructed based on the image translation model construction method.

[0047] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the image translation model construction method or the image translation method as described above is implemented.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image translation model construction method or image translation method as described above is implemented.

[0049] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any one of the above-mentioned image translation model construction methods or image translation methods.

[0050] The image translation model construction method, image translation method and device provided by the present invention construct an image translation model based on the SAM model, further improve the accuracy of image translation by utilizing the excellent feature expression ability of SAM, and realize interactive translation from visible light to infrared images by utilizing the interactive decoding ability of SAM. In addition, by adding only a small number of learnable parameters, fast model fine-tuning can be achieved to realize efficient model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 This is one of the flow charts of the method for constructing an image translation model provided by the present invention;

[0053] Figure 2 It is a flow chart of the method for obtaining a training sample group provided by the present invention;

[0054] Figure 3 It is a structural schematic diagram of the image translation model provided by the present invention;

[0055] Figure 4 This is the second flow chart of the method for constructing an image translation model provided by the present invention;

[0056] Figure 5 It is a flowchart of the image translation method provided by the present invention;

[0057] Figure 6 It is a structural schematic diagram of the image translation model construction device provided by the present invention;

[0058] Figure 7 It is a structural schematic diagram of the image translation device provided by the present invention;

[0059] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] In view of the problem that the image translation effect of converting visible light images into pseudo infrared images in the prior art is poor and non-interactive, in order to improve the image translation effect, the inventive concept of the present invention is to build an image translation model based on the SAM (SegmentAnything Model) model for the translation task of visible light to infrared images, further improve the accuracy of image translation by utilizing the excellent feature expression ability of the visual large model in the SAM model, and realize interactive translation of visible light to infrared images by utilizing the interactive decoding capability of SAM.

[0062] Based on the above-mentioned inventive concept, the present invention provides an image translation model construction method, an image translation method and a device, which can be applied to the translation scenario of visible light to infrared images in image processing technology to provide a more accurate and interactive image translation solution.

[0063] In the description of the embodiments of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0064] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1 This is one of the flow charts of the method for constructing an image translation model provided by the present invention. Figure 1 As shown, the method comprises the following steps:

[0065] Step 110, obtaining a training sample group, wherein any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0066] Step 120, extracting image features of the visible light sample image based on the encoding module in the initial model;

[0067] Step 130, obtaining a prompt feature, a learnable mask output feature, and an infrared output feature, and splicing the prompt feature, the mask output feature, and the infrared output feature to obtain a spliced ​​feature;

[0068] Step 140, performing mask image prediction and infrared image prediction based on the image features and the splicing features, respectively, to obtain a predicted mask image and a predicted infrared image;

[0069] Step 150, based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, the initial model is iterated to obtain an image translation model.

[0070] Specifically, the image translation model in this embodiment can be constructed based on the SAM (Segment Anything Model) model. In order to train a translation model from visible light to infrared images, a training sample group is first obtained, where each training sample group includes a visible light sample image and its corresponding infrared label image, and a mask label image.

[0071] Here, the visible light sample image and its corresponding infrared label image can be realized through the visible light-infrared pairing dataset, where pairing means that each pixel of the visible light image and the infrared image corresponds to the same target. First, collect the public datasets with visible light-infrared pairing images, including LLVIP, M3FD, DroneVehicle and other datasets. Among them, for video type datasets, extract 1 frame of image every 10 frames; merge all datasets to form a larger visible light-infrared pairing dataset; divide the training set and test set according to the ratio of 8:2, and use the visible light images in the training set as visible light sample images, and use their paired infrared images as the infrared label images corresponding to the visible light sample images, and at the same time as the infrared labels of the training samples.

[0072] In order to use the SAM model for interactive image translation, pixel-level segmentation annotation of the image is required. The mask label image can be a binary mask image of the visible light sample image, which is obtained by pixel segmentation of the visible light sample image. The mask label image can be used as a mask label for training samples. One visible light sample image can correspond to multiple mask label images. For example, each category in the visible light sample image can correspond to a mask label image.

[0073] Although some existing data sets provide segmentation annotations, they only contain limited categories. Using such data to train SAM results in poor segmentation effects on unseen categories. To solve this problem, the step of determining the mask label image in an embodiment of the present invention includes:

[0074] Input the visible light sample image into the SAM segmentation model, segment the visible light sample image, and obtain a multi-value segmentation map;

[0075] Based on the intersection-over-union ratio threshold and the stability score threshold, the multi-value segmentation map is converted into a binary segmentation map, and the binary segmentation map is used as the mask label image.

[0076] Specifically, for obtaining the mask label image, SAM can be used to segment the visible light sample image to obtain pseudo-segmentation annotations, and the pseudo-segmentation annotations are used as labels for model training.

[0077] Figure 2 FIG. 1 is a flow chart of a method for obtaining a training sample group provided by the present invention. Figure 2 As shown, firstly, a paired visible light sample image and an infrared tag portrait are obtained, and the visible light sample image is input into the SAM model to obtain a multi-value segmentation map. The multi-value segmentation map reflects the category of each pixel point in the visible light sample image through different pixel values.

[0078] Then, based on the intersection-over-union ratio threshold and the stability score threshold, the multi-value segmentation map is converted into a binary segmentation map, and the binary segmentation map is used as the mask label image. For example, here we can use a 32×32 input grid, predict the intersection-over-union ratio IoU threshold as 0.86, the stability score threshold as 0.92, and remove the segmented areas with an area less than 100 pixels to convert the multi-value segmentation map into a binary segmentation map.

[0079] Assume there are m pairs of visible light-infrared data, and the i-th pairing data gets n i segmentation areas, each segmentation area can correspond to a binary segmentation map. Therefore, after this step, we will get There are training sample groups, each of which contains a visible light sample image, an infrared label image, and a mask label image.

[0080] After obtaining the training sample group, an initial model based on SAM can be built. The structure of the initial model can inherit the structure of SAM, and the model structure is improved to realize the image translation function. The trained initial model is the image translation model, which can be used for the translation of visible light to infrared images.

[0081] Figure 3 Schematic diagram of the structure of the image translation model provided by the present invention. Figure 3 As shown, after the model is built, step 120 can be executed to extract image features of the visible light sample image based on the encoding module in the initial model.

[0082] The initial model includes a coding module for extracting image features of the visible light sample image. In the embodiment of the present invention, a vision transformer (ViT) can be used as the coding module, and structures with different parameter amounts such as ViT-B, ViT-L and ViT-H can be used. The image coding module of the SAM model has been trained on a large amount of segmentation data and has a general feature extraction capability. Therefore, the embodiment of the present invention can directly use the pre-trained ViT in SAM for initialization and freeze the coding module during the training process.

[0083] In step 130, the prompt feature, the learnable mask output feature and the infrared output feature are obtained, and the prompt feature, the mask output feature and the infrared output feature are spliced ​​to obtain a spliced ​​feature.

[0084] Specifically, various prompt information may be input into the model, such as point prompts, box prompts or text prompts, and the prompt encoder may be used to convert various prompt information into prompt features; in order to predict the mask image, a learnable mask output feature may be introduced; in order to predict the infrared image, a learnable infrared output feature may be introduced. Here, the prompt feature may be a prompt token, the learnable mask output feature may be an output token, and the infrared output feature may be an infrared output token. The prompt feature, the learnable mask output feature and the infrared output feature may be pre-set matrices.

[0085] Then, the prompt feature, the mask output feature and the infrared output feature are concatenated, specifically, they can be concatenated in the sequence dimension to obtain the concatenated feature.

[0086] For example, the learnable infrared output token is represented by T inf ∈R 1×d , where d is the embedding dimension. Assume that the prompt token in the original SAM is T prompt ∈R n×d , the output token is T out ∈R o×d , n and o are the lengths of the prompt token and output token respectively. Then, the token sequence of the concatenated feature can be expressed as T = [T inf ; T prompt ; T out ]∈R (n+o+1)×d Here [X; Y] represents the operation of concatenating two matrices X and Y along a certain dimension.

[0087] It should be noted that step 120 and step 130 may be performed simultaneously or sequentially. When performed sequentially, the execution order of the two is not limited, and step 120 may be performed first and then step 130, or step 130 may be performed first and then step 120.

[0088] On this basis, step 140 can be executed to perform mask image prediction and infrared image prediction based on the image features and the splicing features, respectively, to obtain a predicted mask image and a predicted infrared image, respectively. In some embodiments, step 140 specifically includes:

[0089] Step 141, inputting the image features and the splicing features into the decoding module of the initial model, obtaining the output image features and the output splicing features output by the decoding module, and calculating the correlation between the output image features and the output splicing features;

[0090] Step 142, performing mask image prediction based on the correlation, output image features and mask multi-layer perceptron to obtain a predicted mask image;

[0091] Step 143, predicting the infrared image based on the correlation, the output image features and the infrared multi-layer perceptron to obtain a predicted infrared image.

[0092] Specifically, Figure 3 As shown in the figure, the image translation model also includes a decoding module. The decoder receives the concatenated feature token sequence and image features as input, which are processed by two decoder layers in succession to realize the deep interaction between token features and image features, and obtain output image features and output concatenated features. The output image features are passed through the transposed convolution layer to obtain the updated output image features.

[0093] Subsequently, the correlation between the output image features and the output concatenated features is calculated through the attention layer. This correlation is processed using a masked multi-layer perceptron MLP to obtain dynamic weights, which will be used to transform the updated output image features (the output of the transposed convolutional layer) to obtain a binary segmentation prediction, that is, a predicted mask image.

[0094] In addition, in order to obtain infrared prediction, the embodiment of the present invention introduces a new MLP structure, namely, an infrared multi-layer perceptron, which is used to perform infrared image prediction based on correlation and output image features to obtain a predicted infrared image.

[0095] In some embodiments, infrared image prediction is performed based on correlation, output image features, and an infrared multi-layer perceptron to obtain a predicted infrared image, that is, step 143 specifically includes:

[0096] Step 143-1, based on the feature fusion module in the initial model, the image feature and the output image feature are fused to obtain a fused image feature;

[0097] Step 143-2, based on the infrared multi-layer perceptron, transform the correlation to obtain the infrared prediction weight;

[0098] Step 143 - 3 , based on the infrared prediction weights, transform the fused image features to obtain a predicted infrared image.

[0099] Specifically, the initial model also includes a feature fusion module, which is used to fuse the image features output by the encoder and the output image features output by the decoding module to obtain fused image features. The image features can represent the underlying detail features of the visible light sample image, and the output image features can represent the high-level semantic features of the visible light sample image, thereby realizing the fusion of the underlying detail features and the high-level semantic features, thereby improving the prediction effect of the infrared image.

[0100] Then, an infrared multi-layer perceptron is applied to transform the correlation and obtain a dynamic infrared prediction weight, which is used to transform the fused image features to obtain a pixel-level predicted infrared image.

[0101] In some embodiments, the encoding module includes multiple encoding layers, and the image features include image features of each level output by each encoding layer. Step 143-1 specifically includes:

[0102] Based on the feature fusion module in the initial model, at least two levels of image features in each level of image features are fused to obtain an initial fused feature;

[0103] The initial fusion features and the output image features are fused to obtain fused image features.

[0104] Specifically, the fused image features can be achieved through two fusions. First, the image features of at least two levels in the encoding module are fused to obtain initial fused features. For example, the image features output by the last layer and the image features output by the middle layer can be fused; the image features output by each encoding layer can also be fused, which is not specifically limited in the embodiment of the present invention. Then, the initial fused features and the output image features are fused to obtain fused image features, and the fused image features obtained have stronger feature representation.

[0105] In one embodiment, it is assumed that the image feature output by the last layer of the encoding module ViT is F last ∈R d×h×w , the image feature output by the third layer of ViT is F mid ∈R d×h×w , Figure 3 The output image features of the decoder layer 2 are

[0106] The feature fusion module processes these three image features to obtain fused image features. Specifically, the transposed convolution layer is first used to transform F last and F mid After fusion and 4 times upsampling, the initial fusion features are obtained:

[0107]

[0108] Among them, ConvT represents the transposed convolution with a stride of 2, GELU (Gaussian Error Linear Unit) is the nonlinear activation function, and LN represents layer normalization.

[0109] Next, use the following operation to merge the initial features And the output image feature F dec Fusion is performed to obtain the final fused image feature F fuse :

[0110]

[0111] Among them, Conv represents 3×3 convolution, and ConvT represents transposed convolution with a stride of 2.

[0112] On this basis, step 150 can be executed to iterate the parameters of the initial model based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, to obtain an image translation model. Specifically, it includes:

[0113] Step 151, determining the infrared prediction loss based on the difference between the infrared tag image and the predicted infrared image;

[0114] Step 152, determining a mask prediction loss based on a difference between the mask label image and the predicted mask image;

[0115] Step 153, based on the infrared prediction loss and the mask prediction loss, parameters of the learnable infrared output features, the infrared multi-layer perceptron and the feature fusion module are iterated to obtain an image translation model.

[0116] Specifically, Figure 3 The model shown outputs two prediction results: the predicted mask image P mask and predicted infrared image P inf In order to learn the model parameters, the mask prediction loss is defined for the difference between the mask label image and the predicted mask image, which can also be called the segmentation loss L seg ; Define the infrared prediction loss based on the difference between the infrared label image and the predicted infrared image, which can also be called the pixel regression loss L reg .

[0117] Mask prediction loss L seg Combined with binary cross entropy loss L ce and Dice loss L dice , for example, can be expressed as follows:

[0118]

[0119] in, is the true binary segmentation mask, i.e., the mask label image.

[0120] The infrared prediction loss can be implemented using L1 loss or L2 loss. For example, using L1 loss can be expressed as follows:

[0121]

[0122] in, is the real infrared image, i.e., the infrared label image, and ⊙ is the element-by-element matrix multiplication. The regions with non-zero values ​​are used to calculate the regression loss.

[0123] The final model loss function can be expressed as:

[0124] L=L seg +L reg

[0125] Then, the visible light to infrared image translation model can be trained. Here, the parameters of the encoding module, decoding module and mask multi-layer perceptron can be frozen, and only the learnable infrared output features, infrared multi-layer perceptron and feature fusion module are iterated to achieve efficient training. The initial model completed by training is the image translation model.

[0126] This embodiment can be implemented using the PyTorch framework. The optimizer used is Adam, the initial learning rate is 1e-3, the training rounds are 24 rounds, and the learning rate is reduced to 1 / 10 at the 10th and 20th rounds of iteration.

[0127] The image translation model construction method provided by the embodiment of the present invention constructs an image translation model based on the SAM model, further improves the accuracy of image translation by utilizing the excellent feature expression ability of SAM, and realizes interactive translation from visible light to infrared images by utilizing the interactive decoding ability of SAM. In addition, by adding only a small number of learnable parameters, fast model fine-tuning can be achieved to realize efficient model training.

[0128] Furthermore, the image translation model trained in this way supports both automatic image translation and interactive image translation. In two different situations, the test set can be used to evaluate the accuracy of the model. In order to evaluate the image translation effect, the mean absolute error can be used as an evaluation indicator. For automatic image translation, for each test image, a pseudo-infrared image of each visible light image is obtained, and the mean absolute error between the pseudo-infrared image and the real infrared image is calculated. The mean absolute error of all test images is used to evaluate the model. For interactive image translation, for each image in the test set, all segmented areas are traversed, the mean absolute error is calculated for each segmented area, and all mean absolute errors are averaged to obtain the final evaluation indicator.

[0129] Based on the above embodiments, Figure 4 FIG. 2 is a flow chart of the method for constructing an image translation model provided by the present invention. Figure 4 As shown, the operation steps of the method are:

[0130] Step 1: Build a visible light-infrared paired dataset for model training and evaluation;

[0131] Step 2: Use SAM to segment the visible light image and generate binary mask annotations;

[0132] Step 3: Build a visible light to infrared image translation model based on SAM;

[0133] Step 4: Use the training set to train the SAM-based visible light to infrared image translation model to obtain model parameters;

[0134] Step 5: Use the test set to test the SAM-based visible to infrared image translation model and evaluate the model performance.

[0135] Based on any of the above embodiments, Figure 5 is a flow chart of the image translation method provided by the present invention, such as Figure 5 As shown, an image translation method is provided, comprising:

[0136] Step 510, obtaining a visible light image to be translated;

[0137] Step 520, inputting the visible light image into the image translation model to obtain the infrared image and the mask image output by the image translation model; wherein the image translation model is constructed based on the above-mentioned image translation model construction method.

[0138] Specifically, the image translation model constructed by the above method can realize the translation of visible light images to pseudo infrared images in the application stage. This embodiment provides an automatic image translation method, which means that the infrared image and the mask image can be obtained by inputting only the visible light image without manually inputting prompt information such as points and boxes.

[0139] During the image translation process, the model can use k×k (for example, k=32) grid point coordinates instead of manual prompts, and a binary segmentation mask map and an infrared prediction map can be obtained for each grid point. If a pixel point is associated with multiple segmentation masks, the infrared grayscale value of the pixel is the average of the infrared grayscale values ​​corresponding to each segmentation mask. If a pixel value does not correspond to any segmentation mask, the pixel coordinates are used as a point prompt to obtain the corresponding segmentation mask map and infrared grayscale map, and then the infrared grayscale value of the pixel is obtained. In this way, a pseudo infrared image of the input visible light image can be automatically obtained.

[0140] Preferably, the image translation model also supports interactive image translation, which means that manual input of point or frame information is required to translate a specific area in a visible light image. That is, step 520 specifically includes:

[0141] Step 521, receiving prompt information input by the user;

[0142] Step 522: input the prompt information and the visible light image into the image translation model to obtain the infrared image and the mask image corresponding to the prompt information output by the image translation model.

[0143] Specifically, the prompt information input by the user is first received. The prompt information can be a point of interest, a target area box or a prompt text. The input of the image translation model includes the prompt information and the visible light image. The translation model can predict a binary segmentation mask image and a pseudo infrared image. For the image and the associated prompt, the mean absolute error of the non-zero area of ​​the binary segmentation mask is calculated. The input prompt point is obtained using the center point of each true segmentation area, and the input prompt box is obtained using the tightest bounding box of each true segmentation area.

[0144] The device provided by the present invention is described below. The device described below and the method described above can be referenced to each other.

[0145] Based on the above embodiments, Figure 6 is a schematic diagram of the structure of the image translation model construction device provided by the present invention, such as Figure 6 As shown, a device for constructing an image translation model is provided, comprising:

[0146] The sample group acquisition unit 610 is used to acquire a training sample group, where any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0147] A feature extraction unit 620, configured to extract image features of the visible light sample image based on the encoding module in the initial model;

[0148] A feature splicing unit 630 is used to obtain a prompt feature, a learnable mask output feature and an infrared output feature, and splice the prompt feature, the mask output feature and the infrared output feature to obtain a spliced ​​feature;

[0149] An image prediction unit 640 is used to perform mask image prediction and infrared image prediction based on the image features and the splicing features, respectively, to obtain a predicted mask image and a predicted infrared image, respectively;

[0150] The parameter iteration unit 650 is used to perform parameter iteration on the initial model based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, so as to obtain the image translation model.

[0151] The image translation model construction device provided by the embodiment of the present invention constructs an image translation model based on the SAM model, further improves the accuracy of image translation by utilizing the excellent feature expression ability of SAM, and realizes interactive translation from visible light to infrared images by utilizing the interactive decoding ability of SAM. In addition, by adding only a small number of learnable parameters, fast model fine-tuning can be achieved to achieve efficient model training.

[0152] Based on the above embodiment, the image prediction unit is specifically used for:

[0153] Inputting the image features and the splicing features into a decoding module of the initial model, obtaining output image features and output splicing features output by the decoding module, and calculating the correlation between the output image features and the output splicing features;

[0154] Performing mask image prediction based on the correlation, the output image features and a mask multi-layer perceptron to obtain the predicted mask image;

[0155] Infrared image prediction is performed based on the correlation, the output image features and an infrared multi-layer perceptron to obtain the predicted infrared image.

[0156] Based on the above embodiment, the image prediction unit is further specifically used for:

[0157] Based on the feature fusion module in the initial model, the image feature and the output image feature are fused to obtain a fused image feature;

[0158] Based on the infrared multi-layer perceptron, the correlation is transformed to obtain an infrared prediction weight;

[0159] Based on the infrared prediction weights, the fused image features are transformed to obtain the predicted infrared image.

[0160] Based on the above embodiment, the image prediction unit is further specifically used for:

[0161] Based on the feature fusion module in the initial model, at least two levels of image features in the image features of each level are fused to obtain an initial fused feature;

[0162] The initial fusion feature and the output image feature are fused to obtain the fusion image feature.

[0163] Based on the above embodiment, the parameter iteration unit is specifically used for:

[0164] determining an infrared prediction loss based on a difference between the infrared tag image and the predicted infrared image;

[0165] determining a mask prediction loss based on a difference between the mask label image and the predicted mask image;

[0166] Based on the infrared prediction loss and the mask prediction loss, parameter iteration is performed on the learnable infrared output feature, the infrared multi-layer perceptron and the feature fusion module to obtain the image translation model.

[0167] Based on the above embodiment, the system further includes a mask label image determining unit, which is used to:

[0168] Inputting the visible light sample image into the SAM segmentation model, segmenting the visible light sample image to obtain a multi-value segmentation map;

[0169] Based on an intersection-over-union ratio threshold and a stability score threshold, the multi-value segmentation map is converted into a binary segmentation map, and the binary segmentation map is used as the mask label image.

[0170] Based on the above embodiments, Figure 7 is a schematic diagram of the structure of the image translation device provided by the present invention, such as Figure 7 As shown, an image translation device is provided, comprising:

[0171] An image acquisition unit 710, used to acquire a visible light image to be translated;

[0172] An image translation unit 720, configured to input the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model;

[0173] The image translation model is constructed based on the image translation model construction method.

[0174] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the image translation model construction method, which includes:

[0175] Obtain a training sample group, where any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0176] Extracting image features of the visible light sample image based on the encoding module in the initial model;

[0177] Acquire a prompt feature, a learnable mask output feature, and an infrared output feature, and concatenate the prompt feature, the mask output feature, and the infrared output feature to obtain a concatenated feature;

[0178] Based on the image features and the splicing features, respectively, a mask image prediction and an infrared image prediction are performed to obtain a predicted mask image and a predicted infrared image;

[0179] Based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, the initial model is iterated on parameters to obtain the image translation model.

[0180] The processor may call logic instructions in the memory to execute an image translation method, the method comprising:

[0181] Obtaining a visible light image to be translated;

[0182] Inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model;

[0183] The image translation model is constructed based on the image translation model construction method.

[0184] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0185] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the image translation model construction method provided by the above methods, the method comprising:

[0186] Obtain a training sample group, where any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0187] Extracting image features of the visible light sample image based on the encoding module in the initial model;

[0188] Acquire a prompt feature, a learnable mask output feature, and an infrared output feature, and concatenate the prompt feature, the mask output feature, and the infrared output feature to obtain a concatenated feature;

[0189] Based on the image features and the splicing features, respectively, a mask image prediction and an infrared image prediction are performed to obtain a predicted mask image and a predicted infrared image;

[0190] Based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, the initial model is iterated on parameters to obtain the image translation model.

[0191] When the computer program is executed by a processor, the computer can perform the image translation method provided by the above methods, which includes:

[0192] Obtaining a visible light image to be translated;

[0193] Inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model;

[0194] The image translation model is constructed based on the image translation model construction method.

[0195] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the image translation model construction method provided by the above methods, the method comprising:

[0196] Obtain a training sample group, where any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image;

[0197] Extracting image features of the visible light sample image based on the encoding module in the initial model;

[0198] Acquire a prompt feature, a learnable mask output feature, and an infrared output feature, and concatenate the prompt feature, the mask output feature, and the infrared output feature to obtain a concatenated feature;

[0199] Based on the image features and the splicing features, respectively, a mask image prediction and an infrared image prediction are performed to obtain a predicted mask image and a predicted infrared image;

[0200] Based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, the initial model is iterated on parameters to obtain the image translation model.

[0201] When the computer program is executed by a processor, the computer program is implemented to perform the image translation method provided by the above methods, and the method includes:

[0202] Obtaining a visible light image to be translated;

[0203] Inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model;

[0204] The image translation model is constructed based on the image translation model construction method.

[0205] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0206] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an image translation model, characterized in that: include: Obtain a training sample group, where any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image; Extracting image features of the visible light sample image based on the encoding module in the initial model; Acquire a prompt feature, a learnable mask output feature, and an infrared output feature, and concatenate the prompt feature, the mask output feature, and the infrared output feature to obtain a concatenated feature; Based on the image features and the splicing features, respectively, a mask image prediction and an infrared image prediction are performed to obtain a predicted mask image and a predicted infrared image; Based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, performing parameter iteration on the initial model to obtain the image translation model; The performing mask image prediction and infrared image prediction based on the image features and the splicing features respectively to obtain a predicted mask image and a predicted infrared image respectively includes: Inputting the image features and the splicing features into a decoding module of the initial model, obtaining output image features and output splicing features output by the decoding module, and calculating the correlation between the output image features and the output splicing features; Performing mask image prediction based on the correlation, the output image features and a mask multi-layer perceptron to obtain the predicted mask image; Infrared image prediction is performed based on the correlation, the output image features and an infrared multi-layer perceptron to obtain the predicted infrared image.

2. The image translation model construction method according to claim 1, characterized in that: The step of performing infrared image prediction based on the correlation, the output image features and an infrared multi-layer perceptron to obtain the predicted infrared image includes: Based on the feature fusion module in the initial model, the image feature and the output image feature are fused to obtain a fused image feature; Based on the infrared multi-layer perceptron, the correlation is transformed to obtain an infrared prediction weight; Based on the infrared prediction weights, the fused image features are transformed to obtain the predicted infrared image.

3. The image translation model construction method according to claim 2, characterized in that: The encoding module includes a plurality of encoding layers, the image features include image features of each level output by each encoding layer, and the feature fusion module based on the initial model fuses the image features with the output image features to obtain fused image features, including: Based on the feature fusion module in the initial model, at least two levels of image features in the image features of each level are fused to obtain an initial fused feature; The initial fusion feature and the output image feature are fused to obtain the fusion image feature.

4. The method for constructing an image translation model according to claim 3, characterized in that: The step of performing parameter iteration on the initial model based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, to obtain the image translation model comprises: determining an infrared prediction loss based on a difference between the infrared tag image and the predicted infrared image; determining a mask prediction loss based on a difference between the mask label image and the predicted mask image; Based on the infrared prediction loss and the mask prediction loss, parameter iteration is performed on the learnable infrared output feature, the infrared multi-layer perceptron and the feature fusion module to obtain the image translation model.

5. The method for constructing an image translation model according to any one of claims 1 to 4, characterized in that: The step of determining the mask label image comprises: Inputting the visible light sample image into the SAM segmentation model, segmenting the visible light sample image to obtain a multi-value segmentation map; Based on an intersection-over-union ratio threshold and a stability score threshold, the multi-value segmentation map is converted into a binary segmentation map, and the binary segmentation map is used as the mask label image.

6. An image translation method, characterized in that: include: Obtaining a visible light image to be translated; Inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model; The image translation model is constructed based on the image translation model construction method according to any one of claims 1 to 5.

7. The image translation method according to claim 6, characterized in that: The step of inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model includes: Receive prompt information input by the user; The prompt information and the visible light image are input into an image translation model to obtain an infrared image and a mask image corresponding to the prompt information output by the image translation model.

8. An image translation model construction device, characterized in that: include: A sample group acquisition unit, used to acquire a training sample group, wherein any group of training samples includes a visible light sample image and its corresponding infrared label image, and a mask label image; A feature extraction unit, used for extracting image features of the visible light sample image based on the encoding module in the initial model; A feature splicing unit, used for obtaining a prompt feature, a learnable mask output feature and an infrared output feature, and splicing the prompt feature, the mask output feature and the infrared output feature to obtain a spliced ​​feature; An image prediction unit, used to perform mask image prediction and infrared image prediction based on the image features and the splicing features, respectively, to obtain a predicted mask image and a predicted infrared image, respectively; a parameter iteration unit, configured to perform parameter iteration on the initial model based on the difference between the infrared label image and the predicted infrared image, and the difference between the mask label image and the predicted mask image, so as to obtain the image translation model; The image prediction unit is specifically used for: Inputting the image features and the splicing features into a decoding module of the initial model, obtaining output image features and output splicing features output by the decoding module, and calculating the correlation between the output image features and the output splicing features; Performing mask image prediction based on the correlation, the output image features and a mask multi-layer perceptron to obtain the predicted mask image; Infrared image prediction is performed based on the correlation, the output image features and an infrared multi-layer perceptron to obtain the predicted infrared image.

9. An image translation device, characterized in that: include: An image acquisition unit, used for acquiring a visible light image to be translated; An image translation unit, used for inputting the visible light image into an image translation model to obtain an infrared image and a mask image output by the image translation model; The image translation model is constructed based on the image translation model construction method according to any one of claims 1 to 5.

Images