Super-resolution reconstruction method for infrared image, and construction method for super-resolution reconstruction model for infrared image

Abstract

Provided in the present disclosure is a super-resolution reconstruction method for an infrared image. The method comprises: acquiring first basic feature maps respectively corresponding to an auxiliary image of a target scene at N different preset resolutions, and acquiring second basic feature maps respectively corresponding to an original infrared image of the target scene at the N different preset resolutions, wherein N≥2 and is an integer, and the resolution of the auxiliary image is higher than the resolution of the original infrared image; and on the basis of the first basic feature maps and the second basic feature maps, performing feature map fusion and reconstruction, in order to obtain a super-resolution infrared image.

Description

Infrared image super-resolution reconstruction method and reconstruction model construction method TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of infrared image processing, and particularly relates to an infrared image super-resolution reconstruction method and a reconstruction model construction method. BACKGROUND

[0002] Due to the limitations of imaging mechanisms and detector sizes, the resolution of common infrared devices is difficult to meet the detection requirements, and the high cost limits the promotion and use of infrared devices. Therefore, an infrared image super-resolution reconstruction method is proposed in the related art. Specifically, the low-resolution infrared image itself is up-sampled (for example, interpolation processing, deconvolution processing, etc.) to reconstruct a high-resolution infrared image. However, it is found in actual application that the imaging accuracy of the reconstructed high-resolution infrared image is limited by the weak texture characteristics of the low-resolution infrared image itself, and the details are severely lost in the reconstruction process, which is quite different from the real high-resolution infrared image captured by a high-precision infrared device. SUMMARY

[0003] The present disclosure aims to at least solve one of the technical problems existing in the prior art, and proposes an infrared image super-resolution reconstruction method, a system, a model construction method, and a computer readable medium.

[0004] In a first aspect, an embodiment of the present disclosure provides an infrared image super-resolution reconstruction method, which comprises the following steps:

[0005] obtaining first basic feature maps corresponding to an auxiliary image of a target scene at N different preset resolutions, and obtaining second basic feature maps corresponding to an original infrared image of the target scene at the N different preset resolutions, N is an integer greater than or equal to 2, and the resolution of the auxiliary image is greater than that of the original infrared image;

[0006] performing feature map fusion reconstruction according to the first basic feature maps and the second basic feature maps to obtain a super-resolution infrared image.

[0007] In some embodiments, the step of performing feature map fusion reconstruction according to the first basic feature maps and the second basic feature maps comprises:

[0008] for each preset resolution, fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution to obtain a modal fusion feature map corresponding to the target preset resolution;

[0009] performing fusion reconstruction according to the modal fusion feature maps corresponding to the preset resolutions to obtain the super-resolution infrared image.

[0010] In some embodiments, the reconstruction method is configured with a first working mode;

[0011] In the first working mode, the step of fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution includes:

[0012] fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution according to an attention mechanism to obtain a first modal fusion feature map, the modal fusion feature map including the first modal fusion feature map.

[0013] In some embodiments, the reconstruction method is configured with a second working mode;

[0014] In the second working mode, the step of fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution includes:

[0015] fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution without setting an attention mechanism to obtain a second modal fusion feature map, the modal fusion feature map including the second modal fusion feature map.

[0016] In some embodiments, the reconstruction method is configured with a third working mode;

[0017] In the third working mode, the step of fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution to obtain a modal fusion feature map corresponding to the target preset resolution includes:

[0018] fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution according to an attention mechanism to obtain a first modal fusion feature map;

[0019] fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution without setting an attention mechanism to obtain a second modal fusion feature map;

[0020] The modal fusion feature map includes the first modal fusion feature map and the second modal fusion feature map.

[0021] In some embodiments, in the third working mode, the step of fusing the modal fusion feature maps corresponding to each preset resolution for reconstruction includes:

[0022] fusing and reconstructing the first modal fusion feature maps corresponding to each preset resolution to obtain corresponding first super-resolution infrared images, and fusing and reconstructing the second modal fusion feature maps corresponding to each preset resolution to obtain corresponding second super-resolution infrared images;

[0023] subtracting the second super-resolution infrared image from the first super-resolution infrared image to obtain a corresponding difference image, weighting the difference image according to a preset weight coefficient, and superimposing the weighted difference image on the first super-resolution infrared image to obtain a final super-resolution infrared image.

[0024] In some embodiments, the step of obtaining first base feature maps corresponding to the auxiliary image of the target scene at N different preset resolutions and obtaining second base feature maps corresponding to the original infrared image of the target scene at N different preset resolutions comprises:

[0025] obtaining first to-be-processed images corresponding to the auxiliary image at N different preset resolutions through sampling processing, and obtaining second to-be-processed images corresponding to the original infrared image at N different preset resolutions through sampling processing;

[0026] performing feature extraction processing on each first to-be-processed image to obtain the first base feature maps corresponding to the N different preset resolutions, respectively, and performing feature extraction processing on each second to-be-processed image to obtain the second base feature maps corresponding to the N different preset resolutions, respectively.

[0027] In some embodiments, the step of performing feature extraction on the first to-be-processed image corresponding to the target preset resolution comprises:

[0028] performing feature extraction on the first to-be-processed image corresponding to the target preset resolution in a first target domain and a second target domain, respectively, to obtain a corresponding first feature map and a second feature map, and the first base feature map comprises the first feature map and the second feature map.

[0029] In some embodiments, the step of performing feature extraction on the second to-be-processed image corresponding to the target preset resolution comprises:

[0030] performing feature extraction on the second to-be-processed image corresponding to the target preset resolution in a first target domain and a second target domain, respectively, to obtain a corresponding third feature map and a fourth feature map, and the second base feature map comprises the third feature map and the fourth feature map.

[0031] The first target domain and the second target domain are different domains, and the first target domain and the second target domain are selected from one of a frequency domain, a spatial domain, a color domain, and a gradient domain.

[0032] In some embodiments, the step of fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution comprises:

[0033] fusing the first feature map corresponding to the target preset resolution and the third feature map to obtain a first domain fusion feature map corresponding to the target preset resolution;

[0034] fusing the second feature map corresponding to the target preset resolution and the fourth feature map to obtain a second domain fusion feature map corresponding to the target preset resolution;

[0035] fusing the first domain fusion feature map corresponding to the target preset resolution and the second domain fusion feature map to obtain the modality fusion feature map.

[0036] In some embodiments, the step of obtaining, through sampling processing, the first to-be-processed image corresponding to the auxiliary image at N different preset resolutions comprises:

[0037] taking the auxiliary image as a first to-be-processed image, and performing N-1 times of down-sampling processing on the auxiliary image to obtain N-1 corresponding first to-be-processed images;

[0038] In some embodiments, the step of obtaining, through sampling processing, the second to-be-processed image corresponding to the original infrared image at N different preset resolutions comprises:

[0039] taking the original infrared image as a second to-be-processed image, and performing N-1 times of up-sampling processing on the original infrared image to obtain N-1 corresponding second to-be-processed images.

[0040] In some embodiments, in the case where the attention mechanism is provided, the step of fusing the two feature maps comprises:

[0041] calculating similarity data between the two feature maps according to the attention mechanism, the similarity data comprising similarity degrees corresponding to a plurality of different feature dimensions;

[0042] generating attention weight data according to the similarity degree data, the attention weight data comprising a plurality of attention weights corresponding one-to-one to the similarity degrees in the similarity degree data, the attention weight being positively correlated with the corresponding similarity degree;

[0043] According to the attention weight data, one of the two feature maps is image weighted to obtain a corresponding weighted guided feature map, and the weighted guided feature map is fused with the other of the two feature maps.

[0044] In some embodiments, the step of fusing and reconstructing according to the modal fusion feature maps corresponding to each preset resolution comprises:

[0045] For each preset resolution, the modal fusion feature map corresponding to the target preset resolution and the modal fusion feature map corresponding to the adjacent preset resolution adjacent in size to the target preset resolution are fused to obtain a scale fusion feature map corresponding to the target preset resolution.

[0046] According to the scale fusion feature maps corresponding to each preset resolution, the super-resolution infrared image is obtained by fusing and reconstructing.

[0047] In some embodiments, before the steps of obtaining the first base feature map and obtaining the second base feature map, the method further comprises:

[0048] obtaining an original high-resolution image of the target scene captured by a first shooting module and an original infrared image of the target scene captured by a second shooting module;

[0049] performing registration processing on the original high-resolution image and the original infrared image to extract the auxiliary image that is spatially aligned with the original infrared image from the original high-resolution image;

[0050] The reconstruction method is configured with a fourth working mode;

[0051] In the fourth working mode, after the step of performing registration processing on the original high-resolution image and the original infrared image, the method further comprises:

[0052] All element values in the matrix corresponding to the auxiliary image are replaced with 0.

[0053] In a second aspect, the embodiments of the present disclosure further provide a method for constructing an infrared image super-resolution reconstruction model, which comprises:

[0054] constructing an infrared image super-resolution reconstruction initial model, the infrared image super-resolution reconstruction initial model being configured to reconstruct a corresponding super-resolution infrared image by using the reconstruction method of any one of claims 1 to 19 based on the input auxiliary image and original infrared image;

[0055] The infrared image super-resolution reconstruction model is trained according to an initial model of the infrared image super-resolution reconstruction by using a pre-acquired sample set, the sample set includes a plurality of samples, each sample includes an auxiliary image, an original infrared image and a label image, the label image is another infrared image of a scene corresponding to the original infrared image, and the resolution of the label image is greater than the resolution of the original infrared image.

[0056] In some embodiments, in the process of training the initial model of the infrared image super-resolution reconstruction, the loss function configured by the initial model of the infrared image super-resolution reconstruction is as follows:

[0057] , wherein LOSS(H IR , TAG IR ) represents a loss value of the super-resolution infrared image H IR output by the initial model of the infrared image super-resolution reconstruction and the corresponding label image TAG IR , L MSE represents a pixel value mean square error between the super-resolution infrared image H IR and the corresponding label image TAG IR , L PSNR represents a peak signal-to-noise ratio between the super-resolution infrared image H IR and the corresponding label image TAG IR , L SSIM represents a structural similarity between the super-resolution infrared image H IR and the corresponding label image TAG IR , and α1, α2 and α3 are respectively pre-configured weight coefficients, and A is a pre-configured normalization coefficient.

[0058] In a third aspect, the embodiments of the present disclosure further provide an infrared image super-resolution reconstruction system, wherein the reconstruction system is configured to implement the reconstruction method provided in the first aspect, and the reconstruction system comprises:

[0059] a multi-scale feature acquisition module configured to acquire first basic feature maps corresponding to an auxiliary image of a target scene at N different preset resolutions, and to acquire second basic feature maps corresponding to an original infrared image of the target scene at N different preset resolutions, N≥2 and is an integer, and the resolution of the auxiliary image is greater than the resolution of the original infrared image;

[0060] a reconstruction module configured to perform feature map fusion reconstruction according to each of the first basic feature maps and each of the second basic feature maps to obtain a super-resolution infrared image.

[0061] In a fourth aspect, the present disclosure also provides a computer readable medium, and the computer readable medium stores a computer program, wherein the computer program is executed by a processor to implement the steps in the reconstruction method according to the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0062] FIG. 1 is a flowchart of an infrared image super-resolution reconstruction method according to an embodiment of the present disclosure;

[0063] FIG. 2 is a flowchart of another infrared image super-resolution reconstruction method based on the method shown in FIG. 1;

[0064] FIG. 3 is a flowchart of an optional implementation method of step S1 in the present disclosure;

[0065] FIG. 4 is a schematic diagram of a principle of performing fusion reconstruction based on N first to-be-processed images and N second to-be-processed images in the present disclosure;

[0066] FIG. 5 is a flowchart of an optional implementation method of step S2 in the present disclosure;

[0067] FIG. 6 is a flowchart of an optional implementation method of step S201 in the present disclosure;

[0068] FIG. 7 is a schematic diagram of a principle of modal fusion based on an attention mechanism in the present disclosure;

[0069] FIG. 8 is a flowchart of another optional implementation method of step S201 in the present disclosure;

[0070] FIG. 9 is a flowchart of yet another optional implementation method of step S201 in the present disclosure;

[0071] FIG. 10 is a flowchart of an optional implementation method of step S3 in the present disclosure;

[0072] FIG. 11 is a flowchart of an optional implementation method of step S301 in the present disclosure;

[0073] FIG. 12 is a flowchart of an optional implementation method of step S302 in the present disclosure;

[0074] FIG. 13 is a flowchart of another infrared image super-resolution reconstruction method according to an embodiment of the present disclosure;

[0075] FIG. 14 is a flowchart of yet another infrared image super-resolution reconstruction method according to an embodiment of the present disclosure;

[0076] FIG. 15 is a flowchart of a construction method of an infrared image super-resolution reconstruction model according to an embodiment of the present disclosure;

[0077] FIG. 16 is a structural block diagram of an infrared image super-resolution reconstruction system according to an embodiment of the present disclosure;

[0078] FIG. 17 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] For those skilled in the art to better understand the technical solutions of the present disclosure, the present disclosure will be further described in detail below in combination with the drawings and specific embodiments.

[0080] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the ordinary meaning understood by those skilled in the art to which the present disclosure pertains. The terms "first", "second", and similar terms used in the present disclosure do not denote any order, quantity, or importance, but are used to distinguish different components. Similarly, the terms "one", "a", or "the" and similar terms do not denote a quantity limitation, but mean that at least one exists. The terms "include", "comprise", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects.

[0081] In the various drawings, the same elements are denoted by similar reference numerals. For the sake of clarity, not all parts of the drawings are drawn to scale. In addition, some parts that are well known can not be shown in the drawings.

[0082] Many specific details of the present disclosure are described below in order to better understand the present disclosure. However, as those skilled in the art can understand, the present disclosure can be implemented without these specific details.

[0083] In the present disclosure, the sampling processing of an image refers to up-sampling processing or down-sampling processing of the image.

[0084] Among them, the up-sampling processing is used to improve the resolution of the image. Through up-sampling, an image larger (with increased resolution) than the original image can be obtained, but this process does not necessarily increase the details or amount of information of the image, because up-sampling essentially "speculates" the value of the newly added pixel according to the information of the existing pixel. The up-sampling methods that can be used in the present disclosure include but are not limited to: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, and transposed convolution.

[0085] Down-sampling processing is used to reduce the resolution of an image, by down-sampling a smaller (reduced resolution) image can be obtained, by reducing the number of pixels in the image to reduce the complexity and storage space requirements of the image. In the down-sampling process, the pixels in the original image are merged or discarded to generate a lower resolution version of the image. The down-sampling process can be lossy (i.e. some image details are lost), or lossless (although in practical applications, lossless down-sampling is usually less common, as it usually requires the image to have special properties or structure). The down-sampling methods that can be used in the present disclosure include but are not limited to: down-sampling nearest neighbor interpolation (Nearest Neighbor Interpolation), averaging method (Averaging or Mean Pooling), maximum value method (Max Pooling), region merging method (Region Merging).

[0086] In the present disclosure, the fusion of images refers to the process of synthesizing two or more images into one image according to the fusion algorithm; if the resolutions of the images to be fused are inconsistent, the images to be fused need to be sampled to achieve resolution uniformity before fusion, and then fusion is performed. The fusion algorithms that can be used in the present disclosure include but are not limited to: summation method, weighted summation method, weighted average method, maximum or minimum value selection method, feature fusion method.

[0087] In the present disclosure, "domain" (Domain) is a concept involved in the feature extraction processing process, which is used to represent the features of the image in different domains (attributes). Generally, the domains of the image include: spatial domain, frequency domain, gradient domain, color domain.

[0088] Among them, the spatial domain (Spatial Domain) feature refers to the feature obtained by feature extraction in the space composed of pixel values or signal values of the image; the frequency domain (Frequency Domain) feature refers to the feature obtained by feature extraction after the image is converted from the spatial domain to the frequency domain, the frequency domain image is represented as a combination of different frequency components, which reflect the speed of gray or color change in the image; the gradient domain (Gradient Domain) feature refers to the feature obtained by feature extraction after the image is converted from the spatial domain to the frequency domain, the gradient domain focuses on the size and direction of the brightness or color change of the pixels in the image, which uses the brightness or color change of the pixels in the local area of the image to describe the image features. The color domain (Color Domain) feature, also known as the color domain feature, refers to the feature obtained by feature extraction on the color information of the image, which is a global feature of the image, which describes the surface properties of the scene corresponding to the image or image region, and the color domain focuses on the color information of the image.

[0089] FIG. 1 is a flowchart of an infrared image super-resolution reconstruction method provided by an embodiment of the present disclosure. As shown in FIG. 1, the reconstruction method comprises the following steps.

[0090] In step Sa, a first basic feature map corresponding to each of N different preset resolutions of an auxiliary image of a target scene is obtained, and a second basic feature map corresponding to each of the N different preset resolutions of an original infrared image of the target scene is obtained.

[0091] Wherein, N is an integer greater than or equal to 2, and the resolution of the auxiliary image is greater than that of the original infrared image.

[0092] The first basic feature map of the auxiliary image at different scales (i.e. N different preset resolutions) is obtained through multi-scale feature extraction processing, that is, the detailed information of the auxiliary image at different scales is obtained. That is, there is a corresponding first basic feature map for each of the N preset resolutions.

[0093] The second basic feature map of the original infrared image at different scales is obtained through multi-scale feature extraction processing, that is, the detailed information of the original infrared image at different scales is obtained. That is, there is a corresponding second basic feature map for each of the N preset resolutions.

[0094] That is, there is a corresponding first basic feature map and a corresponding second basic feature map for each of the N preset resolutions.

[0095] It should be noted that the first / second basic feature map corresponding to the preset resolution described in the present disclosure refers to the first / second basic feature map with the preset resolution. In addition, in some embodiments, the first basic feature map corresponding to the preset resolution can be one feature map, two feature maps (see later embodiments), or multiple feature maps. Similarly, the second basic feature map corresponding to the preset resolution can be one feature map, two feature maps (see later embodiments), or multiple feature maps.

[0096] In the present disclosure, the auxiliary image can be a visible light image. A conventional visible light shooting device (for example, a mobile phone, a camera, a video camera, etc.) can shoot a visible light image with a relatively high resolution, so the visible light image can be used as the auxiliary image in the present disclosure.

[0097] In step Sb, feature map fusion reconstruction is performed according to each first basic feature map and each second basic feature map to obtain a super-resolution infrared image.

[0098] In the present disclosure, the first base feature map and the second base feature map corresponding to each preset resolution are fused through step Sb, that is, the infrared super-resolution reconstruction based on multi-scale (multiple preset resolutions) and multi-modal (the modal corresponding to the auxiliary image and the modal corresponding to the infrared image) is realized. Compared with the original infrared image, the auxiliary image has higher resolution and retains more detailed information of the target scene. The infrared image super-resolution reconstruction based on the high-resolution auxiliary image can make up for the lack of detail and texture information in the original infrared image, improve the effect of the reconstructed super-resolution infrared image, and improve the accuracy and rich details of the reconstructed super-resolution infrared image.

[0099] In some embodiments, each of the N different preset resolutions is greater than or equal to the resolution of the original infrared image and less than or equal to the resolution of the auxiliary image.

[0100] FIG. 2 is a flowchart of another infrared image super-resolution reconstruction method based on the method shown in FIG. 1. As shown in FIG. 2, the reconstruction method includes the following steps S1-S3.

[0101] Step S1, obtaining the first base feature map corresponding to the auxiliary image of the target scene at N different preset resolutions, and obtaining the second base feature map corresponding to the original infrared image of the target scene at N different preset resolutions.

[0102] Step S2, for each preset resolution, fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution to obtain a modal fusion feature map corresponding to the target preset resolution.

[0103] Step S3, fusing the modal fusion feature maps corresponding to each preset resolution to obtain the super-resolution infrared image.

[0104] In the embodiments of the present disclosure, the base feature maps of the auxiliary image and the original infrared image at different scales are fused to obtain the modal fusion feature maps corresponding to each scale, and then the modal fusion feature maps corresponding to each scale are fused and reconstructed, which is beneficial to improve the accuracy and rich details of the reconstructed super-resolution infrared image. The infrared image super-resolution reconstruction method provided in the embodiments of the present disclosure has the characteristics of high accuracy, rich details, high flexibility, strong robustness, etc.

[0105] In some embodiments, the reconstruction method is configured with a first working mode and a second working mode.

[0106] In the first working mode, step S2 specifically includes: fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution according to the attention mechanism to obtain a first modal fusion feature map; and the modal fusion feature map corresponding to the target preset resolution includes the first modal fusion feature map.

[0107] In the second working mode, step S2 specifically includes: fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution without setting the attention mechanism to obtain a second modal fusion feature map; and the modal fusion feature map corresponding to the target preset resolution includes the second modal fusion feature map.

[0108] Optionally, in some embodiments, the reconstruction method is configured with a third working mode. In the third working mode, step S32 specifically includes: fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution according to the attention mechanism to obtain a first modal fusion feature map; and fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution without setting the attention mechanism to obtain a second modal fusion feature map; and at this time, the modal fusion feature map corresponding to the target preset resolution includes the first modal fusion feature map and the second modal fusion feature map.

[0109] In the third working mode, step S3 specifically includes: first, fusing reconstruction is performed according to the first modal fusion feature map corresponding to each preset resolution to obtain a corresponding first super-resolution infrared image; and fusing reconstruction is performed according to the second modal fusion feature map corresponding to each preset resolution to obtain a corresponding second super-resolution infrared image; then, the second super-resolution infrared image and the first super-resolution infrared image are subjected to subtraction operation to obtain a corresponding difference map, the difference map is subjected to weighting processing according to a preset weight coefficient, and the weighted difference map is superimposed on the first super-resolution infrared image to obtain a final super-resolution infrared image.

[0110] In the first working mode and the third working mode, the step of fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution according to the attention mechanism to obtain the first modal fusion feature map includes: first, calculating similarity degree data between the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution according to the attention mechanism, the similarity degree data including: similarity degrees corresponding to a plurality of different feature dimensions; then, generating corresponding attention weight data according to the similarity degree data, the attention weight data including: a plurality of attention weights corresponding to the similarity degrees in the similarity degree data, the attention weight being positively correlated with the corresponding similarity degree; then, performing image weighting processing on the first base feature map corresponding to the target preset resolution according to the attention weight data to obtain a fusion guide image; and then, fusing the fusion guide image and the second base feature map corresponding to the target preset resolution to obtain the first modal fusion feature map.

[0111] In the present disclosure, for each dimension feature in the feature matrix of the first / second base feature map, the greater the corresponding similarity degree, the greater the attention weight given, and the smaller the corresponding similarity degree, the smaller the attention weight given. That is, similar features are enhanced by large weight values, and conflicting features are suppressed by small weight values.

[0112] FIG. 3 is a flow chart of an optional implementation method of step S1 in the embodiments of the present disclosure. As shown in FIG. 3, in some embodiments, step S1 includes:

[0113] Step S101: obtaining, by sampling processing, a first to-be-processed image corresponding to each of the N different preset resolutions of the auxiliary image.

[0114] Step S102: performing feature extraction processing on each first to-be-processed image to obtain a first base feature map corresponding to each of the N different preset resolutions.

[0115] Through the above steps S101 and S102, the first base feature map corresponding to each of the N different preset resolutions of the auxiliary image can be obtained.

[0116] Step S103: obtaining, by sampling processing, a second to-be-processed image corresponding to each of the N different preset resolutions of the original infrared image.

[0117] Step S104: performing feature extraction processing on each second to-be-processed image to obtain a second base feature map corresponding to each of the N different preset resolutions.

[0118] Through the above steps S103 and S104, the second base feature map corresponding to each of the N different preset resolutions of the original infrared image can be obtained.

[0119] It should be noted that the technical solution of the present disclosure does not limit the execution order of steps S101-S102 and steps S103-S104. It is only required to ensure that step S101 is executed before step S102, and step S103 is executed before step S104. As an example, steps S101 and S103 are executed synchronously first, and then steps S102 and S104 are executed synchronously.

[0120] In actual application, the corresponding feature extraction algorithm can be pre-set according to actual needs, such as color feature extraction algorithm, texture feature extraction algorithm, shape feature extraction algorithm, Local Binary Patterns (LBP) algorithm, Histogram of Oriented Gradient (HOG) algorithm, Scale-invariant feature transform (SIFT) algorithm, gradient domain feature extraction algorithm, frequency domain feature extraction algorithm, etc. It should be noted that the selected feature extraction algorithm can be one or more.

[0121] In addition, a pre-designed convolutional neural network (CNN) can also be used to perform feature extraction processing on the first and second to-be-processed images. Convolutional neural network is a neural network specially designed for processing data with grid structure, such as image data. They extract the scale-invariant features (also known as scale-invariant features) in the image through a series of convolutional layers, activation layers, pooling layers, and fully connected layers.

[0122] In some embodiments, the maximum preset resolution in the N different preset resolutions is equal to the resolution of the auxiliary image, and the minimum preset resolution in the N different preset resolutions is equal to the resolution of the original infrared image.

[0123] For example, the resolution of the original infrared image is W1*H1, and the resolution of the auxiliary image is WN*HN. At this time, the N different preset resolutions in order of resolution from small to large are: W1*H1, W2*H2, …, WN*HN, W1

[0124] FIG. 4 is a schematic diagram of a principle of performing fusion reconstruction based on N first to-be-processed images and N second to-be-processed images in an embodiment of the present disclosure. As shown in FIG. 4.

[0125] As a specific optional implementation of step S101, the auxiliary image is taken as a first to-be-processed image, and N-1 first to-be-processed images are obtained by performing N-1 times of down-sampling based on the auxiliary image.

[0126] The first to-be-processed image obtained by the first time of down-sampling is obtained by performing down-sampling on the auxiliary image, the first to-be-processed image obtained by the i th time of down-sampling is obtained by performing down-sampling on the first to-be-processed image obtained by the (i-1) th time of down-sampling, 2≤i≤N-1 and i is an integer.

[0127] That is, by performing N-1 times of gradual down-sampling based on the auxiliary image, the first to-be-processed images corresponding to the auxiliary image at N different preset resolutions can be obtained, and the resolution of the first to-be-processed image obtained by the N-1 th time of down-sampling (i.e., the first to-be-processed image with the smallest resolution) is the same as the resolution of the original infrared image.

[0128] As a specific optional implementation of step S103, the original infrared image is taken as a second to-be-processed image, and N-1 second to-be-processed images are obtained by performing N-1 times of up-sampling based on the original infrared image.

[0129] The second to-be-processed image obtained by the first time of up-sampling is obtained by performing up-sampling on the auxiliary image, the second to-be-processed image obtained by the j th time of up-sampling is obtained by performing up-sampling on the second to-be-processed image obtained by the (j-1) th time of up-sampling, 2≤j≤N-1 and j is an integer.

[0130] That is, by performing N-1 times of gradual up-sampling based on the auxiliary image, the second to-be-processed images corresponding to the original infrared image at N different preset resolutions can be obtained, and the resolution of the second to-be-processed image obtained by the N-1 th time of up-sampling (i.e., the second to-be-processed image with the largest resolution) is the same as the resolution of the auxiliary image.

[0131] In the embodiment of the present disclosure, by gradually up-sampling the infrared image and gradually down-sampling the auxiliary image, the details of different scales are preserved as much as possible by gradually guiding from coarse to fine scales, which is especially suitable for super-resolution reconstruction tasks with large multiples.

[0132] In the embodiments of the present disclosure, the greater the value of N is, the more the detail information of different scales is, and the higher the accuracy and the more the details of the finally reconstructed super-resolution infrared image are. However, the operation complexity and the data processing amount in the running process are increased accordingly, and the running efficiency is reduced. Considering the imaging accuracy and the running efficiency, the value of N in the present disclosure satisfies: 2≤N≤6.

[0133] It should be noted that, in the case of continuously downsampling and extracting features of the auxiliary image, and continuously upsampling and extracting features of the original infrared image to obtain the first basic feature map and the second basic feature map corresponding to the N preset resolutions respectively, as shown in FIG. 4, it is only one optional embodiment in the present disclosure.

[0134] In addition, the maximum preset resolution in the above-mentioned N different preset resolutions is equal to the resolution of the auxiliary image, and the minimum preset resolution in the N different preset resolutions is equal to the resolution of the original infrared image, which is also only one optional embodiment in the present disclosure. In the present disclosure, the maximum preset resolution can also be less than the resolution of the auxiliary image, and the minimum preset resolution can also be greater than the resolution of the original infrared image. The present disclosure does not limit this.

[0135] In some embodiments, as a specific optional implementation of step S102, the step of performing feature extraction on the first to-be-processed image corresponding to the target preset resolution includes: performing feature extraction on the first to-be-processed image corresponding to the target preset resolution in a first target domain to obtain a corresponding first feature map, and performing feature extraction on the first to-be-processed image corresponding to the target preset resolution in a second target domain to obtain a corresponding second feature map. At this time, the first basic feature map corresponding to the target preset resolution includes the first feature map and the second feature map.

[0136] As a specific optional implementation of step S104, the step of performing feature extraction on the second to-be-processed image corresponding to the target preset resolution includes: performing feature extraction on the second to-be-processed image corresponding to the target preset resolution in the first target domain to obtain a corresponding third feature map, and performing feature extraction on the second to-be-processed image corresponding to the target preset resolution in the second target domain to obtain a corresponding fourth feature map. At this time, the second basic feature map corresponding to the target preset resolution includes the third feature map and the fourth feature map.

[0137] The first target domain and the second target domain are different domains, and the first target domain and the second target domain are respectively selected from one of a frequency domain, a spatial domain, a color domain and a gradient domain.

[0138] Correspondingly, the step of fusing the first base feature map corresponding to the target preset resolution and the second base feature map corresponding to the target preset resolution to obtain the first modal fusion feature map comprises: fusing the first feature map corresponding to the target preset resolution and the third feature map to obtain the first domain fusion feature map corresponding to the target preset resolution; fusing the second feature map corresponding to the target preset resolution and the fourth feature map to obtain the second domain fusion feature map corresponding to the target preset resolution; and fusing the first domain fusion feature map corresponding to the target preset resolution and the second domain fusion feature map (generally, the first domain fusion feature map and the second domain fusion feature map are first mapped to the same domain, and then fused) to obtain the modal fusion feature map.

[0139] For the convenience of those skilled in the art to better understand the technical solutions of the present disclosure, the following will be exemplarily described in conjunction with specific examples. Among them, the first target domain is the gradient domain, and the second target domain is the frequency domain. The first feature map is referred to as the first gradient domain feature map, the second feature map is referred to as the first frequency domain feature map, the third feature map is referred to as the second gradient domain feature map, and the fourth feature map is referred to as the second frequency domain feature map. At this time, the first base feature map includes: the first gradient domain feature map and the first frequency domain feature map; and the second base feature map includes: the second gradient domain feature map and the second frequency domain feature map.

[0140] FIG. 5 is a flow chart of an optional implementation method of step S2 in the embodiment of the present disclosure. As shown in FIG. 5, in some embodiments, step S2 comprises:

[0141] Step S201, fusing the first gradient domain feature map corresponding to the target preset resolution and the second gradient domain feature map to obtain the gradient fusion feature map corresponding to the target preset resolution (i.e., the first domain fusion feature map); and fusing the first frequency domain feature map corresponding to the target preset resolution and the second frequency domain feature map to obtain the frequency fusion feature map corresponding to the target preset resolution (i.e., the second domain fusion feature map).

[0142] Step S202, fusing the gradient fusion feature map corresponding to the target preset resolution and the frequency fusion feature map to obtain the modal fusion feature map.

[0143] For the convenience of those skilled in the art to better understand the technical solutions of the present disclosure, the following will be exemplarily described in conjunction with specific examples. Among them, N different preset resolutions in order of resolution from small to large are: W1*H1, W2*H2, …, WN*HN, and N first to-be-processed images are respectively denoted as VIS1, VIS2, …, VISN, and N second to-be-processed images are respectively denoted as IR1, IR2, …, IRN. The nth first to-be-processed image VISn and the nth second to-be-processed image IRn have the same resolution Wn*Hn, n∈[1, N] and n is an integer.

[0144] In step S102, feature extraction is performed on the nth first to-be-processed image VISn to obtain corresponding first gradient domain feature map G VISn and first frequency domain feature map F VISn In step S104, feature extraction is performed on the nth second to-be-processed image IRn to obtain corresponding second gradient domain feature map G IRn and first frequency domain feature map F IRn , which can be expressed as: G VISn = E G (f gt (VISn)) F VISn = E F (f ft (VISn)) G IRn = E G (f gt (IRn)) F IRn = E F (f ft (IRn))

[0145] wherein f gt () represents a gradient domain conversion function (characterizing conversion of an image to a gradient domain), E G () represents a gradient domain feature extraction function (characterizing feature extraction processing performed on the image converted to the gradient domain), f ft () represents a frequency domain conversion function (characterizing conversion of an image to a frequency domain), and E F () represents a frequency domain feature extraction function (characterizing feature extraction processing performed on the image converted to the frequency domain).

[0146] In step S201, the first gradient domain feature map G VISn and the second gradient domain feature map G IRn are fused to obtain a gradient fusion feature map F Gn corresponding to the nth preset resolution Wn*Hn, and the first frequency domain feature map and the second frequency domain feature map are fused to obtain a frequency fusion feature map F Fn corresponding to the nth preset resolution Wn*Hn, which can be expressed as: F Gn = fusion G (G VISn , G IRn ) F Fn = fusion F (F VISn , F IRn )

[0147] fusion Gdenotes a gradient domain feature map fusion processing function, fusion F denotes a frequency domain feature map fusion processing function.

[0148] In step S202, one gradient fusion feature map F Gn and one frequency fusion feature map F Fn are fused in the following two ways.

[0149] Way one, the gradient fusion feature map F Gn is converted to the frequency domain to obtain a gradient fusion frequency domain feature map f ft (F Gn ), and the gradient fusion frequency domain feature map f ft (F Gn ) is superimposed and fused with the frequency fusion feature map F Fn to obtain a modality fusion feature map F fusion _n corresponding to the nth preset resolution, which can be expressed as follows:

[0150] , wherein denotes superimposed fusion processing of the image.

[0151] Way two, the frequency fusion feature map F Fn is converted to the gradient domain to obtain a frequency fusion gradient domain feature map f gt (F Fn ), and the frequency fusion gradient domain feature map f gt (F Fn ) is fused with the gradient fusion feature map F Gn to obtain a modality fusion feature map F fusion _n corresponding to the nth preset resolution, which can be expressed as follows:

[0152] FIG. 6 is a flowchart of an optional implementation method of step S201 in the embodiment of the present disclosure. FIG. 7 is a schematic diagram of one principle of modality fusion based on an attention mechanism in the embodiment of the present disclosure. As shown in FIGS. 6 and 7, step S201 includes:

[0153] Step S2011a, according to the attention mechanism, calculating first similarity degree data between a first gradient domain feature map and a second gradient domain feature map, and generating corresponding first attention weight data according to the first similarity degree data.

[0154] The first similarity degree data includes first similarities corresponding to multiple different feature dimensions, and the first attention weight data includes multiple first attention weights corresponding one-to-one to the first similarities in the first similarity degree data, and the first attention weight is positively correlated with the corresponding first similarity.

[0155] In step S2011a, first, the first gradient domain feature map and the second gradient domain feature map are respectively subjected to feature extraction according to the selected attention mechanism, to obtain a feature matrix corresponding to the first gradient domain feature map (generally including multiple feature dimensions, determined by the specific selected attention mechanism) and a feature matrix corresponding to the second gradient domain feature map; then, the similarity degree of the feature matrix corresponding to the first gradient domain feature map and the feature matrix corresponding to the first gradient domain feature map in each feature dimension is calculated, so that a similarity degree data (i.e. first similarity degree data, essentially a similarity degree matrix) can be obtained, which includes the similarity degrees corresponding to multiple feature dimensions.

[0156] In the embodiments of the present disclosure, the selected attention mechanism includes but is not limited to: channel attention mechanism (Channel Attention Mechanism, abbreviated as CAM), spatial attention mechanism (Spatial Attention Mechanism, abbreviated as SAM).

[0157] After obtaining the first similarity degree data, the corresponding first attention weight data (i.e. first attention weight data, essentially an attention weight matrix) can be obtained by normalizing the first similarity degree data (for example, using a softmax function for normalization).

[0158] In the present disclosure, for each dimension feature in the feature matrix of the first / second gradient domain feature map, the greater the corresponding similarity degree, the greater the attention weight given, and the smaller the corresponding similarity degree, the smaller the attention weight given. That is, by large weight value, the similar features are enhanced, and by small weight value, the conflicting features are suppressed.

[0159] Step S2012a, image weighting processing is performed on the first gradient domain feature map according to the first attention weight data, to obtain a gradient domain weighted guided feature map corresponding to the first gradient domain feature map.

[0160] Through step S2012a, the gradient domain features of the auxiliary image can be realized as guided information.

[0161] Step S2013a, the gradient domain weighted guided feature map and the second gradient feature map are fused to obtain a first gradient fusion feature map as a gradient fusion feature map corresponding to the target preset resolution.

[0162] Through step S2013a, the guided information can be realized to be superimposed on the second gradient feature map.

[0163] The process of obtaining the gradient fusion feature map F corresponding to the nth preset resolution Wn*Hn through steps S2011a-S2013a can be represented by the following formula: Gn

[0164] Atten (G VISn , G IRn ) represents first similarity degree data calculated based on an attention mechanism of the first gradient domain feature map G VISn and the second gradient domain feature map G IRn , and first attention weight data obtained according to the first similarity degree data. represents an element-wise multiplication operation (i.e., the aforementioned image weighting processing). The operation result of Atten (G VISn , G IRn ) is used as guide information for superimposed fusion with the second gradient domain feature map G IRn .

[0165] Step S2014a: calculating second similarity degree data between the first frequency domain feature map and the second frequency domain feature map according to an attention mechanism, and generating corresponding second attention weight data according to the second similarity degree data.

[0166] The second similarity degree data includes second similarities corresponding to multiple different feature dimensions, and the second attention weight data includes multiple second attention weights corresponding one-to-one to the second similarities in the second similarity degree data, and the second attention weight is positively correlated with the corresponding second similarity.

[0167] Step S2015a: performing image weighting processing on the first frequency domain feature map according to the second attention weight data, to obtain a frequency domain weighted guide feature map corresponding to the first frequency domain feature map.

[0168] Step S2016a: fusing the frequency domain weighted guide feature map with the second frequency feature map to obtain a first frequency fusion feature map as a frequency fusion feature map corresponding to the target preset resolution.

[0169] Similar to the steps S2011a-S2013a, in the steps S2014a-S2016a, the attention mechanism is introduced when the frequency feature map is fused, and the frequency domain feature of the auxiliary image is used as guide information.

[0170] The process of obtaining the frequency fusion feature map F corresponding to the nth preset resolution Wn*Hn through steps S2014a-S2016a can be represented by the following formula: Gn

[0171] Atten (FVISn , F IRn ) represents the second similarity degree data calculated based on the attention mechanism of the first frequency domain feature map G VISn and the second frequency domain feature map G IRn and the second attention weight data obtained according to the second similarity degree data; represents an element-wise multiplication operation (i.e., the aforementioned weighting processing); The operation result of F IRn is taken as the guide information for superimposed fusion with the second frequency domain feature map F Gn .

[0172] In the present disclosure, by taking the gradient domain features and the frequency domain features of the auxiliary image as the guide information, the conflict between the two modalities is reduced; by setting the attention mechanism in the multi-modal fusion process, the loyalty of the guide information to the infrared image can be controlled.

[0173] FIG. 8 is a flow chart of another optional implementation method of step S201 in the embodiment of the present disclosure. As shown in FIG. 8, the first working mode shown in FIG. 6 is essentially a working mode considering the modal conflict; the present embodiment also provides a working mode not considering the modal conflict, which is referred to as the second working mode.

[0174] In some embodiments, the reconstruction method is configured with the second working mode; in the second working mode, step S201 includes:

[0175] Step S2011b, fuse the first gradient domain feature map and the second gradient domain feature map without setting the attention mechanism to obtain a second gradient fusion feature map as the gradient fusion feature map corresponding to the target preset resolution.

[0176] Step S2012b, fuse the first frequency domain feature map and the second frequency domain feature map without setting the attention mechanism to obtain a second frequency fusion feature map as the frequency fusion feature map corresponding to the target preset resolution.

[0177] Step S2011b can be expressed by the following formula: F Gn =fusion G (G VISn , G IRn )=F(G VISn , G IRn )

[0178] Step S2012b can be expressed by the following formula: F Fn =fusion F (F VISn , F IRn )=F(F VISn , F IRn )

[0179] wherein F() represents a feature map fusion processing without setting an attention mechanism, including but not limited to weighted summation fusion processing, maximum value selection fusion processing, average value fusion processing, etc.

[0180] FIG. 9 is a flow chart of another optional implementation method of step S201 in the embodiment of the present disclosure. As shown in FIG. 9, different from the working mode considering the modal conflict (i.e. the first working mode) in FIG. 6 and the working mode not considering the modal conflict (i.e. the second working mode) in FIG. 8, the embodiment further provides a working mode adjusting the conflict weight, i.e. the third working mode.

[0181] In the third working mode, step S201 includes steps S2011a-S2016a in FIG. 6 and steps S2011b-S2012b in FIG. 8. At this time, the gradient fusion feature maps corresponding to the target preset resolution include the first gradient fusion feature map and the second gradient fusion feature map. The frequency fusion feature maps corresponding to the target preset resolution include the first frequency fusion feature map and the second frequency fusion feature map.

[0182] In the third working mode, step S202 includes: fusing the first gradient fusion feature map and the first frequency fusion feature map to obtain the first modal fusion feature map; and fusing the second gradient fusion feature map and the second frequency fusion feature map to obtain the second modal fusion feature map.

[0183] At this time, the modal fusion feature maps corresponding to the target preset resolution include the first modal fusion feature map and the second modal fusion feature map.

[0184] Correspondingly, in the third working mode, step S3 includes: firstly, fusing and reconstructing according to the first modal fusion feature maps corresponding to each preset resolution to obtain the corresponding first super-resolution infrared image; and fusing and reconstructing according to the second modal fusion feature maps corresponding to each preset resolution to obtain the corresponding second super-resolution infrared image; then, performing subtraction operation on the second super-resolution infrared image and the first super-resolution infrared image to obtain the corresponding difference image, performing weighting processing on the difference image according to the preset weight coefficient, and superimposing the weighted difference image on the first super-resolution infrared image to obtain the final super-resolution infrared image. At this time, the final super-resolution infrared image H IR may be expressed by the following formula. H IR = H model1 + β (H model2 - H model1 )

[0185] wherein H model1 represents the first super-resolution infrared image, H model2denotes a second super-resolution infrared image, H model2 -H model1 denotes a difference map, and β denotes the preset weight coefficient.

[0186] In some embodiments, the preset weight coefficient β is configured to be adjustable. That is, the user can dynamically adjust the value of β according to needs.

[0187] It should be noted that the first super-resolution infrared image H model1 may be regarded as a super-resolution infrared image obtained by step S3 in the first working mode (considering modal conflicts), and the second super-resolution infrared image H model2 may be regarded as a super-resolution infrared image obtained by step S3 in the second working mode (without considering modal conflicts).

[0188] In the embodiments of the present disclosure, the reconstruction method can be configured with at least one of the first working mode, the second working mode, and the third working mode. When the reconstruction method is configured with at least two of the first working mode, the second working mode, and the third working mode, the user can select the corresponding working mode according to the use scenario, that is, the reconstruction method of the present disclosure can support switching of the working mode.

[0189] FIG. 10 is a flow chart of an optional implementation method of step S3 in the embodiments of the present disclosure. As shown in FIG. 10, in some embodiments, step S3 includes:

[0190] Step S301, for each preset resolution, fusing the modal fusion feature map corresponding to the target preset resolution and the modal fusion feature map corresponding to the adjacent preset resolution adjacent in size to the target preset resolution, to obtain a scale fusion feature map corresponding to the target preset resolution.

[0191] In the present disclosure, for each scale resolution, the modal fusion feature map of the scale resolution is fused with the modal fusion feature map of the adjacent scale resolution (i.e., the adjacent preset resolution), to obtain a corresponding scale fusion feature map, realizing bidirectional fusion of adjacent scale features, and improving the sensitivity of the model to different size information.

[0192] It should be noted that when the target preset resolution is the minimum preset resolution, it has only one adjacent preset resolution, i.e., the second smallest preset resolution; when the target preset resolution is the maximum preset resolution, it has only one adjacent preset resolution, i.e., the second largest preset resolution. When the target preset resolution is neither the minimum preset resolution nor the maximum preset resolution, it has two adjacent preset resolutions.

[0193] FIG. 11 is a flowchart of an optional implementation method of step S301 in the embodiments of the present disclosure. As shown in FIG. 11, in some embodiments, step S301 includes step S3011 and step S3012.

[0194] Step S3011: sample each of the modality fusion feature maps corresponding to the adjacent preset resolutions to obtain modality fusion sampling feature maps corresponding to the adjacent preset resolutions, the resolution of the modality fusion sampling feature map being the same as that of the modality fusion feature map corresponding to the target preset resolution.

[0195] Step S3012: fuse the modality fusion sampling feature maps corresponding to the adjacent preset resolutions with the modality fusion feature map corresponding to the target preset resolution to obtain the scale fusion feature map.

[0196] For better understanding of the present disclosure by those skilled in the art, an exemplary description is given below with N≥3.

[0197] When n is 1, the target preset resolution is the first preset resolution W1*H1 (i.e., the minimum preset resolution), and there is only one adjacent preset resolution, i.e., the second preset resolution W2*H2. At this time, the modality fusion feature map F fusion _1 corresponding to the first preset resolution is fused with the modality fusion feature map F fusion _2 corresponding to the adjacent preset resolution to obtain the scale fusion feature map F out1 _1 corresponding to the first preset resolution W1*H1, which can be expressed as follows: F out1 _1 = MSF[F fusion _1, DOWN_1(F fusion _2)]

[0198] Wherein, MSF[] represents a multi-scale fusion processing function, and DOWN_1() represents down-sampling a picture to the first preset resolution W1*H1.

[0199] When 2≤n≤N-1, the target preset resolution is the nth preset resolution Wn*Hn (i.e., a non-minimum preset resolution and a non-maximum preset resolution), and there are two adjacent preset resolutions, i.e., the (n-1)th preset resolution Wn-1*Hn-1 and the (n+1)th preset resolution Wn+1*Hn+1. At this time, the modality fusion feature map F fusion _n corresponding to the nth preset resolution is fused with the modality fusion feature maps F fusion _n-1 and F fusion _n+1 corresponding to the adjacent preset resolutions to obtain the scale fusion feature map F outn _1 corresponding to the nth preset resolution Wn*Hn, which can be expressed as follows: F outn= MSF [UP_n (F fusion _n-1), F fusion _n, DOWN_n (F fusion _n+1)]

[0200] wherein UP_n() represents up-sampling the picture to the nth preset resolution Wn*Hn, and DOWN_n() represents down-sampling the picture to the nth preset resolution Wn*Hn.

[0201] When n takes the value of N, the target preset resolution is the Nth preset resolution WN*HN (i.e., the maximum preset resolution), and the corresponding adjacent preset resolution is only one, i.e., the (N-1)th preset resolution WN-1*HN-1. At this time, the modal fusion feature map F fusion _N corresponding to the Nth preset resolution is fused with the modal fusion feature map F fusion _N-1 corresponding to the adjacent preset resolution to obtain the scale fusion feature map F out1 corresponding to the Nth preset resolution WN*HN, which can be expressed as: F outN = MSF [UP_N (F fusion _N-1), F fusion _N]

[0202] wherein UP_N(F fusion _N-1) represents up-sampling the picture to the Nth preset resolution WN*HN.

[0203] Step S302: fusing and reconstructing according to the scale fusion feature maps corresponding to each preset resolution to obtain the super-resolution infrared image.

[0204] It should be noted that in the third working mode, step S301 specifically includes: for each preset resolution, firstly, fusing the first modal fusion feature map corresponding to the target preset resolution and the first modal fusion feature map corresponding to the adjacent preset resolution adjacent to the target preset resolution in size to obtain the first scale fusion feature map corresponding to the target preset resolution; and then fusing the second modal fusion feature map corresponding to the target preset resolution and the second modal fusion feature map corresponding to the adjacent preset resolution adjacent to the target preset resolution in size to obtain the second scale fusion feature map corresponding to the target preset resolution. At this time, the scale fusion feature map corresponding to the target preset resolution includes the first scale fusion feature map and the second scale fusion feature map.

[0205] Correspondingly, in the third working mode, the step S302 specifically comprises: firstly, performing fusion reconstruction on the first-degree fusion feature maps corresponding to each preset resolution to obtain a first super-resolution infrared image; then, performing fusion reconstruction on the second-degree fusion feature maps corresponding to each preset resolution to obtain a second super-resolution infrared image; then, performing subtraction operation on the second super-resolution infrared image and the first super-resolution infrared image to obtain a corresponding difference image, and performing weighting processing on the difference image according to a preset weight coefficient; finally, superimposing the weighted difference image on the first super-resolution infrared image to obtain a final super-resolution infrared image.

[0206] FIG. 12 is a flow chart of an optional implementation method of the step S302 in the embodiments of the present disclosure. As shown in FIG. 12, in some embodiments, the step S302 comprises:

[0207] The step S3021 comprises: performing fusion processing on the scale fusion feature maps corresponding to each preset resolution to obtain a scale comprehensive fusion feature map.

[0208] Specifically, each scale fusion feature map with a resolution less than the maximum preset resolution is first up-sampled to the maximum preset resolution, and then the N scale fusion feature maps with the maximum preset resolution are fused to obtain the scale comprehensive fusion feature map.

[0209] The step S3022 comprises: fusing the scale comprehensive fusion feature map with the second to-be-processed image with the maximum preset resolution to obtain a super-resolution infrared image, and the resolution of the super-resolution infrared image is equal to the maximum preset resolution.

[0210] In the present disclosure, the scale comprehensive fusion feature map is fused with the second to-be-processed image with the maximum preset resolution to ensure that as much information as possible from the original infrared image is retained in the super-resolution infrared image.

[0211] Of course, in some embodiments, the step S3022 can not be performed after the step S3021 is completed, and in this case, the scale comprehensive fusion feature map obtained in the step S3021 can be directly used as the super-resolution infrared image. Such a case should also be within the protection scope of the present disclosure.

[0212] FIG. 13 is a flow chart of another infrared image super-resolution reconstruction method provided by the embodiments of the present disclosure. As shown in FIG. 13, before the step S1, there further comprises:

[0213] The step S01 comprises: obtaining an original high-resolution image of a target scene captured by a first shooting module and an original infrared image of the target scene captured by a second shooting module, and the resolution of the original high-resolution image is greater than the resolution of the original infrared image.

[0214] Step S02, performing registration processing on the original high-resolution image and the original infrared image, so as to extract the auxiliary image which is spatially aligned with the original infrared image from the original high-resolution image.

[0215] In the present disclosure, registration of images refers to spatially aligning two images so as to achieve spatial or geometric consistency between them.

[0216] For example, the first photographing module is a visible light photographing module, and the second photographing module is an infrared photographing module. Generally, the photographing field of view of the infrared photographing module (i.e., the second photographing module) is relatively small, while the photographing field of view of other visible light photographing modules is relatively large. When the visible light photographing module and the infrared photographing module are used to photograph a target scene at the same position, the scene area in the original visible light image captured by the visible light photographing module is larger than the scene area in the original infrared image captured by the infrared photographing module. Therefore, the original infrared image is taken as a reference, and the original visible light image is registered with the original infrared image, that is, a part of the original visible light image having the same scene area as the original infrared image is extracted as an auxiliary image.

[0217] In the registration process, the original infrared image is first up-sampled to a resolution of the final desired super-resolution infrared image (referred to as a desired resolution), and then the original high-resolution image is aligned with the infrared image which has been up-sampled and has the desired resolution.

[0218] In addition, it should be noted that, in the registration process of the original high-resolution image with the original infrared image taken as a reference, the resolution of the original visible light image can also be adjusted (for example, the original visible light image is down-sampled to appropriately reduce the resolution) according to actual needs. That is, in the present disclosure, the resolution of the extracted auxiliary image can be less than or equal to the resolution of the original high-resolution image output by the first photographing module.

[0219] FIG. 14 is a flowchart of another infrared image super-resolution reconstruction method provided by an embodiment of the present disclosure. As shown in FIG. 14, in some embodiments, the reconstruction method is configured with a fourth working mode. After step S02, step S03 is further included.

[0220] Step S03, detecting whether the current is running in the fourth working mode.

[0221] If it is detected that the current is running in the fourth working mode, step S04 is executed; if it is detected that the current is not running in the fourth working mode, step S1 is executed.

[0222] Step S04, replacing all element values in the matrix corresponding to the auxiliary image with 0.

[0223] After step S04 is finished, step S1 is executed.

[0224] In the present disclosure, the four working modes meet the needs of users in different scenarios. Under normal circumstances, the first working mode (the working mode considering the modal conflict) is started, and the infrared super-resolution reconstructed image containing rich texture information is observed; when the comprehensiveness of the observed target is considered, the second working mode (the working mode without considering the modal conflict) is started, which is similar to the dual-light fusion effect, and the target information in the image is maximized; when the infrared temperature information and the target texture information need to be considered comprehensively, the third working mode (the working mode with adjustable conflict weight) is started, and the weight β is adjusted to realize different superposition effects; in the case that the visible light image quality is poor at night or the like, the fourth working mode (the working mode without guided information) is started to ensure the infrared super-resolution reconstruction effect.

[0225] Based on the same inventive concept, the present disclosure also provides a method for constructing an infrared image super-resolution reconstruction model. FIG. 15 is a flowchart of a method for constructing an infrared image super-resolution reconstruction model according to an embodiment of the present disclosure. As shown in FIG. 15, the method includes the following steps:

[0226] BZ1, an initial infrared image super-resolution reconstruction model is constructed, which is configured to reconstruct a corresponding super-resolution infrared image based on the input auxiliary image and the original infrared image by using the reconstruction method provided in the foregoing embodiments.

[0227] BZ2, the initial infrared image super-resolution reconstruction model is trained according to a pre-acquired sample set to obtain an infrared image super-resolution reconstruction model, the sample set includes a plurality of samples, each sample includes an auxiliary image, an original infrared image and a label image, the label image is another infrared image of the scene corresponding to the original infrared image, and the resolution of the label image is greater than that of the original infrared image.

[0228] The label image can be a high-definition infrared image of the corresponding target scene acquired by a high-definition infrared camera device.

[0229] In some embodiments, in the process of training the initial infrared image super-resolution reconstruction model, the loss function configured by the initial infrared image super-resolution reconstruction model (the purpose of model training is to minimize the value of the loss function) is as follows:

[0230] wherein, LOSS(H IR , TAG IR ) represents the super-resolution infrared image H IRloss value of the corresponding label image TAG IR , L MSE represents the pixel value mean square error between the super-resolution infrared image H IR and the corresponding label image TAG IR , L PSNR represents the peak signal-to-noise ratio between the super-resolution infrared image H IR and the corresponding label image TAG IR , L SSIM represents the structural similarity degree between the super-resolution infrared image H IR and the corresponding label image TAG IR , and α1, α2 and α3 are respectively pre-configured weight coefficients, and A is a pre-set normalization coefficient for mapping L PSNR to the range of [0, 1] (for example, A takes the value of 40).

[0231] In some embodiments, the pixel value mean square error L MSE :

[0232] wherein, represents the pixel value of the jth pixel in the label image TAG IR , and represents the pixel value of the jth pixel in the super-resolution infrared image H IR , 1≤j≤J and j is an integer, and J is the total number of pixels in the super-resolution infrared image.

[0233] In some embodiments, the peak signal-to-noise ratio L PSNR :

[0234] wherein Maxvalue is a constant representing the maximum value that the image pixel can take; for example, for float-type image data, the value range of the image pixel is [0, 1], and at this time, Maxvalue takes the value of 1; for uint8-type image data, the value range of the image pixel is [0, 255], and at this time, Maxvalue takes the value of 255.

[0235] The structural similarity degree L SSIM can be obtained by a structural similarity index (SSIM) algorithm.

[0236] In some embodiments, the structural similarity degree L SSIM can be obtained by the following formula: L SSIM = [l(H IR , TAG IR )] γ1 · [c(HIR , TAG IR )] γ2 · [s(H IR , TAG IR )] γ3

[0237] wherein the structural similarity degree L SSIM is composed of three parts: a brightness part l(H IR , TAG IR ), a contrast part c(H IR , TAG IR ) and a structure part s(H IR , TAG IR ), and γ1, γ2, γ3 are used to adjust the proportion of the three parts.

[0238] wherein, denotes the pixel mean of the super-resolution infrared image H IR , denotes the pixel mean of the label image TAG IR , denotes the pixel variance of the super-resolution infrared image H IR , denotes the pixel variance of the label image TAG IR , denotes the pixel covariance of the super-resolution infrared image H IR and the label image TAG IR .

[0239] C1, C2 and C3 are constants set for avoiding the denominator being 0.

[0240] It should be noted that other loss functions can also be used in the embodiments of the present disclosure, such as L1 loss function, L2 loss function, etc. The present disclosure does not limit this.

[0241] Based on the same inventive concept, the embodiments of the present disclosure also provide an infrared image super-resolution reconstruction system. FIG. 16 is a structural block diagram of an infrared image super-resolution reconstruction system provided by an embodiment of the present disclosure. As shown in FIG. 16, the infrared image super-resolution reconstruction system can be used to implement the infrared image super-resolution reconstruction method provided by the foregoing embodiments, and the infrared image super-resolution reconstruction system comprises a multi-scale feature acquisition module and a reconstruction module.

[0242] The multi-scale feature acquisition module is configured to acquire, through multi-scale feature extraction processing, first base feature maps respectively corresponding to the auxiliary image of the target scene at N different preset resolutions, and acquire second base feature maps respectively corresponding to the original infrared image of the target scene at the N different preset resolutions, where N is an integer greater than or equal to 2, and the resolution of the auxiliary image is greater than the resolution of the original infrared image.

[0243] The reconstruction module is configured to perform feature map fusion reconstruction according to the first base feature maps and the second base feature maps, to obtain a super-resolution infrared image.

[0244] For specific descriptions of the above modules, refer to the corresponding content in the foregoing embodiments, which will not be described here again.

[0245] Based on the same inventive concept, the embodiments of the present disclosure further provide an electronic device. FIG. 17 is a structural schematic diagram of an electronic device according to an embodiment of the present disclosure. As shown in FIG. 17, the electronic device according to an embodiment of the present disclosure includes one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the infrared image super-resolution reconstruction method according to any of the foregoing embodiments. The one or more I / O interfaces 103 are connected between the processor and the memory, and are configured to realize information interaction between the processor and the memory.

[0246] The processor 101 is a device with data processing capability, including but not limited to a central processing unit (CPU) and the like. The memory 102 is a device with data storage capability, including but not limited to a random access memory (RAM, more specifically, SDRAM, DDR, etc.), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory (FLASH). The I / O interface 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus) and the like.

[0247] In some embodiments, the processor 101, the memory 102, and the I / O interface 103 are connected to each other through a bus 104, and further connected to other components of the computing device.

[0248] In some embodiments, the one or more processors 101 include a field programmable gate array.

[0249] According to an embodiment of the present disclosure, a computer readable medium is further provided. The computer readable medium stores a computer program, and when the program is executed by a processor, the steps in the infrared image super-resolution reconstruction method according to any of the foregoing embodiments are implemented.

[0250] In particular, the processes described above with reference to the flow charts can be implemented as a computer software program in accordance with the embodiments of the present disclosure. For example, the embodiments of the present disclosure include a computer program product having a computer program tangibly embodied on a machine readable medium, the computer program containing program code for executing the methods illustrated by the flow charts. In such embodiments, the computer program can be downloaded and installed from a network via a communication portion, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the system of the present disclosure are executed.

[0251] Note that the computer readable medium shown in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF or the like, or any suitable combination of the above.

[0252] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0253] It is to be understood that the above-described embodiments are merely exemplary of the principles of the present disclosure and that various modifications can be made thereto without departing from the spirit and scope of the present disclosure. Such modifications are intended to be included within the scope of the present disclosure.

Claims

1. An infrared image super-resolution reconstruction method, wherein, The method comprises: obtaining first basic feature maps corresponding to an auxiliary image of a target scene at N different preset resolutions, and obtaining second basic feature maps corresponding to an original infrared image of the target scene at the N different preset resolutions, N being an integer greater than or equal to 2, the resolution of the auxiliary image being greater than the resolution of the original infrared image; performing feature map fusion and reconstruction according to the first basic feature maps and the second basic feature maps to obtain a super-resolution infrared image.

2. The reconstruction method of claim 1, wherein, The step of performing feature map fusion and reconstruction according to the first basic feature maps and the second basic feature maps comprises: for each preset resolution, fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution to obtain a modal fusion feature map corresponding to the target preset resolution; performing fusion and reconstruction according to the modal fusion feature maps corresponding to the preset resolutions to obtain the super-resolution infrared image.

3. The reconstruction method of claim 2, wherein, The reconstruction method is configured with a first working mode; in the first working mode, the step of fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution comprises: fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution according to an attention mechanism to obtain a first modal fusion feature map, the modal fusion feature map including the first modal fusion feature map.

4. The reconstruction method of claim 2 or 3, wherein, The reconstruction method is configured with a second working mode; in the second working mode, the step of fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution comprises: fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution without setting an attention mechanism to obtain a second modal fusion feature map, the modal fusion feature map including the second modal fusion feature map.

5. The reconstruction method of any one of claims 2 to 4, wherein, The reconstruction method is configured with a third working mode; in the third working mode, the step of fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution comprises: fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution according to an attention mechanism to obtain a first modal fusion feature map; fusing the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution without setting an attention mechanism to obtain a second modal fusion feature map; the modal fusion feature map includes the first modal fusion feature map and the second modal fusion feature map.

6. The reconstruction method of claim 5, wherein, in the third working mode, the step of performing fusion and reconstruction according to the modal fusion feature maps corresponding to the preset resolutions comprises: According to the first modal fusion feature map corresponding to each preset resolution, fusion reconstruction is performed to obtain a corresponding first super-resolution infrared image; and according to the second modal fusion feature map corresponding to each preset resolution, fusion reconstruction is performed to obtain a corresponding second super-resolution infrared image; The second super-resolution infrared image and the first super-resolution infrared image are subtracted to obtain a corresponding difference image, the difference image is weighted according to a preset weight coefficient, and the weighted difference image is superimposed on the first super-resolution infrared image to obtain a final super-resolution infrared image.

7. The reconstruction method of any one of claims 2 to 6, wherein, The steps of obtaining the first basic feature map corresponding to the auxiliary image of the target scene under N different preset resolutions and obtaining the second basic feature map corresponding to the original infrared image of the target scene under N different preset resolutions include: The first to-be-processed image corresponding to the auxiliary image under N different preset resolutions is obtained through sampling processing, and the second to-be-processed image corresponding to the original infrared image under N different preset resolutions is obtained through sampling processing; The first to-be-processed image corresponding to the target preset resolution is subjected to feature extraction, and the second to-be-processed image corresponding to the target preset resolution is subjected to feature extraction.

8. The reconstruction method of claim 7, wherein, The steps of feature extraction on the first to-be-processed image corresponding to the target preset resolution include: The first to-be-processed image corresponding to the target preset resolution is subjected to feature extraction in the first target domain and the second target domain to obtain a corresponding first feature map and a second feature map, and the first basic feature map includes the first feature map and the second feature map; The steps of feature extraction on the second to-be-processed image corresponding to the target preset resolution include: The second to-be-processed image corresponding to the target preset resolution is subjected to feature extraction in the first target domain and the second target domain to obtain a corresponding third feature map and a fourth feature map, and the second basic feature map includes the third feature map and the fourth feature map; The first target domain and the second target domain are different domains, and the first target domain and the second target domain are selected from one of a frequency domain, a spatial domain, a color domain and a gradient domain.

9. The reconstruction method of claim 8, wherein, The steps of fusion of the first basic feature map corresponding to the target preset resolution and the second basic feature map corresponding to the target preset resolution include: The first feature map corresponding to the target preset resolution and the third feature map are fused to obtain a first domain fusion feature map corresponding to the target preset resolution; The second feature map corresponding to the target preset resolution and the fourth feature map are fused to obtain a second domain fusion feature map corresponding to the target preset resolution; The first domain fusion feature map corresponding to the target preset resolution and the second domain fusion feature map are fused to obtain the modal fusion feature map.

10. The reconstruction method of any one of claims 7 to 9, wherein, The step of obtaining, through sampling processing, the first to-be-processed image corresponding to the auxiliary image at N different preset resolutions comprises: The auxiliary image is taken as a first to-be-processed image, and N-1 first to-be-processed images corresponding to the auxiliary image are obtained through N-1 times of down-sampling processing based on the auxiliary image. The step of obtaining, through sampling processing, the second to-be-processed image corresponding to the original infrared image at N different preset resolutions comprises: The original infrared image is taken as a second to-be-processed image, and N-1 second to-be-processed images corresponding to the original infrared image are obtained through N-1 times of up-sampling processing based on the original infrared image.

11. The reconstruction method of any one of claims 3 to 10, wherein, In the case of being provided with an attention mechanism, the step of fusing the two feature maps comprises: Similarity data between the two feature maps is calculated according to the attention mechanism, the similarity data comprising similarity degrees corresponding to different feature dimensions; Attention weight data is generated according to the similarity degree data, the attention weight data comprising: a plurality of attention weights corresponding to the similarity degrees in the similarity degree data, the attention weights being positively correlated with the corresponding similarity degrees; One of the two feature maps is subjected to image weighting processing according to the attention weight data to obtain a corresponding weighted guided feature map, and the weighted guided feature map is fused with the other of the two feature maps.

12. The reconstruction method of any one of claims 2 to 11, wherein, The step of fusing and reconstructing the modal fusion feature maps corresponding to each preset resolution comprises: For each preset resolution, the modal fusion feature map corresponding to the target preset resolution and the modal fusion feature map corresponding to an adjacent preset resolution adjacent in size to the target preset resolution are fused to obtain a scale fusion feature map corresponding to the target preset resolution; The scale fusion feature maps corresponding to each preset resolution are fused and reconstructed to obtain the super-resolution infrared image.

13. The reconstruction method of any one of claims 1 to 12, wherein, Before the steps of obtaining the first basic feature map and obtaining the second basic feature map, the method further comprises: An original high-resolution image of the target scene captured by a first shooting module and an original infrared image of the target scene captured by a second shooting module are obtained; The original high-resolution image and the original infrared image are subjected to registration processing to extract the auxiliary image that is spatially aligned with the original infrared image from the original high-resolution image; The reconstruction method is configured with a fourth working mode; In the fourth working mode, after the step of registering the original high-resolution image and the original infrared image, the method further comprises: All element values in a matrix corresponding to the auxiliary image are replaced with 0.

14. A method for constructing an infrared image super-resolution reconstruction model, wherein, The method comprises: An infrared image super-resolution reconstruction initial model is constructed, the infrared image super-resolution reconstruction initial model being configured to reconstruct a corresponding super-resolution infrared image by using the reconstruction method in any one of the above claims 1 to 13 based on an input auxiliary image and an original infrared image. The infrared image super-resolution reconstruction model is obtained by training an initial model for infrared image super-resolution reconstruction by using a pre-acquired sample set, the sample set comprising a plurality of samples, each sample comprising an auxiliary image, an original infrared image and a label image, the label image being another infrared image of a scene corresponding to the original infrared image, and the resolution of the label image being greater than the resolution of the original infrared image.

Images